OpenSolaris

1. Community: Performance
   1. Announcements
   2. News
   3. Blogs
   4. Discussions
   5. OpenSolaris NUMA project
      1. Observability
         1. DTrace scripts
         2. APIs
         3. Debugging
         4. Wish list
         5. Tools
            1. lgrpinfo(1)
               1. lgrpinfo(1M) man page
            2. plgrp(1)
               1. plgrp(1M) proposed man page.
            3. pmadvise(1)
               1. pmadvise(1) manual
            4. pmap(1)
               1. PSARC proposal for pmap extensions (draft)
            5. Perl lgroup module
               1. Sun::Solaris::Lgrp - Perl interface to Solaris liblgrp(3LIB) library.
            6. ps(1)
            7. prstat(1)
            8. NUMA Observability Tools PSARC case
            9. Instructions for downloading tools
      2. Improved Latency Discovery
      3. 8 CPU chip Opteron
      4. Load Balancing
   6. libMicro
      1. Sample Comparison
   7. FileBench
      1. Sample Comparison
      2. Quick Start Guide to FileBench
         1. Example Comparison of the multistreamread.f workload
      3. Gotchas
   8. Events
   9. Leaders
   10. Technical Documents
   11. Observers
   12. Files
   13. Roles and Grants

Description

This is the web page for the work needed to improve MPO for machines with 8 Opteron CPU chips which is the biggest configuration supported by Hypertransport now. These NUMA machines have a more complicated node topology than the smaller Opteron configurations with up to 4 hops from a given node/CPU chip to the furthest node. All of our previous Opterons have only had up to 2 hops in remote latency.

The 8 node topology is typically like a ladder that is bounded by a horizontal rung at the top and bottom. It looks like a rectangle with 2 horizontal rungs and 4 nodes arranged up each side. The topology may be optimized to lower the number of hops needed for some or all of the nodes to reach all the other nodes by criss crossing the rungs in the middle of the rectangle and/or directly connecting the corners of the rectangle. Currently, each node has 3 Hypertransport links and all the links in the system can't be used for connecting nodes to each other because some are needed for I/O.

These topologies also have the interesting property that some of the nodes can acess all of the resources in the machine in fewer hops than some of the other nodes. The nodes in the middle of the rectangle can access all the other nodes in the machine in the fewer hops than some or all of the nodes on the corners can. Currently, MPO does not optimize for nodes in this sweet spot of the node topology.

This project is intended to enhance MPO in order to improve functionality and performance on the 8 node Opteron machines (such as Tyan, etc.). This may include making improvements and fixing any bugs with the existing MPO support for Opteron, lgroup topology, and optimizations. The lgroup topology may need some work to handle the more complicated node topology better and the existing thread placement mechanism will need to be able to recognize and prefer to home threads to nodes that can reach all the resources in the machine in the fewest hops first. Some performance evaluation, analysis, and further optimizations will need to be done too.