|
|
The Project This project proposes improving the generic support of x86 CPUs that implement the Machine Check Architecture (MCA). It also proposes to create a model-specific plugin interface whereby model-specific MCA implementation may supplement amd extend the generic support rather than stand alone. Finally, the project proposes to redesign the cpu-module interface such that the MCA support it provides will be usable in both virtualized and bare-metal environments. Background Prior to the "amd64 FMA" project (putback in February 2006) support for MCA was woeful, with no intelligent enabling of error detectors, no polling for errors that do not produce a machine check, and a clumsy machine-check handler which barfed an incomplete and raw dump of error telemetry to the console before panic. There was no model-specific MCA support whatsoever. The "amd64 FMA" project, putback in February 2006, delivered detailed model-specific support for AMD family 0xf revisions B to E (also known as AMD Opteron/Athlon64/Turion64, K8 etc; revisions B to E all have DDR1 memory) and moderately improved generic MCA support. The AMD family family 0xf revs B to E support included: - performing MCA and model-specific configuration beyond what the BIOS had setup - enabling all MCA banks and machine-check on all uncorrectable error types - an error poller for errors that do not machine-check (such as correctable memory errors) - an error handler that collects all telemetry and bakes it into a model-specific error report which is dispatched for logging and diagnosis via a robust mechanism that will carry telemetry across a panic/reboot cycle - detailed knowledge of model-specific error types, able to produce an ereport error class from the raw telemetry and to specify the appropriate ereport payload members to include for that error class - a memory-controller driver able to discover all aspects of memory configuration and to map system addresses to memory resource - a topology enumerator to describe cpu and memory topology and properties to the topology library, for use in diagnosis software - eversholt diagnosis rules to diagnose all telemetry - corresponding fault classes and sun.com/msg knowledge articles The generic support, applied to anything other than the AMD models above including all Intel processors and any AMD models outside the model range above, included: - generic MCA detector and error type enabling - a machine check exception handler which dumps all MCA telemetry to the console, still in raw form - no error poller - no preparation of ereports (even for panic events), and therefore no diagnosis of any kind The above support was backported to Solaris 10 Update 2. The model-specific AMD support was updated in build 51 to include AMD family 0xf revision F and G support. Those revisions use DDR2 memory, and most work was required in the memory-controller driver. This additonal support has been backported to Solaris 10 Update 4 (pending release at this time). The Problem 1) The first problem is that the generic cpu MCA support is light on features: - it lacks an error poller; we won't log memory correctable ECC errors, for example - it does not raise ereports for logging and diagnosis, even for errors that result in a panic - on an error that results in a panic at best you have some raw, uninterpreted error register values and no error classification or interpretation; if the system lacks console logging no evidence at all may remain when the system resets. This generic support is applied to all Intel processors, and to AMD processors not handled by already-delivered AMD model-specific support. Where a platform with full model-specific support is upgradeable to new processors that drop in to existing sockets and where the new processor family/model/stepping is no longer covered by the existing model-specific support, the platform after upgrade will regress to the dumb generic cpu support. So for all Intel and for new AMD processors there is a window of OpenSolaris builds during which MCA support is very light - it takes some time to implement full model-specific support for a cpu. Improving generic cpu support will raise the base level of support for all processors that implement MCA/MCE features. Purely from the Sun point of view, and not a concern for OpenSolaris itself, the delay in introducing new model-specific support is further stretched in having to backport to a suitable Solaris 10 Update release; it is also possible that products based on a new cpu model can revenue-release before corresponding full FMA support for the cpu model is available in Solaris 10. 2) The second problem is that the current cpu module implementations (generic and AMD family 0xf) and the cpu module interface itself are completely invalildated when not running natively. A separate project will be proposed for FMA/Xen support, but generic cpu MCA support and the cpu module interface changes required for the current project proposal should be made with that requirement in mind. 3) A third problem is that the current cpu module design leads to substantial code duplication across model-specific implementations (e.g., implementing an error poller in each). This should not be necessary given the common base MCA that they adhere to. Key Objectives 1) Enrich the generic cpu support to include an error poller, error type classification (as far as the architectural definition allows), raising of ereports capturing all the telemetry for logging and some degree of diagnosis (again as much as is possible generically). 2) Design a new cpu model-specific MCA module interface which will supplement generic cpu MCA support to avoid code duplication and improve maintainability. This new interface should be rich enough to allow the refactoring and redelivery of exising AMD model-specific support, and should be designed with upcoming Intel model-specific support in mind. 3) Identify vendor-specific but nevertheless fairly widely- applicable (to many models from that vendor) features that can be captured in an intermediate level of model-specific support for new CPU models from that vendor. For example in the absence of a fully-featured memory-controller driver we may be unable to determine the full memory topology and provide full error address to memory resource translation, but both AMD and Intel in recent models have hardware registers that can identify where errors have occured (albeit at a lower resolution than a full memory-controller driver may be able to provide). Such support could provide a level of "generic AMD" and "generic Intel" support for new cpu releases that is better than the lowest common denominator generic support, and which is later not loaded in favour of full model-specific support once that is available. 4) Redesign the cpu module interface to cope with running both natively and in a virtualized setting, Xen Solaris domain 0 in particular. The new interface should not assume any 1:1 relationship between Solaris logical cpus (cpu_t structures) and real physical cores on the system, should not assume that being bound to a Solaris cpu means we can perform reads from hardware registers all from the same real core, etc. Instead the points at which we acquire telemetry and other hardware details should be well-defined, and the information gathered via calls into a newly-defined native or virtualized layer. Implementation of the virtualized layer itself is left to another project. 5) The project deliverables must include some degree of hardware error-injection, or of error simulation if that is not possible. Existing test harnesses should be extended to include generic cpu scenarios. 6) As with any FMA project, this should include an FMA portfolio detailing all events, diagnosis thereof, ASRU boundaries etc. To date there are no open FMA portfolios so those outside of Sun will have little idea of what this involves - that will become clear during the progres of the project. 7) This project will not deliver full model-specific support for any Intel cpus nor for upcoming "AMD family 0x10" processors. It should, however, proceed with the needs of those projects in mind and should colloborate with those projects to understand those needs. Community Participation Participation and contribution from the OpenSolaris community is encouraged. This project will use public documentation from both AMD and Intel, available from the websites of those companies. As such, there is no vendor NDA or similar barrier to community participation. It may be necessary for some design decisions to be made by or in conjunction with those with access to NDA documentation, but these are expected to be the exception. Error injector sources are currently in the closed tree. Otherwise all applicable source is available in OpenSolaris. The event registry is now available externally to Sun, so there is no reason that the FMA portfolio cannot be prepared and published externally as part of this project. |