Platforms II Dezs Sima 2011 December Ver 1

Platforms II. Dezső Sima 2011 December (Ver. 1. 6) Sima Dezső, 2011

3. Platform architectures

Contents • Platform architectures • 3. 1. Design space of the basic platform architecture • 3. 2. The driving force for the evolution of platform architectures • 3. 3. DT platforms • 3. 3. 1. Design space of the basic architecture of DT platforms • 3. 3. 2. Evolution of Intel’s home user oriented multicore DT platforms • 3. 3. 3. Evolution of Intel’s business user oriented multicore DT platforms • 3. 4. DP server platforms • 3. 4. 1. Design space of the basic architecture of DP server platforms • 3. 4. 2. Evolution of Intel’s low cost oriented multicore DP server platforms • 3. 4. 3. Evolution of Intel’s performance oriented multicore DP server platforms

Contents • 3. 5. MP server platforms • 3. 5. 1. Design space of the basic architecture of MP server platforms • 3. 5. 2. Evolution of Intel’s multicore MP server platforms • 3. 5. 3. Evolution of AMD’s multicore MP server platforms

3. 1. Design space of the basic platform architecture

3. 1 Design space of the basic platform architecture (1) Platform architecture Architecture of the processor subsystem Architecture of the memory subsystem • Interpreted only for DP/MP systems • In SMPs: Specifies the interconnection of the processors and the chipset • In NUMAs: Specifies the interconnections between the processors Specifies • the point and • the layout Architecture of the I/O subsystem Specifies the structure of the I/O subsystem (Will not be discussed) of the interconnection Example: Core 2/Penryn based MP SMP platform P P P . . FSB . . MCH . . ICH Processors are connected to the MCH by individual buses ICH • Memory is attached to the MCH • There are serial FB-DIMM channels . . MCH P . . ICH The chipset consist of two parts designated as the MCH and the ICH

3. 1 Design space of the basic platform architecture (2) The notion of Basic platform architecture Platform architecture Architecture of the processor subsystem Architecture of the memory subsystem Basic platform architecture Architecture of the I/O subsystem

3. 1 Design space of the basic platform architecture (2) The notion of Basic platform architecture Platform architecture Architecture of the processor subsystem Architecture of the memory subsystem Basic platform architecture Architecture of the I/O subsystem

3. 1 Design space of the basic platform architecture (3) Architecture of the processor subsystem Interpreted only for DP and MP systems. The interpretation depends on whether the multiprocessor system is an SMP or NUMA Architecture of the processor subsystem SMP systems NUMA systems Scheme of attaching the processors to the rest of the platform Scheme of interconnecting the processors Examples. . P P FSB MCH . . . . P ICH P . . Memory P . .

3. 1 Design space of the basic platform architecture (4) a) Scheme of attaching the processors to the rest of the platform (In case of SMP systems) Scheme of attaching the processors to the rest of the platform MP platforms DP platforms Single FSB P P MCH Single FSB Dual FSBs P MCH Memory P P MCH P P Memory P P MCH Quad FSBs P P Memory P P MCH P . . Memory P P Dual FSBs Memory

3. 1 Design space of the basic platform architecture (5) b) Scheme of interconnecting the processors (In case of NUMA systems) Scheme of interconnecting the processors Partially connected mesh Fully connected mesh Memory P P Memory

3. 1 Design space of the basic platform architecture (6) The notion of Basic platform architecture Platform architecture Architecture of the processor subsystem Architecture of the memory subsystem Basic platform architecture Architecture of the I/O subsystem

3. 1 Design space of the basic platform architecture (7) Architecture of the memory subsystem (MSS) Point of attaching the MSS Layout of the interconnection

3. 1 Design space of the basic platform architecture (8) a) Point of attaching the MSS (Memory Subsystem) (1) Platform MCH Memory ? Platform Processor Memory Point of attaching the MSS

3. 1 Design space of the basic platform architecture (9) Point of attaching the MSS – Assessing the basic design options (2) Point of attaching the MSS Attaching memory to the MCH (Memory Control Hub) Attaching memory to the processor(s) • Longer access time (~ 20 – 70 %), • Shorter access time (~ 20 – 70 %), • As the memory controller is on the MCH die, the memory type (e. g. DDR 2 or DDR 3) and speed grade is not bound to the processor chip design. • As the memory controller is on the processor die, the memory type (e. g. DDR 2 or DDR 3) and speed grade is bound to the processor chip design.

3. 1 Design space of the basic platform architecture (10) Related terminology Point of attaching the MSS Attaching memory to the MCH (Memory Control Hub) Attaching memory to the processor(s) DT platforms DP/MP platforms DT Systems with off-die memory controllers Shared memory DP/MP systems DT Systems with on-die memory controllers Distributed memory DP/MP systems SMP systems (Symmetrical Multiporocessors) NUMA systems (Systems w/ non uniform memory access)

3. 1 Design space of the basic platform architecture (11) Example 1: Point of attaching the MSS in DT systems Point of attaching the MSS Attaching memory to the MCH Attaching memory to the processor(s) Processor FSB MCH Memory ICH DT System with off-die memory controller Memory MCH ICH DT System with on-die memory controller Examples Intel’s processors before Nehalem Intel’s Nehalem and subsequent processors

3. 1 Design space of the basic platform architecture (12) Example 2: Point of attaching the MSS in SMP-based DP servers Point of attaching the MSS Attaching memory to the MCH Processor Attaching memory to the processor(s) Processor Memory FSB MCH Processor Memory ICH • Shared memory DP server aka Symmetrical Multiprocessor (SMP) • Memory does not scale with the number of processors MCH ICH • Distributed memory DP server aka System w/ non-uniform memory access (NUMA) • Memory scales with the number of processors Examples Intel’s processors before Nehalem Intel’s Nehalem and subsequent processors

3. 1 Design space of the basic platform architecture (13) Point of attaching the MSS Attaching memory to the MCH Examples Attaching memory to the processor(s) Ultra. SPARC II (1 C) (~1997) Ultra. SPARC III (2001) and all subsequent Sun lines AMD’s K 7 lines (1 C) (1999 -2003) Opteron server lines (2 C) (2003) and all subsequent AMD lines POWER 4 (2 C) (2001) PA-8800 (2004) PA-8900 (2005) and all previous PA lines Core 2 Duo line (2 C) (2006) and all preceding Intel lines Core 2 Quad line (2 x 2 C) (2006/2007) Penryn line (2 x 2 C) (2008) Montecito (2 C) (2006) POWER 5 (2 C) (2005) and subsequent POWER families Nehalem lines (4) (2008) and all subsequent Intel lines Tukwila (4 C) (2010? ? ) Figure: Point of attaching the MSS

3. 1 Design space of the basic platform architecture (14) b) Layout of the interconnection Attaching memory via serial links Attaching memory via parallel channels Data are transferred over parallel buses Data are transferred over point-to-point links in form of packets 1 01 15 MC MC 1 0 0 t E. g: 16 cycles/packet on a 1 -bit wide link t 01 E. g: 64 bits data + address, command control as well as clock signals in each cycle MC t E. g: 4 cycles/packet on a 4 -bit wide link Figure: Attaching memory via parallel channels or serial links

3. 1 Design space of the basic platform architecture (15) b 1) Attaching memory via parallel channels The memory controller and the DIMMs are connected • by a single parallel memory channel • or a few number of memory channels to synchron DIMMs, such as SDRAM, DDR 2 or DDR 3 DIMMs. Example 1: Attaching DIMMs via a single parallel memory channel to the memory controller that is implemented on the chipset [45]

3. 1 Design space of the basic platform architecture (16) Example 2: Attaching DIMMs via 3 parallel memory channels to memory controllers implemented on the processor die (This is actually Intel’s the Tylersburg DP platform, aimed at the Nehalem-EP processor, used for up to 6 cores) [46]

3. 1 Design space of the basic platform architecture (17) The number of lines of the parallel channels The number of lines needed depend on the kind of the memory modules, as indicated below: SDRAM 168 -pin DDR 184 -pin DDR 2 240 - pin DDR 3 240 -pin All these DIMM modules provide an 8 -byte wide datapath and optionally ECC and registering.

3. 1 Design space of the basic platform architecture (18) b 2) Attaching memory via serial links Serial memory links are point-to-point interconnects that use differential signaling. Attaching memory via serial links Serial links attach FB-DIMMs Serial links attach S/P converters w/ parallel channels . . Serial . . link Proc. /MCH . . Serial link S/P . . link . . Proc. /MCH Serial . . FB-DIMMs provide buffering and S/P conversion

3. 1 Design space of the basic platform architecture (19) Example 1: FB-DIMM links in Intel’s Bensley DP platform aimed at Core 2 processors-1 Xeon 5400 Xeon 5200 Xeon 5000 Xeon 5100 Xeon 5300 / / / (Harpertown) (Dempsey) (Woodcrest) (Clowertown) 2 x 2 C 2 C 2 x 1 C 2 C 2 x 2 C 65 nm Pentium 4 Prescott DP (2 x 1 C)/ Core 2 (2 C/2*2 C) 65 nm Pentium 4 Prescott DP (2 C)/ Core 2 (2 C/2*2 C) FSB E 5000 MCH ESI 631*ESB/ 632*ESB IOH FB-DIMM w/DDR 2 -533 ESI: Enterprise System Interface 4 PCIe lanes, 0. 25 GB/s per lane (like the DMI interface, providing 1 GB/s transfer rate in each direction)

3. 1 Design space of the basic platform architecture (20) Example 2: SMI links in Intel’s the Boxboro-EX platform aimed at the Nehalem-EX processors-1 Xeon 6500 (Nehalem-EX) (Becton) SMB SMB Nehalem-EX (8 C) Westmere-EX (10 C) SMB SMI links Xeon E 7 -2800 or (Westmere-EX) QPI Nehalem-EX (8 C) Westmere-EX (10 C) QPI DDR 3 -1067 7500 IOH ESI ICH 10 SMB SMB SMI links DDR 3 -1067 ME SMI: Serial link between the processor and the SMB: Scalable Memory Buffer with Parallel/serial conversion Nehalem-EX aimed Boxboro-EX scalable DP server platform (for up to 10 cores)

3. 1 Design space of the basic platform architecture (21) Example 2: The SMI link of Intel’s the Boxboro-EX platform aimed at the Nehalem-EX processors-2 [26] SMB • The SMI interface builds on the Fully Buffered DIMM architecture with a few protocol changes, such as those intended to support DDR 3 memory devices. • It has the same layout as FB-DIMM links (14 outbound and 10 inbound differential lanes as well as a few clock and control lanes). • It needs altogether about 50 PC trails.

3. 1 Design space of the basic platform architecture (22) Design space of the architecture of the MSS Layout of the interconnection Attaching memory via parallel channels Attaching memory to the MCH Serial links attach S/Pconverters w/ par. channels . . S/P . . . MCH . . . . S/P . . P. . P . . . S/P . . . P. . P S/P . . . Attaching memory to the processor(s) Serial links attach FB-DIMMs . . Point of attaching memory Parallel channels attach DIMMs Attaching memory via serial links . .

3. 1 Design space of the basic platform architecture (23) Max. number of memory channels that can be implemented while using particular design options of the MSS Subsequent fields from left to right and from top to down of the design space of the architecture of MSS allow to implement an increasing number of memory channels (n M), as discussed in Section 4. 2. 5 and indicated in the next figure.

3. 1 Design space of the basic platform architecture (24) Layout of the interconnection Design space of the architecture of the MSS Attaching memory via parallel channels Attaching memory to the MCH Serial links attach S/Pconverters w/ par. channels . . S/P . . . MCH . . . . S/P . . P. . P . . . S/P . . . P. . P S/P . . . Attaching memory to the processor(s) Serial links attach FB-DIMMs . . Point of attaching memory Parallel channels attach DIMMs Attaching memory via serial links . . n. C

3. 1 Design space of the basic platform architecture (25) The design space of the basic platform architecture-1 Platform architecture Architecture of the processor subsystem Architecture of the memory subsystem Basic platform architecture Architecture of the I/O subsystem

3. 1 Design space of the basic platform architecture (26) The design space of the basic platform architectures-2 Obtained as the combinations of the options available for the main aspects discussed. Basic platform architecture Architecture of the processor subsystem Scheme of attaching the processors (In case of SMP systems) Scheme of interconnecting the processors (In case of NUMA systems) Architecture of the memory subsystem (MSS) Point of attaching the MSS Layout of the interconnection

3. 1 Design space of the basic platform architecture (27) The design space of the basic platform architectures of DT, DP and MP platforms will be discussed next in the Sections 3. 3. 1, 3. 4. 1 and 3. 5. 1. Design space of the basic architecture of particular platforms Design space of the basic architecture of DT platforms Section 3. 3. 1 Design space of the basic architecture of DP server platforms Section 3. 4. 1 Design space of the basic architecture of MP server platforms Section 3. 5. 1

3. 2. The driving force for the evolution of platform architectures

3. 2 The driving force for the evolution of platform architectures (1) The peak per processor bandwidth demand of a platform Let’s consider a single processor of a platform and the bandwidth available for it (BW). The available (peak) memory bandwidth of a processor (BW) is a product of • the number of memory channels available per processor (n. M) , • their width (w) as well as • the transfer rate of the memory used (f. M): BW = n. M x w x f. M • BW needs to be scaled with the peak performance of the processor. • The peak performance of the processor increases linearly with the core count (n C). The per processor memory bandwidth (BW) needs to be scaled with the core count (n C).

3. 2 The driving force for the evolution of platform architectures (2) If we assume a constant width for the memory channel (w = 8 Byte), it can be stated that n. M x f M needs to be scaled with the number of cores that is it needs to be doubled approximately every two years. This statement summarizes the driving force • for raising the bandwidth of the memory subsystem, and • at the same time also it is the major motivation for the evolution of platform architectures.

3. 2 The driving force for the evolution of platform architectures (3) The bandwidth wall • As recently core counts (n. C) double roughly every two years, also the per processor bandwidth demand of platforms (BW) doubles roughly every two years, as discussed before, • On the other hand, memory speed (f. M) doubles only approximately every four years, as indicated e. g. in the next figure for Samsung’s memory technology.

3. 2 The driving force for the evolution of platform architectures (4) Evolution of the memory technology of Samsung [12] The time span between e. g. DDR-400 and DDR 3 -1600 is approximately 7 years, this means roughly a doubling of memory speeds (f. M) in every 4 years.

3. 2 The driving force for the evolution of platform architectures (5) • This fact causes a widening gap between the bandwidth demand achievable bandwidth growth due to increasing memory speed. • This fact can be designated as the bandwidth wall. It is the task of the developers of platform architectures to overcome the bandwidth wall by providing the needed number of memory channels.

3. 2 The driving force for the evolution of platform architectures (6) The square root rule of scaling the number of memory channels It can be shown that in case when the core count (n C) increases according to Moore’s law and memory subsystems will be evolved by using in them the fastest available memory devices, as typical, then the number of per processor available memory channels needs be scaled as n. M(n. C) = √ 2 x √n. C to provide altogether a linear bandwidth scaling with the core count (n C). Then the scaled number of memory channels available per processor (n M(n. C)) and the increased device speed (f. M) together will provide the needed linear scaling of the per processor bandwidth (BW) with n. C. The above relationship can be termed as the square root rule of scaling the number of memory channels. Remark For multiprocessors incorporating n. P processors then the total number of memory channels of the platform (NM) amounts to NM = n P x n M

3. 3. DT platforms • 3. 3. 1. Design space of the basic architecture of DT platforms • 3. 3. 2. Evolution of Intel’s home user oriented multicore DT platforms • 3. 3. 3. Evolution of Intel’s business user oriented multicore DT platforms

3. 3 DT platforms 3. 3. 1 Design space of the basic architecture of DT platforms

3. 3. 1 Design space of the basic architecture of DT platforms (1) Point of attaching the MSS DT platforms . . P . . MCH . . . ICH Pentium D/EE to Penryn (Up to 4 C) 1. G. Nehalem to Sandy Bridge (Up to 6 C) P. . . P . . ICH . . . S/P . . . S/P ICH . . P MCH . . Serial links attach. Serial links Parallel channels S/P conv. w/ par. chan. attach FB-Di. MMs attach DIMMs . . Attaching memory via parallel channels P . . Attaching memory via serial links Attaching memory to the processor . . Layout of the interconnection Attaching memory to the MCH . . No. of mem. channels

3. 3. 1 Design space of the basic architecture of DT platforms (2) Attaching memory to the MCH Pentium D/EE 2 x 1 C (2005/6) Core 2 2 C (2006) Core 2 Quad 2 x 2 C (2007) Penryn 2 C/2 x 2 C (2008) Attaching memory to the processor 1. G. Nehalem 4 C (2008) Westmere-EP 6 C (2010) 2. G. Nehalem 4 C (2009) Westmere-EP 2 C+G (2010) Sandy Bridge 2 C/4 C+G (2011) Sandy Bridge-E 6 C (2011) No need for higher memory bandwidth through serial memory interconnection No. of memory channels Seria l links attach Parallel channels attach DIMMs FB-DIMMs Point of attaching the MSS Serial links attach S/P converters w/ par. channels Attaching memory via parallel channels Attaching memory via serial links Layout of the interconnection Evolution of Intel’s DT platforms (Overview)

3. 3. 2 Evolution of Intel’s home user oriented multicore DT platforms (1) Up to DDR 2 -667 Pentium D/ Pentium EE (2 x 1 C) FSB 945/955 X/975 X MCH DMI ICH 7 Anchor Creek (2005) 2/4 DDR 2 DIMMs up to 4 ranks Up to DDR 3 -1067 Up to DDR 2 -800 Up to DDR 3 -1067 Core 2 2 C Core 2 Quad (2 x 2 C) /Penryn (2 C/2*2 C) FSB 965/3 -/4 - Series MCH DMI ICH 8/9/10 Bridge Creek (2006) (Core 2 aimed) Salt Creek (2007) (Core 2 Quad aimed) Boulder Creek (2008) (Penryn aimed) 1. gen. Nehalem (4 C)/ Westmere-EP (6 C) QPI X 58 IOH DMI ICH 10 Tylersburg (2008)

3. 3. 2 Evolution of Intel’s home user oriented multicore DT platforms (2) Up to DDR 3 -1067 Up to DDR 3 -1333 1. gen. Nehalem (4 C)/ Westmere-EP (6 C) QPI X 58 IOH DMI 2. gen. Nehalem (4 C)/ Westmere-EP (2 C+G) DMI FDI 5 - Series PCH Up to DDR 3 -1333 Sandy Bridge (4 C+G) DMI 2 FDI 6 - Series PCH ICH 10 Tylersburg (2008) Kings Creek (2009) Sugar Bay (2011)

3. 3. 2 Evolution of Intel’s home user oriented multicore DT platforms (3) Up to DDR 3 -1600 Up to DDR 3 -1067 1. gen. Nehalem (4 C)/ Westmere-EP (6 C) QPI X 58 IOH Sandy Bridge-E (4 C)/6 C) DMI 2 X 79 PCH DMI ICH 10 Tylersburg (2008) DDR 3 -1600: up to 1 DIMM per channel DDR 3 -1333: up to 2 DIMMs per channel Waimea Bay (2011)

3. 3. 3 Evolution of Intel’s business user oriented multicore DT platforms (1) Up to DDR 2 -800 Up to DDR 3 -1067 Up to DDR 2 -667 Pentium D/ Pentium EE (2 x 1 C) 2/4 DDR 2 DIMMs up to 4 ranks Core 2 (2 C) Core 2 Quad (2 x 2 C) Penryn (2 C/2*2 C) FSB 945/955 X/975 X MCH DMI ICH 7 LCI 82573 E Gb. E (Tekoe) ME Gigabit Ethernet LAN connection Lyndon (2005) Q 965/Q 35/Q 45 MCH ME DMI C-link ICH 8/9/10 LCI/GLCI 82566/82567 LAN PHY Up to DDR 3 -1333 2. gen. Nehalem (4 C) Westmere-EP (2 C+G) FDI DMI Q 57 PCH PCIe 2. 0/SMbus 2. 0 82578 Gb. E LAN PHY Gigabit Ethernet LAN connection Averill Creek (2006) (Core 2 aimed) Weybridge (2007) (Core 2 Quad aimed) Mc. Creary (2008) (Penryn aimed) ME Piketon (2009)

3. 3. 3 Evolution of Intel’s business user oriented multicore DT platforms (2) Up to DDR 3 -1333 2. gen. Nehalem (4 C) Westmere-EP (2 C+G) FDI DMI Q 57 PCH Sandy Bridge (4 C+G) FDI ME PCIe 2. 0/SMbus 2. 0 82578 Gb. E LAN PHY Gigabit Ethernet LAN connection Piketon (2009) Up to DDR 3 -1333 DMI 2 Q 67 PCH ME PCIe 2. 0/SMbus 2. 0 Gb. E LAN Gigabit Ethernet LAN connection Sugar Bay (2011)

3. 4. DP server platforms • 3. 4. 1. Design space of the basic architecture of DP server platforms • 3. 4. 2. Evolution of Intel’s low cost oriented multicore DP server platforms • 3. 4. 3. Evolution of Intel’s performance oriented multicore DP server platforms

3. 4 DP server platforms 3. 4. 1 Design space of the basic architecture of DP server platforms

3. 4. 1 Design space of the basic architecture of DP server platforms (1) P P P. . MCH. . Core 2/Penryn 2 C/2 x 2 C (2006/7) P P Nehalem-EP to Sandy Bridge -EP/EN Up to 8 C (2009/11) P . . . . ICH P . . . . MCH . . . . ICH 90 nm Pentium 4 DP 2 x 1 C (2005) MCH P . . ICH P P . . MCH . . . . 65 nm Pentium 4 DP 2 x 1 C Core 2/Penryn 2 C/2 x 2 C (2006/7) P P . . S/P P . . ICH . . S/P . . ICH . . . . S/P . . MCH . . . . S/P P . . Serial links attach. Serial links Parallel channels S/P conv. w/ par. chan. attach FB-Di. MMs attach DIMMs P NUMA . . Attaching memory via parallel channels Dual FSBs . . Attaching memory via serial links Single FSB . . Layout of the interconnection DP platforms . . Nehalem-EX/Westmere-EX 8 C/10 C (2010/11) n. M

3. 4. 1 Design space of the basic architecture of DP server platforms (2) Scheme of attaching and interconnecting DP processors Parallel channels attach DIMMs Single FSB Serial links attach. Serial links S/P converters attach FB-Di. MMs w/ par. chan. Attaching memory via parallel channels Attaching memory via serial links Layout of the interconnection SMP NUMA Dual FSBs Eff. Sandy Bridge-EN 8 C (2011) Core 2 2 C/ 90 nm Pentium 4 DP Eff. Core 2 Quad 2 x 2 C/ Romley-EN 2 x 1 C (2006) Penryn 2 C/2 x 2 C Nehalem-EP 4 C (2009) (Paxville DP) (2006/2007) Westmere-EP 6 C (2010) HP (Cranberry Lake) (Tylersburg-EP) HP Sandy Bridge-EP 8 C (20 11) Romley-EP HP 65 nm Pentium 4 DP 2 x 1 C Core 2 2 C Core 2 Quad 2 x 2 C Penryn 2 C/2 x 2 C (2006/2007) HP (Bensley) No. of memory channels Evolution of Intel’s DP platforms (Overview) Nehalem-EX/ Westmere-EX 8 C/10 C (2010/2011) (Boxboro-EX) No. of memory channels n. M

3. 4. 2 Evolution of Intel’s low cost oriented multicore DP server platforms (1) 3. 4. 2 Evolution of Intel’s low cost oriented multicore DP server platforms

3. 4. 2 Evolution of Intel’s low cost oriented multicore DP server platforms (2) Evolution from the Pentium 4 Prescott DP aimed DP platform (up to 2 cores) to the Penryn aimed Cranberry Lake DP platform (up to 4 cores) Xeon 5300 (Clowertown) 2 x 2 C Xeon DP 2. 8 /Paxville DP) 90 nm Pentium 4 Prescott DP (2 C) FSB Xeon 5400 Xeon 5200 or (Harpertown) 4 C 2 C Core 2 (2 C/ Core 2 Quad (2 x 2 C)/ Penryn (2 C/2 x 2 C) Core 2 (2 C/ Core 2 Quad (2 x 2 C)/ /Penryn (2 C/2 x 2 C) FSB E 7520 MCH E 5100 MCH HI 1. 5 ICH 5 R/ 6300 ESB IOH or DDR-266/333 DDR 2 -400 ESI DDR 2 -533/667 ICHR 9 90 nm Pentium 4 Prescott DP aimed DP server platform (for up to 2 C) Penryn aimed Cranberry Lake DP server platform (for up to 4 C) HI 1. 5 (Hub Interface 1. 5) 8 bit wide, 66 MHz clock, QDR, 66 MB/s peak transfer rate ESI: Enterprise System Interface 4 PCIe lanes, 0. 25 GB/s per lane (like the DMI interface, providing 1 GB/s transfer rate in each direction)

3. 4. 2 Evolution of Intel’s low cost oriented multicore DP server platforms (3) Evolution from the Penryn aimed Cranberry Lake DP platform (up to 4 cores) to the Sandy Bridge-EP aimed Romley-EP DP platform (up to 8 cores) Xeon 5300 (Clowertown) 2 x 2 C or Xeon 5400 Xeon 5200 or (Harpertown) 4 C 2 C Core 2 (2 C/2 x 2 C)/ Penryn (2 C/4 C) proc. Sandy Bridge-EN (8 C) Socket B 2 FSB QPI Sandy Bridge-EN (8 C) Socket B 2 DMI 2 E 5100 MCH ESI E 5 -2400 Sandy Bridge–EN 8 C C 600 PCH DDR 2 -533/667 DDR 3 -1600 ICHR 9 Penryn aimed Cranberry Lake DP platform (for up to 4 C) Sandy Bridge-EN (Socket B 2) aimed Romley-EN DP server platform (for up to 8 cores)

3. 4. 3 Evolution of Intel’s performance oriented multicore DP server platforms (1) 3. 4. 3 Evolution of Intel’s performance oriented multicore DP server platforms

3. 4. 3 Evolution of Intel’s performance oriented multicore DP server platforms (2) Evolution from the Pentium 4 Prescott DP aimed DP platform (up to 2 cores) to the Core 2 aimed Bensley DP platform (up to 4 cores) Xeon 5400 Xeon 5200 Xeon 5000 Xeon 5100 Xeon 5300 / / / (Harpertown) (Dempsey) (Woodcrest) (Clowertown) 2 x 2 C 2 C 2 x 1 C 2 C 2 x 2 C Xeon DP 2. 8 /Paxville DP) 90 nm Pentium 4 Prescott DP (2 x 1 C) FSB 65 nm Pentium 4 Prescott DP (2 C)/ Core 2 (2 C/2*2 C) FSB E 7520 MCH E 5000 MCH HI 1. 5 ICH 5 R/ 6300 ESB IOH 65 nm Pentium 4 Prescott DP (2 x 1 C)/ Core 2 (2 C/2*2 C) ESI DDR-266/333 DDR 2 -400 90 nm Pentium 4 Prescott DP aimed DP server platform (for up to 2 C) HI 1. 5 (Hub Interface 1. 5) 8 bit wide, 66 MHz clock, QDR, 66 MB/s peak transfer rate 631*ESB/ 632*ESB IOH FB-DIMM w/DDR 2 -533 Core 2 aimed Bensley DP server platform (for up to 4 C) ESI: Enterprise System Interface 4 PCIe lanes, 0. 25 GB/s per lane (like the DMI interface, providing 1 GB/s transfer rate in each direction)

3. 4. 3 Evolution of Intel’s performance oriented multicore DP server platforms (3) Evolution from the Core 2 aimed Bensley DP platform (up to 4 cores) to the Nehalem-EP aimed Tylersburg-EP DP platform (up to 6 cores) Nehalem-EP (4 C) Westmere-EP (6 C) QPI QPI 65 nm Pentium 4 Prescott DP (2 C)/ Core 2 (2 C/2*2 C) DDR 3 -1333 65 nm Pentium 4 Prescott DP (2 C)/ Core 2 (2 C/2*2 C) FSB 55 xx IOH 1 ESI FBDIMM w/DDR 2 -533 5000 MCH ME DDR 3 -1333 CLink ICH 9/ICH 10 Nehalem-EP aimed Tylersburg-EP DP server platform with a single IOH (for up to 6 C) ESI Nehalem-EP (4 C) Westmere-EP (6 C) 631*ESB/ 632*ESB IOH QPI QPI 65 nm Core 2 aimed high performance Bensley DP server platform (for up to 4 C) DDR 3 -1333 55 xx IOH 1 ESI QPI ME Nehalem-EP (4 C) Westmere-EP (6 C) 55 xx IOH 1 DDR 3 -1333 CLink ICH 9/ICH 10 ME: Management Engine chipset with PCI 2. 0 1 First Nehalem-EP aimed Tylersburg-EP DP server platform with dual IOHs (for up to 6 C)

3. 4. 3 Evolution of Intel’s performance oriented multicore DP server platforms (4) Basic system architecture of the Sandy Bridge-EN and -EP aimed Romley-EN and –EP DP server platforms Xeon 55 xx / (Gainestown) Nehalem –EP (4 C) Westmere-EP (6 C) QPI Xeon 56 xx (Gulftown) Nehalem-EP aimed Tylersburg-EP DP server platform (for up to 6 cores) Nehalem-EP (4 C) Westmere-EP (6 C) DMI DDR 3 -1333 34 xx PCH E 5 -2600 Sandy Bridge–EP 8 C Sandy Bridge-EP (8 C) Socket R DDR 3 -1333 ME E 5 -2600 Sandy Bridge-EP 8 C QPI 1. 1 Sandy Bridge-EP (8 C) Socket R DMI 2 C 600 PCH DDR 3 -1600 Sandy Bridge-EP (Socket R) aimed Romley-EP DP server platform (for up to 8 cores) (LGA 2011)

3. 4. 3 Evolution of Intel’s performance oriented multicore DP server platforms (5) Contrasting the Nehalem-EP aimed Tylersburg-EP DP platform (up to 6 cores) to the Nehalem-EX aimed very high performance scalable Boxboro-EX DP platform (up to 10 cores) Xeon 5600 Xeon 5500 or (Gulftown) (Gainestown) Nehalem –EP (4 C) Westmere-EP (6 C) QPI Nehalem –EP (4 C) Westmere-EP (6 C) ESI DDR 3 -1333 34 xx PCH DDR 3 -1333 ME Nehalem-EP aimed Tylersburg-EP DP server platform (for up to 6 cores) Xeon 6500 (Nehalem-EX) (Becton) SMB SMB Nehalem-EX (8 C) Westmere-EX (10 C) SMB SMI links Xeon E 7 -2800 or (Westmere-EX) QPI Nehalem-EX (8 C) Westmere-EX (10 C) QPI DDR 3 -1067 7500 IOH ESI ICH 10 SMB SMB SMI links DDR 3 -1067 ME SMI: Serial link between the processor and the SMB: Scalable Memory Buffer with Parallel/serial conversion Nehalem-EX aimed Boxboro-EX scalable DP server platform (for up to 10 cores)

3. 5. MP server platforms • 3. 5. 1. Design space of the basic architecture of MP server platforms • 3. 5. 2. Evolution of Intel’s multicore MP server platforms • 3. 5. 3. Evolution of AMD’s multicore MP server platforms

3. 5 MP server platforms 3. 5. 1 Design space of the basic architecture of MP server platforms

3. 5. 1 Design space of the basic architecture of MP server platforms (1) P P P . . MCH . . . ICH Pentium 4 MP 1 C (2004) P P P P P . . MCH. . ICH P P . . MCH P . . ICH . . Serial links attach FB-Di. MMs Parallel channels attach DIMMs P Quad FSBs . . ICH Core 2/Penryn up to 6 C P P P P P S/P . . MCH . . . . ICH. . 90 nm Pentium 4 MP 2 x 1 C . . S/P . . ICH . . S/P . . . ICH P. . S/P. . MCH P . . S/P P . . . Serial links attach. S/P conv. w/ par. chan. Attaching memory via parallel channels P Dual FSBs . . Attaching memory via serial links Single FSB . . Layout of the interconnection MP SMP platforms . .

3. 5. 1 Design space of the basic architecture of MP server platforms (2) . . . S/P P . . S/P . . . . S/P . . P P . . S/P . . . . S/P S/P . . . P P . . S/P P . . . . P . . Serial links attach FB-Di. MMs . . . Serial links attach. S/P conv. w/ par. chan. . . . Attaching memory via serial links . . . . P . . S/P . . . . AMD Direct Connect Architecture 2. 0 (2010) . . P . . AMD Direct Connect Architecture 1. 0 (2003) P P . . Parallel channels attach DIMMs . . Attaching memory via parallel channels . . S/P P . . . P . . Inter proc. BW . . . Layout of the interconnection . . Fully connected mesh Mem. BW Partially connected mesh MP NUMA platforms . . Nehalem-EX/Westmere up to 10 C (2010/11)

3. 5. 1 Design space of the basic architecture of MP server platforms (3) Scheme of attaching and interconnecting MP processors Single FSB Serial links attach. Serial links Parallel channels S/P converters attach FB-Di. MMs attach DIMMs w/ par. chan. Attaching memory via parallel channels Attaching memory via serial links Layout of the interconnection SMP Dual FSBs NUMA Quad FSBs Pentium 4 MP 1 C (2004) (Not named) Part. conn. mesh Fully conn. mesh AMD DCA 1. 0 (2003) AMD DCA 2. 0 (2010) Core 2/Penryn up to 6 C (2006/2007) Caneland 90 nm Pentium 4 MP 2 x 1 C (2006) (Truland) No. of memory channels Nehalem-EX/ Westmere up to 10 C (2010/11) (Boxboro-EX) Interproc. bandwidth No. of memory channels Evolution of Intel’s MP platforms (Overview)

3. 5. 2 Evolution of Intel’s multicore MP server platforms (1) 3. 5. 2 Evolution of Intel’s multicore MP server platforms

3. 5. 2 Evolution of Intel’s multicore MP server platforms (2) Evolution from the first generation MP servers supporting SC processors to the 90 nm Pentium 4 Prescott MP aimed Truland MP server platform (supporting up to 2 cores) Xeon MP Xeon 7000 Xeon 7100 / / (Potomac) 1 C (Paxville MP) 2 x 1 C (Tulsa) 2 C Xeon MP 1 SC Pentium 4 Xeon MP 1 C/2 x 1 C FSB XMB Preceding NBs XMB 85001/8501 XMB E. g. DDR-200/266 E. g. HI 1. 5 Pentium 4 Xeon MP 1 C/2 x 1 C E. g. DDR-200/266 HI 1. 5 DDR-266/333 DDR 2 -400 Preceding ICH XMB ICH 5 DDR-266/333 DDR 2 -400 HI 1. 5 266 MB/s Previous Pentium 4 MP aimed MP server platform (for single core processors) 90 nm Pentium 4 Prescott MP aimed Truland MP server platform (for up to 2 C)

3. 5. 2 Evolution of Intel’s multicore MP server platforms (3) Evolution from the 90 nm Pentium 4 Prescott MP aimed Truland MP platform (up to 2 cores) to the Core 2 aimed Caneland MP platform (up to 6 cores) Xeon MP Xeon 7000 Xeon 7100 / / (Potomac) 1 C (Paxville MP) 2 x 1 C (Tulsa) 2 C Pentium 4 Xeon MP 1 C/2 x 1 C FSB XMB Xeon 7200 Xeon 7300 Xeon 7400 / / (Tigerton DC) 1 x 2 C (Tigerton QC) 2 x 2 C (Dunnington 6 C) Core 2 (2 C/2 x 2 C) Penryn (6 C) FSB 4 channels up to 8 DIMMs /channel XMB 85001/8501 XMB 7300 XMB ESI HI 1. 5 DDR-266/333 DDR 2 -400 ICH 5 DDR-266/333 DDR 2 -400 90 nm Pentium 4 Prescott MP aimed Truland MP server platform (for up to 2 C) HI 1. 5 (Hub Interface 1. 5) 8 bit wide, 66 MHz clock, QDR, 266 MB/s peak transfer rate 1 631 x. ESB/ 632 x. ESB FB-DIMM DDR 2 -533/667 Core 2 aimed Caneland MP server platform (for up to 6 C) ESI: Enterprise System Interface 4 PCIe lanes, 0. 25 GB/s per lane (like the DMI interface, providing 1 GB/s transfer rate in each direction) The E 8500 MCH supports an FSB of 667 MT/s and consequently only the SC Xeon MP (Potomac)

3. 5. 2 Evolution of Intel’s multicore MP server platforms (4) Evolution to the Nehalem-EX aimed Boxboro-EX MP platform (that supports up to 10 cores) (In the basic system architecture we show the single IOH alternative) Xeon 7500 (Nehalem-EX) (Becton) 8 C / Xeon 7 -4800 (Westmere-EX) 10 C SMB SMB Nehalem-EX 8 C Westmere-EX 10 C QPI SMB SMB QPI QPI SMB SMB Nehalem-EX 8 C Westmere-EX 10 C QPI DDR 3 -1067 SMB SMB QPI 2 x 4 SMI channels 7500 IOH ESI ICH 10 ME DDR 3 -1067 SMI: Serial link between the processor and the SMBs SMB: Scalable Memory Buffer Parallel/serial converter ME: Management Engine Nehalem-EX aimed Boxboro-EX MP server platform (for up to 10 C)

3. 5. 3 Evolution of AMD’s multicore MP server platforms (1) 3. 5. 3 Evolution of AMD’s multicore MP server platforms [47] (1) Introduced in the single core K 8 -based Opteron DP/MP servers (AMD 24 x/84 x) (6/2003) Memory: 2 channels DDR-200/333 per processor, 4 DIMMs per channel.

3. 5. 3 Evolution of AMD’s multicore MP server platforms (2) 3. 5. 3 Evolution of AMD’s multicore MP server platforms [47] (2) Introduced in the 2 x 6 core K 10 -based Magny-Course (AMD 6100)(3/2010) Memory: 2 x 2 channels DDR 3 -1333 per processor, 3 DIMMs per channel.

5. References

5. References (1) [1]: Wikipedia: Centrino, http: //en. wikipedia. org/wiki/Centrino [2]: Industry Uniting Around Intel Server Architecture; Platform Initiatives Complement Strong Intel IA-32 and IA-64 Targeted Processor Roadmap for 1999, Business Wire, Febr. 24 1999, http: //www. thefreelibrary. com/Industry+Uniting+Around+Intel+Server +Architecture%3 B+Platform. . . -a 053949226 [3]: Intel Core 2 Duo Processor, http: //www. intel. com/pressroom/kits/core 2 duo/ [4]: Keutzer K. , Malik S. , Newton R. , Rabaey J. , Sangiovanni-Vincentelli A. , System Level Design: Orthogonalization of Concerns and Platform-Based Design, IEEE Transactions on Computer-Aided Design of Circuits and Systems, Vol. 19, No. 12, Dec. 2000, pp. 1 -29. [5]: Krazit T. , Intel Sheds Light on 2005 Desktop Strategy, IDG News Service, Dec. 07 2004, http: //pcworld. about. net/news/Dec 072004 id 118866. htm [6]: Perich D. , Intel Volume platforms Technology Leadership, Presentation at HP World 2004, http: //98. 190. 245. 141: 8080/Proceed/HPW 04 CD/papers/4194. pdf [7] Powerful New Intel Server Platforms Feature Array Of Enterprise-Class Innovations. Intel’s Press release, Aug. 2, 2004 , http: //www. intel. com/pressroom/archive/releases/20040802 comp. htm [8]: Smith S. , Multi-Core Briefing, IDF Spring 2005, San Francisco, Press presentation, March 1 2005, http: //www. silentpcreview. com/article 224 -page 2 [9]: An Introduction to the Intel Quick. Path Interconnect, Jan. 2009, http: //www. intel. com/ content/dam/doc/white-paper/quick-path-interconnect-introduction-paper. pdf [10]: Davis L. PCI Express Bus, http: //www. interfacebus. com/PCI-Express-Bus-PCIe-Description. html

5. References (2) [11]: Ng P. K. , “High End Desktop Platform Design Overview for the Next Generation Intel Microarchitecture (Nehalem) Processor, ” IDF Taipei, TDPS 001, 2008, http: //intel. wingateweb. com/taiwan 08/published/sessions/TDPS 001/FA 08%20 IDFTaipei_TDPS 001_100. pdf [12]: Computing DRAM, Samsung. com, http: //www. samsung. com/global/business/semiconductor /products/dram/Products_Computing. DRAM. html [13]: Samsung’s Green DDR 3 – Solution 3, 20 nm class 1. 35 V, Sept. 2011, http: //www. samsung. com/global/business/semiconductor/Greenmemory/Downloads/ Documents/downloads/green_ddr 3_2011. pdf [14]: DDR SDRAM Registered DIMM Design Specification, JEDEC Standard No. 21 -C, Page 4. 20. 4 -1, Jan. 2002, http: //www. jedec. org [15]: Datasheet, http: //download. micron. com/pdf/datasheets/modules/sdram/ SD 9 C 16_32 x 72. pdf [16]: Solanki V. , „Design Guide Lines for Registered DDR DIMM Module, ” Application Note AN 37, Pericom, Nov. 2001, http: //www. pericom. com/pdf/applications/AN 037. pdf [17]: Fisher S. , “Technical Overview of the 45 nm Next Generation Intel Core Microarchitecture (Penryn), ” IDF 2007, ITPS 001, http: //isdlibrary. intel-dispatch. com/isd/89/45 nm. pdf [18]: Razin A. , Core, Nehalem, Gesher. Intel: New Architecture Every Two Years, Xbit Laboratories, 04/28/2006, http: //www. xbitlabs. com/news/cpu/display/20060428162855. html [19]: Haas, J. & Vogt P. , Fully buffered DIMM Technology Moves Enterprise Platforms to the Next Level, ” Technology Intel Magazine, March 2005, pp. 1 -7

5. References (3) [20]: „Introducing FB-DIMM Memory: Birth of Serial RAM? , ” PCStats, Dec. 23, 2005, http: //www. pcstats. com/articleview. cfm? articleid=1812&page=1 [21]: Mc. Tague M. & David H. , „ Fully Buffered DIMM (FB-DIMM) Design Considerations, ” Febr. 18, 2004, Intel Developer Forum, http: //www. idt. com/content/OSA-S 009. pdf [22]: Ganesh B. , Jaleel A. , Wang D. , Jacob B. , Fully-Buffered DIMM Memory Architectures: Understanding Mechanisms, Overheads and Scaling, 2007, [23]: DRAM Pricing – A White Paper, Tachyon Semiconductors, http: //www. tachyonsemi. com/about/papers/DRAM%Pricing. pdf [24]: Detecting Memory Bandwidth Saturation in Threaded Applications, Intel, March 2 2010, http: //software. intel. com/en-us/articles/detecting-memory-bandwidth-saturation-inthreaded-applications/ [25]: Mc. Calpin J. D. , STREAM Memory Bandwidth, July 21 2011, http: //www. cs. virginia. edu/stream/by_date/Bandwidth. html [26]: Rogers B. , Krishna A. , Bell G. , Vu K. , Jiang X. , Solihin Y. , Scaling the Bandwidth Wall: Challenges in and Avenues for CMP Scaling, ISCA 2009, Vol. 37, Issue 1, pp. 371 -382 [27]: Vogt P. , Fully Buffered DIMM (FB-DIMM) Server Memory Architecture: Capacity, Performance, Reliability, and Longevity, Febr. 18 2004, http: //www. idt. com/content/OSA_S 008_FB-DIMM-Arch. pdf [28]: Wikipedia: Intel X 58, 2011, http: //en. wikipedia. org/wiki/Intel_X 58

5. References (4) [29]: Sharma D. D. , Intel 5520 Chipset: An I/O Hub Chipset for Server, Workstation, and High End Desktop, Hotchips 2009, http: //www. hotchips. org/archives/hc 21/2_mon/ HC 21. 24. 200. I-O-Epub/HC 21. 24. 230. Das. Sharma-Intel-5520 -Chipset. pdf [30]: DDR 2 SDRAM FBDIMM, Micron Technology, 2005, http: //download. micron. com/pdf/datasheets/modules/ddr 2/HTF 18 C 64_128_256 x 72 F. pdf [31]: Wikipedia: Fully Buffered DIMM, 2011, http: //en. wikipedia. org/wiki/Fully_Buffered_DIMM [32]: Intel E 8500 Chipset e. Xternal Memory Bridge (XMB) Datasheet, March 2005, http: //www. intel. com/content/dam/doc/datasheet/e 8500 -chipset-external-memorybridge-datasheet. pdf [33]: Intel 7500/7512 Scalable Memory Buffer Datasheet, April 2011, http: //www. intel. com/content/dam/doc/datasheet/7500 -7512 -scalable-memorybuffer-datasheet. pdf [34]: AMD Unveils Forward-Looking Technology Innovation To Extend Memory Footprint for Server Computing, July 25 2007, http: //www. amd. com/us/press-releases/Pages/Press_Release_118446. aspx [35]: Chiappetta M. , More AMD G 3 MX Details Emerge, Aug. 22 2007, Hot Hardware, http: //hothardware. com/News/More-AMD-G 3 MX-Details-Emerge/ [36]: Goto S. H. , The following server platforms AMD, May 20 2008, PC Watch, http: //pc. watch. impress. co. jp/docs/2008/0520/kaigai 440. htm [37]: Wikipedia: Socket G 3 Memory Extender, 2011, http: //en. wikipedia. org/wiki/Socket_G 3_Memory_Extender

5. References (5) [38]: Interfacing to DDR SDRAM with Cool. Runner-II CPLDs, Application Note XAPP 384, Febr. 2003, XILINC inc. [39]: Ahn J. -H. , „Memory Design Overview, ” March 2007, Hynix, http: //netro. ajou. ac. kr/~jungyol/memory 2. pdf [40]: Ebeling C. , Koontz T. , Krueger R. , „System Clock Management Simplified with Virtex-II Pro FPGAs”, WP 190, Febr. 25 2003, Xilinx, http: //www. xilinx. com/support/documentation/white_papers/wp 190. pdf [41]: Kirstein B. , „Practical timing analysis for 100 -MHz digital design, ”, EDN, Aug. 8, 2002, www. edn. com [42]: Jacob B. , Ng S. W. , Wang D. T. , Memory Systems, Elsevier, 2008 [43]: Allan G. , „The outlook for DRAMs in consumer electronics”, EETIMES Europe Online, 01/12/2007, http: //eetimes. eu/show. Article. jhtml? article. ID=196901366&query. Text =calibrated [44]: Ebeling C. , Koontz T. , Krueger R. , „System Clock Management Simplified with Virtex-II Pro FPGAs”, WP 190, Febr. 25 2003, Xilinx, http: //www. xilinx. com/support/documentation/white_papers/wp 190. pdf [45]: Memory technology evolution: an overview of system memory technologies, Technology brief, 9 th edition, HP, Dec. 2010, http: //h 20000. www 2. hp. com/bc/docs/support/Support. Manual/c 00256987. pdf

5. References (6) [46]: Kane L. , Nguyen H. , Take the Lead with Jasper Forest, the Future Intel Xeon Processor for Embedded and Storage, IDF 2009, July 27 2009, ftp: //download. intel. com/embedded/processor/prez/SF 09_EMBS 001_100. pdf [47]: The AMD Opteron™ 6000 Series Platform: More Cores, More Memory, Better Value, March 29 2010, http: //www. slideshare. net/AMDUnprocessed/amd-opteron-6000 -series -platform-press-presentation-final-3564470 [48]: Memory Module Picture 2007, Simmtester, Febr. 21 2007, http: //www. simmtester. com/page/news/showpubnews. asp? num=150