New ATCA compute node Design for PXD ZhenAn

New ATCA compute node Design for PXD Zhen-An Liu Trig. Lab, IHEP Beijing Feb. 8 2011， 6 th International Workshop on DEPFET Detectors and Applications

Outline • • • 2011 -2 -8 Requirements to CN Approach 1: direct upgrade Approach 2: AMC-based Upgrade Example firmware development Summary 6 th DEPFET Workshop in Bonn 2011 2

Requirements to CN • PXD requirements summarized last workshop in Valencia – Based on Function of PANDA CN – – – 5 FPGAs/CN 8 optical links/CN(3 Gbps) Ethernet links/CN 20 G DDR 2 memory/CN (2 G X 5) Flash Memory – Improvements • 6 Gbps bandwidth per link • 4 GB DDR 2 Memory per FPGA • Approaches 1. Modification from present version • 2 GB DDR 2 -4 GB DDR 2, – Space not possible • 3. 125 ->6. 5 Gbps – Possible with SFP+ • done 2. New design • AMC/Carrier, x. TCA compliant • AMC card PCB received • ATCA carrier under design 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 3

Approach 1: modification from PANDA CN • MODIFICATIONS – I 2 C communication capacitor – Voltage/Current monitor circuit – 3. 125/6. 5 Gbps compliant • Production done – 2 boards assembled with virtex 4 fx 60 -11 for SFP+ • PXD readout – 2 boards assembled with virtex 4 fx 60 -10 for SFP 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 4

• Data flow – • Optical link->DDR 2 SDRAM -> PC Optical link@6. 25 Gbps – • Test of New Boards Tested about 1 hour, estimated BER< 1. 0 x 10^-14 P 2 P transmission on backplane – 2011 -2 -8 Failed to transmit data@6. 25 Gbps, limited by bandwidth of backplane Eye-diagram of backplane transmission Eye-diagram of optical link 6 th DEPFET Workshop in Bonn 2011 5

Evaluation Hardware • 4 CN boards are ready for PXD testbenchs – 2 boards with 6. 25 G optical links – 2 boards with 3. 125 G optical links – All boards with 2 GX 5=10 G DDR 2 memories • PXD evaluation can be done based on these boards – 4 G DDR 2 is not so necessary from Itoh-san – Further development (approach 2) will go No problem foreseen for April conclusion 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 6

Approach 2: AMC-based Upgrade • Compute Node – An universal high performance platform prepared for multiple applications – High availability: ATCA-based (full mesh topology in backplane) – FPGA (now Virtex-4), 5 Virtex 4 FX 60 FPGA, 10 GB DDR 2 SDRAM (2 GB per FPGA), High Bandwidth, 320 MB FLASH (64 MB per FPGA) – It is a prototype of HADES/PANDA • But, not satisfied to BESIII, Belle II PXD and may be others systems in – – – • Memory capacity Input and inter Link rate Timing and control capabilities XTCA for physics compliant (IHEP is a founder of this standard) … It is not good to design one board for all applications – Complexity – High cost (prototype) • XTCA is a good choice for next generation – – 1 ATCA Carrier Board with high bandwidth switch + AMC modules Custom AMC modules for different applications flexibility to reuse and upgrade Flexibility for maintenance 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 7

Development of Carrier Board • The mother board – based on x. TCA – allowing backplane data transmission – 4 AMC connectors – FPGA 0 for interconnection s – IPMC routing – Clock/trigger/ distributions – Power conversions – RTM reservation 2011 -2 -8 • Status 6 th DEPFET Workshop in Bonn 2011 8

The structure of the x. FP card(AMC) • • • 2 x 3 Gbps /FPGA 2 x 2 G DDR 2 1 Gbps ethernet 1 UART 1 IPMC/MMC 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 9

Module Sizes and Width • six types of Module available – A Full-size Module is the most common, allowing up to 23. 25 mm high components – A Mid-size Module allows component heights maxed at 11. 65 to 14. 01 mm – A Compact Module allows only 8. 18 mm. – A special carrier card known as hybrid or cutaway carrier is required to hold one Full-size Module or two Compact-size – Each height is paired with a width, single (74 mmx 180 mm) or double (149 mmx 180 mm), describing how many carrier slots the board fills. A double width card allows more component space, but does not provide any additional power or bandwidth because it only uses a single connector. 2011 -2 -8 From PICMG AMC. 0 R 2. 0 10

Hardware Platform Management • • • Management is performed through IPMI messaging over an onboard IPMB referenced throughout this specification as IPMB-L. Each Module has a unique IPMB-L address derived from its Geographic Address. Carrier provides ways to isolate the IPMB-L connection to each Module. MMC: Module Management Controller From PICMG AMC. 0 R 2. 0 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 11

PCB Layout of the x. FP(AMC) • PCB in hand • Soldering to be done soon with XC 5 VLX 50 T • Test to be done in Feb. 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 12

PCB Layout of the x. FP(AMC) • PCB both sides cont. 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 13

PPC version of AMC • XC 5 VFX 70 T with PPC – Order soon after back to work • PCB compatible • No problem in 2 months(Soeren’s concern) 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 14

Firmware development • Communication with DHH – Format not decided yet – Further discussion with Igor – Possible based on BES 3/Belle 2 link? – Practice successful 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 15

Example Firmware Development • Open source Linux • Linux boot and reconfiguration online • Data flow 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 16

Demonstration System Event Gen. . cxx Software Reconstruction. cxx NFS Data token. C macro NFS Detector data. hex　　 framework COPPER Data analysis. C macro FINESSE optical link@2 Gbps PMC CPU 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 Compute. Node 17

Plan for April Evaluation • • New CN with 6 Gbps links is ready for use New design w AMC card will be ready end Feb. Carrier board ready in April MTCA shelf expected April SHOULD BE NO PROBLEM ! 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 18

Summary • Test results show that new ATCA CN boards work well, Optical link @6. 25 Gbps is stable • Example firmware for data transmission and readout developed • AMC-based upgrade is ongoing smoothly Should be no problem for April evaluation! 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 19

Thanks for you attention !

backups 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 21

Power Distribution • Payload power (AMC connector) – +12 V(10. 8 V to 13. 2 V) – Max. 80 W, 7. 4 A(@10. 8 V) • Management power – +3. 3 V +/- 0. 3 V – Max. 150 m. A, At least 165 m. A(Carrier) From PICMG AMC. 0 R 2. 0 BUT • ATCA power supply: 200 W/slot x 80% = 160 W – Only ~35 W for each module if there are 4 AMC modules – ~55 W for each module by increasing power to 300 W/slot ( PIM 300 + power module) – Cooling challenges for 300 W+ power supply 2011 -2 -8 6 th DEPFET Workshop in Bonn 2011 22