IPMI New MMC firmware IN 2 P 3

IPMI: New MMC firmware IN 2 P 3: DAQ working group (19/06/15) PH-ESE-BE collaboration Vincent Bobillier, Stefan Haas, Markus Joos, Julian Mendez, Sylvain Mico and Francois Vasey 19/06/2015 IN 2 P 3: DAQ working group (06/19) 1

IPMI: New MMC firmware Outline q PH-ESE-BE: x. TCA evaluation project q Hardware Platform Management overview q CERN MMC overview q New MMC firmware q Summary 19/06/2015 IN 2 P 3: DAQ working group (06/19) 2

IPMI: New MMC firmware Outline q PH-ESE-BE: x. TCA evaluation project q Hardware Platform Management overview q CERN MMC overview q New MMC firmware q Summary 19/06/2015 IN 2 P 3: DAQ working group (06/19) 3

IPMI: New MMC firmware PH-ESE-BE: x. TCA evaluation project q Major upgrades of the LHC experiments are foreseen over >10 years § Aligned with LHC upgrade long shutdowns: 2013/14, 2018, 2023 q Off-detector electronics of the LHC experiments mostly based on VME § Working very reliably § “Old” technology and doubts about long-term availability q Experiments planning to use MTCA & ATCA for upgrades of their back-end electronics § MTCA (and ATCA): CMS § ATCA: ATLAS q x. TCA advantages § § § Choice of form factors Backplane bandwidth and protocols Cooling and power supply Redundancy (PSU, cooling, Mgmt) Infrastructure monitoring features Interoperability q MTCA and ATCA developments already on-going at CERN and collaborating institutes q Evaluation project started in 2011 19/06/2015 IN 2 P 3: DAQ working group (06/19) 4

IPMI: New MMC firmware PH-ESE-BE: x. TCA evaluation project (Micro. TCA) Micro. TCA Crates Schroff MTCA. 4 11890 Schroff MTCA. 4 + AC/DC CM 100 ELMA MTCA. 4 043 -012 ELMA MTCA. 4 045 -821 Power Modules (PM) NAT DC 840 W Vadatech UTC 010 792 W Vadatech UTC 020 936 W Wiener AC/DC 1000 W Telkoor AC/DC 600 W NAT AC/DC AC 600 W AMCs ELMA Load Board 19/06/2015 Processor Kontron AM 5030 ESD ADIO 24 Processor CCT AM 310 IN 2 P 3: DAQ working group (06/19) Vadatech UTC 001 NAT Kontron AM 4904 5

IPMI: New MMC firmware PH-ESE-BE: x. TCA evaluation project (Advance. TCA) Commercial ATCA Crates Schroff 14 -slot 13 U ATCA 11596 -150 ASIS 14 -slot 13 U ATCA 144 D 422 Comtel 14 slot ELMA 14 slot Schroff 2 -slot ATCA 11990 -705 Custom ATCA Crate ASIS Vertical cooling 19/06/2015 IN 2 P 3: DAQ working group (06/19) 6

IPMI: New MMC firmware PH-ESE-BE: x. TCA evaluation project (Summary) q Micro. TCA § § § Equipment evaluated Specifications written for DC/DC power modules and shelves Price inquiry finished Pre-series of 3 units for final qualification purchased Once approved, contract can be used to simplify purchase order process (CERN) q Advance. TCA § Equipment evaluated § Cooling simulation of custom shelves is ongoing (vertical cooling) § Cooling measurement of custom shelves is ongoing (vertical cooling) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 7

IPMI: New MMC firmware Outline q PH-ESE-BE: x. TCA evaluation project q Hardware Platform Management overview q CERN MMC overview q New MMC firmware q Summary 19/06/2015 IN 2 P 3: DAQ working group (06/19) 8

IPMI: New MMC firmware Hardware Platform Management - Introduction q Purpose: § Monitor (temperatures, voltages, current …) § Control (power management …) § Assure properation (verify compatibility …) q Management modules: § Module Management Controller (MMC) – for AMC management § IPM Controller (IPMC) / Carrier IPM Controller (CIPMC) – ATCA/ATCA carrier management § Shelf Manager (Sh. MC) – for ATCA shelf and Micro. TCA MCHs q Inter-module communication (IPMI) § Intelligent Platform Management Interface § Base specification was announced September 1998 § Hardware Platform Management is based on version 1. 5 19/06/2015 IN 2 P 3: DAQ working group (06/19) 9

IPMI: New MMC firmware Hardware Platform Management - ATCA 19/06/2015 IN 2 P 3: DAQ working group (06/19) 10

IPMI: New MMC firmware Hardware Platform Management - Micro. TCA 19/06/2015 IN 2 P 3: DAQ working group (06/19) 11

IPMI: New MMC firmware MMC Role and features q MMC Role: § Activating / De-activating an AMC card § Providing information about the AMC card o Maximum current o Ports configuration o Clock configuration § Sending alert events (sensor exceed threshold) § Executing IPMI request DC/DC q Features § Power management § Sensor monitoring § Clock and ports management 19/06/2015 Temp. sensors current sensors voltage sensors IN 2 P 3: DAQ working group (06/19) MMC PLL Switch EEPROM FPGA 12

IPMI: New MMC firmware Activation/De-activation state machine § M 0 (not installed) § M 1 (installed) § CM read the FRU information § Current § E-keying § Clock § M 2 (activation requested) § M 3 (activation in progress) § Enable power § E-keying control § Clock configuration § M 4 (active) § M 5 (Deactivation requested) § E-keying de-activation § Clock de-activation M 4 M 3 M 2 M 0 M 1 M 5 M 6 § M 6 (Deactivation in progress) § Disable power 19/06/2015 IN 2 P 3: DAQ working group (06/19) 13

IPMI: New MMC firmware Outline q PH-ESE-BE: x. TCA evaluation project q Hardware Platform Management overview q CERN MMC overview q New MMC firmware q Summary 19/06/2015 IN 2 P 3: DAQ working group (06/19) 14

IPMI: New MMC firmware CERN MMC overview q DESY § Original development § Given to CPPM by Kay Rehlich and Vahan Petrosyan q CPPM § First modification of the source code § Design of a mezzanine card o Can be used as reference design q CERN § Supported by the PH-ESE-BE group q Used in many experiments § CMS: Twin. MUX (CERN) § ATLAS: Argon liquid (LAPP) § LHCb and Alice: AMC 40 (CPPM) o Architecture changed but the AMC 40 will be used for others projects (e. g. : Gamahadron, IN 2 P 3) § IN 2 P 3 projects 19/06/2015 IN 2 P 3: DAQ working group (06/19) 15

IPMI: New MMC firmware Previous version status q Non-implemented features § § § FRU information missing (E-keying, clock configuration) Power sequences fixed AMC ports control not implemented Clock configuration not implemented Remote upgrade not supported q Major issues § Activation / De-activation state machine controlled by the MMC § Sensor event generation: Signed threshold stocked in unsigned variable q Polaris tester outcome § Automatic tests o 14 passed o 14 failed 19/06/2015 IN 2 P 3: DAQ working group (06/19) 16

IPMI: New MMC firmware Motivations q MMC issues § FRU information missing o Did not work with Pigeon Point IPMC § Activation/De-activation state machine is managed by the MMC itself o One patch should be written to be compliant with Vadatech o Automatically goes from M 4 to M 6 with Pigeon Point § User customization was too hard o User have to read the Hardware Management Platform of the standard o No documentation q The CERN MMC is used in many experiments and should be fully compliant with the standard 19/06/2015 IN 2 P 3: DAQ working group (06/19) 17

IPMI: New MMC firmware Outline q PH-ESE-BE: x. TCA evaluation project q Hardware Platform Management overview q CERN MMC overview q New MMC firmware q Summary 19/06/2015 IN 2 P 3: DAQ working group (06/19) 18

IPMI: New MMC firmware E-keying (new) q Control § GET/SET AMC PORT STATE commands (Executed after 12 V enabling) § Interfaces available for control: I 2 C, GPIO q Hardware § Buffers § Switches § FPGA q Configuration described in FRU info (P 2 P connectivity record) #define AMC_POINT_TO_POINT_RECORD_CNT 3 #define AMC_POINT_TO_POINT_RECORD_LIST GENERIC_POINT_TO_POINT_RECORD(0, PORT(0), ETHERNET, BASE_1 G_BX, EXACT_MATCHES) GENERIC_POINT_TO_POINT_RECORD(1, PORT(4), PCIE, GEN 1_NO_SSC, MATCHES_10) GENERIC_POINT_TO_POINT_RECORD(2, PORT(5), PCIE, GEN 1_NO_SSC, MATCHES_10) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 19

IPMI: New MMC firmware Clock configuration (new) q Control § GET/SET CLOCK STATE commands (Executed after 12 V enabling) § Interfaces available for control: I 2 C, GPIO q Hardware § Buffers § Switches § FPGA q Configuration described in FRU info (Clock configuration record) #define AMC_CLOCK_CONFIGURATION_LIST DIRECT_CLOCK_CONNECTION(TCLKA, CIPMC, NO_PLL, RECEIVER, UNSPEC_FAMILY, 0, KHz(8), KHz(7), KHz(9)) DIRECT_CLOCK_CONNECTION(TCLKB, CIPMC, PLL, RECEIVER, UNSPEC_FAMILY, 0, KHz(2), KHz(1), KHz(3)) DIRECT_CLOCK_CONNECTION(FCLKA, CIPMC, PLL, RECEIVER, UNSPEC_FAMILY, 0, MHz(12. 8), MHz(11), MHz(13)) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 20

IPMI: New MMC firmware Additional improvements q Bug fixed § State machine removed (controlled by the carrier manager) § Sensor event generation (negative threshold) q Custom power sequences § Power ON/OFF, Reboot … § Easily defined in a configuration file q LED description § No C functions must be written to define user LED q FRU information § FRU information can be easily customized 19/06/2015 IN 2 P 3: DAQ working group (06/19) 21

IPMI: New MMC firmware Previous architecture Hardware init. Power management Sensors management LED control IPMI interface user_code. c q q q Core User Hardware custom initialization User led control Sensor definition and monitoring FRU information OEM commands 19/06/2015 IN 2 P 3: DAQ working group (06/19) 22

IPMI: New MMC firmware New architecture Hardware init. config. h q q q q Power management fru_info. h Sensors management sensors. h LED control sensors. c IPMI interface Core user_code. c User ekeying. c User led definition Custom hardware definition Power sequences customization FRU information Sensor definition Sensor monitoring AMC port configuration User specific IPMI commands 19/06/2015 IN 2 P 3: DAQ working group (06/19) 23

IPMI: New MMC firmware Module Management Controller – New release q Simplification of the user customization (E. g. : FRU information) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 24

IPMI: New MMC firmware Module Management Controller – New release q Simplification of the user customization (E. g. : LED description) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 25

IPMI: New MMC firmware Module Management Controller – New release q Simplification of the user customization (E. g. : Power sequences) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 26

IPMI: New MMC firmware Module Management Controller - HPM. 1 support q HPM. 1 overview § PICMG Standard § Firmware remote upgrade features of IPM Controller (E. g: MMC) q MMC programming evolution MMC Application HPM Boot loader 19/06/2015 0 0111 1 0000 1110 10110100 1 0010 00 0011 0110110001 1 00 1100 10101100 100101 1000 11010101 1001 IPMI I 2 C Eth. 0 0111 1 0000 1110 10110100 1 0010 00 0011 0110110001 1 00 1100 10101100 100101 01 00 10 110101 1001 IN 2 P 3: DAQ working group (06/19) 27

IPMI: New MMC firmware Module Management Controller - Additional tools q FRU Editor § Generates FRU binary from header file § Write new FRU in the MMC EEPROM q HPM Downloader § Generates HPM image from. hex file § Download the binary in the MMC flash memory q MTCALib § C library for m. TCA (RMCP) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 28

IPMI: New MMC firmware MMC test with ATCA q MMC tested successfully with the Pigeon Point starter kit Shelf Manager RTM Emulation IPMC AMC 1 Emulation AMC 2 (CERN MMC) q Adapter card has been designed to adapt connector to DIM-DDR 3 VLP for factor. § Goal: adapt the PP solution to an existing ATCA carrier (MMC OK as well) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 29

IPMI: New MMC firmware MMC test with ATCA q Successful operation of the CERN MMC in an existing ATCA carrier (LAPP’s IPMC Test board carrier) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 30

IPMI: New MMC firmware Status q MMC source code was re-written § About 90% of the code was modified q HPM. 1 is fully supported § New project: boot loader program § Creation of a tool to program the MMC with HPM. 1 q Improvement of the IPMI compliance § Polaris tester outcome o 26 passed o 2 failed • No point to point connectivity information in the MCH (not a MMC issue) • IPMI command “set blue led” not filtered by the MCH (not a MMC issue) q Tested with NAT, Vadatech and SAMWAY MCHs as well as with Pigeon Point IPMC q User guide documentation 19/06/2015 IN 2 P 3: DAQ working group (06/19) 31

IPMI: New MMC firmware Outline q PH-ESE-BE: x. TCA evaluation project q Hardware Platform Management overview q CERN MMC overview q New MMC firmware q Summary 19/06/2015 IN 2 P 3: DAQ working group (06/19) 32

IPMI: New MMC firmware Summary q New MMC release will be available soon § Foreseen for next week § Licence GPL q Tested in some configurations § Different MCHs (NAT, Vadatech ans SAMWAY) § ATCA carrier (Pigeon Point) § Different AMC cards: AMC Load. Board, Twin. MUX (CMS), AMC 40, GLIB q IPMI compliance improved § Polaris tester outcome (almost 100% compliance) q HPM. 1 support for remote upgrades 19/06/2015 IN 2 P 3: DAQ working group (06/19) 33

IPMI: New MMC firmware Thank you for your attention ! Any question ? 19/06/2015 IN 2 P 3: DAQ working group (06/19) 34

IPMI: New MMC firmware Backup slides 19/06/2015 IN 2 P 3: DAQ working group (06/19) 35

IPMI: New MMC firmware PH-ESE-BE: x. TCA evaluation project (Micro. TCA) MTCA Shelves (up to 40 pcs) Manufacturer ELMA Schroff ü ü Pros Cooling efficiency except for slot 12 and RTM Layout with MCH in middle Cooling homogeneity across slots Mechanical robustness Remote support quality To be done Backplane quality meas. Compliant to spec Vadatech DC power modules (up to 100 pcs) Manufacturer NAT (DC 840, different sources) Output power 840 W Compliant to spec Pros Vadatech (UTC-010) 750 W ü Efficiency 100% IPMI compliant Vadatech (UTC-020) 890 W ü Std compliant MTBF Serial interface Evaluation reports available: https: //espace. cern. ch/ph-dep-ESE-BE-u. TCAEvaluation. Project/default. aspx 19/06/2015 IN 2 P 3: DAQ working group (06/19) 36

IPMI: New MMC firmware Module Management Controller - Role and feature q MMC Role: § AMC Activation / De-activation and Hardware management M 4 M 3 M 2 M 1 19/06/2015 M 0 § M 0 (not installed) M 5 M 6 IN 2 P 3: DAQ working group (06/19) 37

IPMI: New MMC firmware Module Management Controller - Role and feature q MMC Role: § AMC Activation / De-activation and Hardware management M 4 M 3 M 2 M 1 19/06/2015 M 0 M 5 § M 0 (not installed) § M 1 (installed) • • Handle switch monitoring Read FRU Information • Current limit • E-Keying • Clock configuration M 6 IN 2 P 3: DAQ working group (06/19) 38

IPMI: New MMC firmware Module Management Controller - Role and feature q MMC Role: § AMC Activation / De-activation and Hardware management M 4 M 3 § M 0 (not installed) § M 1 (installed) § M 2 (activation requested) • M 2 M 1 19/06/2015 M 0 CM <-> Shelf Manager M 5 M 6 IN 2 P 3: DAQ working group (06/19) 39

IPMI: New MMC firmware Module Management Controller - Role and feature q MMC Role: § AMC Activation / De-activation and Hardware management M 4 M 3 M 2 M 1 19/06/2015 M 0 M 5 § § M 0 (not installed) M 1 (installed) M 2 (activation requested) M 3 (Activation in progress) • • • Enable AMC port Configure clocks Execute power ON sequence M 6 IN 2 P 3: DAQ working group (06/19) 40

IPMI: New MMC firmware Module Management Controller - Role and feature q MMC Role: § AMC Activation / De-activation and Hardware management M 4 M 3 M 2 M 1 19/06/2015 M 0 M 5 § § § M 0 (not installed) M 1 (installed) M 2 (activation requested) M 3 (activation in progress) M 4 (active) • • • Sensor monitoring Power monitoring Handle switch monitoring M 6 IN 2 P 3: DAQ working group (06/19) 41

IPMI: New MMC firmware Module Management Controller - Role and feature q MMC Role: § AMC Activation / De-activation and Hardware management M 4 M 3 M 2 M 1 19/06/2015 M 0 M 5 § § § M 0 (not installed) M 1 (installed) M 2 (activation requested) M 3 (activation in progress) M 4 (active) M 5 (Deactivation requested) • CM <-> Shelf Manager M 6 IN 2 P 3: DAQ working group (06/19) 42

IPMI: New MMC firmware Module Management Controller - Role and feature q MMC Role: § AMC Activation / De-activation and Hardware management M 4 M 3 M 2 M 0 M 5 § § § § M 0 (not installed) M 1 (installed) M 2 (activation requested) M 3 (activation in progress) M 4 (active) M 5 (Deactivation requested) M 6 (Deactivation in progress) • • M 1 19/06/2015 Disable AMC ports Disable clocks Execute Power OFF sequence Inform CM: deactivation finished M 6 IN 2 P 3: DAQ working group (06/19) 43

IPMI: New MMC firmware Module Management Controller – New release q Simplification of the user code architecture Core • • Hardware initialization AMC management IPMI interpretation Sensor management New user code architecture Previous code architecture • • user_code. c • LED functions • GPIO initialization • Sensor definition • Sensor monitoring • OEM commands user_code. h • User’s macros • Function prototypes • • • 19/06/2015 IN 2 P 3: DAQ working group (06/19) fru_info. h config. h • LED definition • User GPIO initialization • Power sequences ekeying. c • Enable/Disable AMC ports sensor. h • Sensor definition sensor. c • Sensor initialization • Sensor reading user_code. c • OEM commands • Controller specific commands 44

IPMI: New MMC firmware Module Management Controller – Before modifications status q Previous architecture Core • • User Hardware initialization AMC management IPMI interpretation Sensor management • user_code. c / user_code. h • FRU: no interpretation (byte array) • LED: 6 functions for management • Pin init: register value modification • Sensor management: • Reading and check for event Hard user customization 19/06/2015 IN 2 P 3: DAQ working group (06/19) 45

IPMI: New MMC firmware Module Management Controller – New architecture Core • • User Hardware initialization AMC management IPMI interpretation Sensor management • • • fru_info. h config. h • LED definition • User GPIO initialization • Power sequences ekeying. c • Enable/Disable AMC ports sensor. h • Sensor definition sensor. c • Sensor initialization • Sensor reading user_code. c • OEM commands • Controller specific commands Easy user customization => About 90% of the code re-written 19/06/2015 IN 2 P 3: DAQ working group (06/19) 46

IPMI: New MMC firmware Module Management Controller - HPM. 1 support q HPM implementation § Configure the microcontroller to automatically boot from the boot loader 19/06/2015 IN 2 P 3: DAQ working group (06/19) 47

IPMI: New MMC firmware Carrier IPMC – Pigeon Point IPMC q Purpose: Evaluation of a commercial IPMC solution (Pigeon Point IPMC) q Licence: PH-ESE-BE (x. TCA Evaluation Project) § § Licence for use Software and Hardware documentation Starter kit User guide Shelf Manager RTM Emulation IPMC AMC 1 Emulation AMC 2 (CERN MMC) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 48

IPMI: New MMC firmware Carrier IPMC – IPMC Adapter Card q Goal: Evaluation of the Pigeon Point IPMC on a custom ATCA blade featuring the LAPP footprint USB Pigeon Point connectors USB 7 cm < 10 cm CPLD DIM connector (LAPP) 19/06/2015 IN 2 P 3: DAQ working group (06/19) 49

IPMI: New MMC firmware Carrier IPMC – IPMC Adapter Card Hotswap, reset, UART, HA [7. . 0] USB UART to USB IPMB-B / IPMB-A IPMB-B_Enable / IPMB-A_Enable Payload User I/O 35 20 Payload / E-Keying 13 16 SPI: IO config 48 V Monitoring CPLD 4 4 UART to USB Debug IPMB-L AMC 1 IPMB-L AMC 2 So. L Eth. Payload / E-Keying 48 V Monitoring Master JTAG I 2 C buffers JTAG (prog) IPMB-L Ethernet Phy Jumpers 19/06/2015 User I/O 9 AMCs (8+1 RTM) 2 AMCs En Pigeon Point IPMC Connector Buffer IPMB-B / IPMB-A IN 2 P 3: DAQ working group (06/19) User I/O 15 50 L A P P C o n ne c t o r

IPMI: New MMC firmware Carrier IPMC – IPMC Adapter Card q Evaluate Pigeon Point IPMC on a custom ATCA blade featuring the LAPP footprint CPLD Pigeon point connectors CPLD 19/06/2015 IN 2 P 3: DAQ working group (06/19) 51

IPMI: New MMC firmware Pigeon Point IPMC evaluation ✔ q Polaris tester ✔ q Ethernet ✔ q Management of up to 4 AMCs ✔ § §✔ RMCP § HPM. 2 and HPM. 3 ✔ § ✔IPMB-A / IPMB-B q §✔ Communication with the Shelf Manager § HPM. 1 ✔ q Management of the ATCA blade §✔ SDR/FRU information §✔ Sensors monitoring (customization, event …) § Communication through IPMB-L bus Activation/De-activation SDR / FRU information Sensor monitoring (events) HPM. 1 q Intelligent and non-intelligent RTM § Activation/De-activation § E-keying and clock configuration q ATCA e-keying and clock configuration q OEM commands 19/06/2015 IN 2 P 3: DAQ working group (06/19) 52