A RadHard Slow Control Network for the CMS

A Rad-Hard Slow Control Network for the CMS Central Tracker A. Marchioro, G. Cervelli, F. Faccio, K. Kloukinas, C. Ljuslin, P. Moreira, E. Murer, C. Paillard, F. Vasey CERN-EP A. Marchioro / EP

Outline u Motivations and Requirements u Architecture u Network and system components: – Token-Ring Network – Local Monitoring – Timing and Trigger Distribution u Network: High level protocol u Project Status A. Marchioro / EP 2

Motivations u Requirements and Architecture no different from traditional slow control system u Merging with timing distribution: unusual ! u Commercial rad-hard control components exist, but not suitable for application and expensive u Special concern: reliability and SEU robustness u Looked at commercial networks: – – – Ethernet Mil 1553 CANbus IBM Token-ring JTAG Other Field-buses … Either too complicated, or don’t work in magnetic fields or unsuitable or too expensive, or… A. Marchioro / EP 3

Requirements u u u u u Must carry data and 40 MHz clock for system synchronization Bi-directional 150 m optical + ~10 m copper Compatible with opto-electronics used for data read-out Redundancy Low digital noise Easy interfaces on the FE ASICs ~120, 000 FE ASICs, ~20, 000 MCMs, ~2, 000 controllers, ~200 links, ~50 control boards in Countingroom Low cost (node price < 100 $ ) A. Marchioro / EP 4

Outline u Motivations and Requirements u Architecture u Network and system components: – Token-Ring Network – Local Monitoring – Timing and Trigger Distribution u Network: High level protocol u Project Status A. Marchioro / EP 5

Det CMS Tracker R-O: General Architecture APVs A/D Memory I V to DAQ A/D I 2 C Front-end Module DCU PLL-Delay FED TTCrx CLK & T 1 Temp CLK - T 1 CCU TTCrx CCU FEC cntrl CCU Data Path Control Path PCI Intfc CCU FEC Control Module A. Marchioro / EP 6

Logical View of Control Ring Simple JTAG, i. e. JTAG sequences setup in SW in FEC Channel Specific Protocols Channels Node Controller Node CCU_8 FEC Channel Controller I 2 C - MBUS etc. Channel Controller I 2 C MBUS PIO JTAG O/E PCI OA Node CCU_1 Ring Protocol A. Marchioro / EP 7

Full control ring Wires between CCU and FE modules CLK+T 1 1 * diff pair (LVDS) I 2 C 2 (TTL OC) Alarm 1 (TTL OC) FE APV module APVs. PLL Mux-Laser driver Wires between CCU modules CLK+T 1 1 diff pair (LVDS) Data 1 diff pair (LVDS) Reset 1 wire (TTL) ( x 2 for redundancy) CCU module CCU LVDS driver O/E converter Dig. laser driver Dig receiver LVDS driver A. Marchioro / EP 8

“Module Skip” Redundancy DOut CCU CCU Secondary DCU DCU DIn + Clk Primary CCU A. Marchioro / EP 9

Outline u Motivations and Requirements u Architecture u Network and system components: – Token-Ring Network – Local Monitoring – Timing and Trigger Distribution u Network: High level u Project Status A. Marchioro / EP 10

FEC: Front-End Control Board PCI/PMC Connector 4 -way ribbon (8 with redundancy) Optical Intfc. From TTC net LVDS TTCrx Control Logic XTAL FIFOs PCI Adapter A. Marchioro / EP 11

CCU: Communication and Control Unit L 1 ST 2 ST 3 ST 4 Trigger Decoder Tj Counter & other timing logic 18 x I 2 C Buses SCL SDA Power Control I 2 C Master Intfc. Ext Reset* Clock Distribution CLKI(A) CLKI(B) CLKO(A) DO(A) Protocol Adapter & Address Decoder Link Controller DI(A) CLKO(B) DI(B) Memory Bus Interface PIO I 2 C Master Intfc. D[0: 7] A[0: 7] R/W CS* Local Bus PDA[0: 7] PDB[0: 7] A. Marchioro / EP 12

CCU-M Cabling With Redundancy Data In-Port A Data Out-Port A Data In-Port B Data Out-Port B CCU Clk In-Port A Clk In-Port B PLLCKSEL DIn_A CKOSEL DOut_A DIn_B DOut_B Clk. Out_A Clk. In_B Clk. Out_B PLL_Clk A. Marchioro / EP 13

Coding of Clock and T 1: Two Options ~ 25 ns Used CLK T 1 CLK+T 1 ( 25 ns CLK T 1 ~ 8 ns ~ 16 ns CLK+T 1 A. Marchioro / EP ) 14

NRZ_I (Invert 1 on Change) Coding 25 ns 1 1 0 0 0 1 u “ 0” : no transitions ( < 3 conseq. ) u “ 1” : transition at start of cycle A. Marchioro / EP 15

Hard Reset (i. e. how to get always out of troubles) Din Reset Vref u Power-Up reset only must be avoided – use long sequence of 1’s (invalid data sequence) in data stream – extra comparator in receiver ASIC A. Marchioro / EP 16

4 -5 Bit Encoding 4 to 5 bit NRZI decoder NRZI encoder A. Marchioro / EP 5 to 4 bit 17

Other network components u RX 40: a RH 80 MHz BW digital receiver with Automatic Gain Control u Laser Driver: a RH Linear driver for analog and digital data PIN diode in preamplifier L. A. reset Reset block B. F. L. A. sf LVDS Tx out RX 40 Exists already A. Marchioro / EP 18

DCU: Detector Control Unit • Local Monitoring Current Sensing Voltage Monitor 3 Temperatures Bus Interface Control STM Control / Status & Data Registers A. Marchioro / EP 8 Inputs 12 bit A/D SCL SDA • Designed in ¼ mm CMOS • ~ 2 x 2 mm 2 • I 2 C Interface • < 10 m. W • 100 msec/conversion Input Mux Bias Generator for Modulator Temp Sensor 19

Timing Distribution: PLL-Delay CLK&T 1 PLL Delay Lines CLK & T 1 extract T 1 I 2 C Slave Interface I 2 C_Add (hardwired) SDA A. Marchioro / EP SCL 20

Library for digital designs Well isolation NMOS enclosed NMOS guard-ring A. Marchioro / EP 21

Digital Library Features Envisaged Operating Voltage 1001 Elements Ring Oscillator in 0. 25 mm Power: 0. 05 m. W/gate*MHz @ 1 V 0. 25 m. W/gate*MHz @ 2. 5 V A. Marchioro / EP 22

Outline u Motivations and Requirements u Architecture u Network and system components: – Token-Ring Network – Local Monitoring – Timing and Trigger Distribution u Network: High level protocol u Project Status A. Marchioro / EP 23

LAN Protocol u Inspired by Token-Ring u Can work with single master and “simpler” slaves u Extensible to any local protocol (I 2 C. Memory, etc. ) u Max data rate: 32 Mbit/sec A. Marchioro / EP 24

LAN Protocol (1) u Ring-like topology – Circulating token indicates bus available – Source node waits for token: » Removes token, inserts data packet – – Packet circulates, passed by all nodes Destination copies packet, sets “S” symbol Packet returns to source, is removed by source Source node inserts new token A. Marchioro / EP 25

LAN Protocol (2) u All nodes can generate packets u For simplicity: – Only the FEC can perform network supervision » Inserts first token » Monitors token integrity » Receives all data packets from CCUs u FEC requires CPU for protocol management A. Marchioro / EP 26

LAN Protocol: Packet Format Universal SOF Dest Src Length Data CRC EOF Channel Specific Ch# Cmd TR# Addr DW (Example for an I 2 C byte write) A. Marchioro / EP 27

LAN Protocol (3 a): Write Example u Transaction sequence: – – – FEC sends command packet CCU receives packet and sets “S” symbol Command packet returns to FEC CCU directs data portion to channel Channel performs action Channel sends ACK packet to CCU specifying TR# A. Marchioro / EP 28

LAN Protocol (3 b): Read Example u Transaction sequence: – – – FEC sends command packet CCU receives packet and sets “S” symbol Command packet returns to FEC CCU directs data portion to channel Channel performs read action Channel sends data back to CCU specifying TR# A. Marchioro / EP 29

LAN Protocol (4): Addressing u CCU Address allocation – FEC has address 0 x 00 – Special Broadcast class: 0 xf_ – Up to 254 CCUs on ring u Channel Addressing » 0: CCU Node controller » 1 -15: I 2 C channels » 16 -31: memory channels » 32 -47: PIO channel (e. g. HV switches) » 48 -63: Event memory controller A. Marchioro / EP 30

Software Architecture u Lynx-OS User Application 1 User Application N Sync I/O Driver Interrupt Handler u Win. NT u Linux ? u Features: Packet Driver – Multi-User – Multi-Task – Synchronous (wait for Ack. ) PCI-Hardware A. Marchioro / EP 31

Outline u Motivations and Requirements u Architecture u Network and system components: – Token-Ring Network – Local Monitoring – Timing and Trigger Distribution u Network: High level protocol u Project Status A. Marchioro / EP 32

Project Status: Hardware u u FEC: PCI-PMC version exists using FPGAs CCU: rad-soft, 6 I 2 C channels, 0. 6 mm CMOS version is fully functional – Currently being remapped to ¼ mm technology, submission ~ Q 1 ‘ 01 – Full ASIC > 250, 000 gates u u u PLL: exists and is OK in ¼ mm CMOS DCU: design ready for submission Q 1 ‘ 00 RX 40: exists in ¼ mm, design OK Laser Driver: rad soft OK, RH submission Q 1 ‘ 00 Cost: components well within requirements A. Marchioro / EP 33

Project Status: Software u You always end up needing more software than you expected u Low level routines: OK u Partitioning architecture: work started u Performance under Win. NT: – 200, 000 I 2 C transactions/sec (limited by I 2 C, polling) A. Marchioro / EP 34