Summary of TDAQ upgrade open meetings TDAQ upgrade

  • Slides: 42
Download presentation
Summary of TDAQ upgrade open meetings

Summary of TDAQ upgrade open meetings

TDAQ upgrade open meetings n Two meetings up to now n December 2008 n

TDAQ upgrade open meetings n Two meetings up to now n December 2008 n n January 2009 n n n http: //indico. cern. ch/conference. Display. py? conf. Id=46823 http: //indico. cern. ch/conference. Display. py? conf. Id=47257 Norman Gee is the convener TDAQ sessions during the Upgrade week n End of February 2009

Items of past meetings n n n n Introduction and time table (N. Gee)

Items of past meetings n n n n Introduction and time table (N. Gee) Current dataflow limitations (D. della Volpe) Insertable B-layer readout (K. Einsweiler) Physics rates and trigger selection (S. Tapprogge) Montecarlo status (Z. Marshall) SLAC ROD development (A. Haas) Presented in the last TDAQ week n Tracker Level 1 trigger (Nikhef workshop) n n n Clustering on chip, Level 0 accept Level 1 calorimeter trigger (Calo trigger workshop) FTK

ATLAS Studies/Plans n For 2012/13: n New tracker Insertable B-layer (IBL) n n For

ATLAS Studies/Plans n For 2012/13: n New tracker Insertable B-layer (IBL) n n For 2018 s. LHC: n n Additional readout, probably with new-style RODs Replace Inner detector (ASICs will be frozen in 12 -18 months) Calorimetry – digital front ends, new forward calo (ASICs) Muons – rates! Approval Timetable n n 2009/early 2010: IBL TDR 2009: Lo. I - ATLAS changes for SLHC 2010: Technical Proposal (+CMS, + SLHC) 2011: Technical Design Report(s)

TDAQ Need to develop upgrade plans n With physics & trigger community: n n

TDAQ Need to develop upgrade plans n With physics & trigger community: n n With detectors and Electronics Coordinator: n n understand requirements – trigger selections and rates, performance of algorithms, handling pileup, consequences of new detector components Dialogue on key parameters (e. g. latency, trigger and data rates, …) Encourage uniformity at detector/TDAQ interface – protocols, h/w, s/w Calo/L 1 Calo interface; L 1 Muon; role of FTK; L 1 or L 1. 5 track trigger… Internal TDAQ: n n n Develop trigger architecture (L 1; L 1. 5; merge L 2 & EF? ) Identify & study options to handle higher data rates, complex events Cost, effort, timetable; evolve into project TDAQ must get up to speed, reach consensus, be ready to write Tech Proposal n Must do R&D while still supporting running experiment as first priority

Aims for the next few months n Submit Expression of Interest for TDAQ Upgrade

Aims for the next few months n Submit Expression of Interest for TDAQ Upgrade R&D in next 1 -2 months n n n Submit R&D Proposal for TDAQ Upgrade R&D in following 3 months n n Details of programme and justification; institute participation by area; schedule; resources needed; impact on existing work Approved by EB on recommendation of USG Used by institutes in support of requests to funding agencies Produce TDAQ Upgrade Baseline Designs (one for Phase I, one for Phase II) n n Short document; full TDAQ consultation before submission Distributed to ATLAS CB; informs all institutes that the programme is starting First versions for Feb 2009 Initially based on first ideas, rough estimates of behaviour, and evolving as studies are done; but also providing a focus , as time is short (We have agreed to do a similar design for L 1 Calo/Calo interface) Initiate needed studies n Will need wide participation

How? n Small TDAQ upgrade oversight group (Chris, David, Livio, Stefan, Norman, ) n

How? n Small TDAQ upgrade oversight group (Chris, David, Livio, Stefan, Norman, ) n n TDAQ open meetings n n n With others as required. Next is 21 Jan ‘ 09; several meetings in upgrade week 23 -27 Feb ‘ 09; Upgrade workshop in TDAQ week May ‘ 09 Activities being started (list will grow) n n n n Requirements capture Current System Limits (TDAQ and detector front ends/readout) Collection of detector information (future) Data flow modelling HLT processing times, CPU & memory resources Level-1 …

Insertable B-layer (IBL) n n Adds 14 M channels to 80 M Front-end based

Insertable B-layer (IBL) n n Adds 14 M channels to 80 M Front-end based on FE-14 chips n n Intermediate storage to manage higher occupancies. Clusters are formed within a column pairs n n Each FE chip in the IBL n n modest bandwidth reductions at high luminosity Has dedicated TTC (down) and Data (up) links Uses individual fibers to maximize compatibility with existing infrastructure TTC link would run at either 40 MHz or possibly 80 MHz The Data link would run at 160 Mbit/s n n Encoding is under discussion Estimated throughput per link @ 3 x 1034 n n Between 80 and 120 Mbit/s (~30% safety factor for IBL) Present B-layer saturates at 2. 5 x 1034

Operation above 1034 for pixel n These results assume a L 1 trigger rate

Operation above 1034 for pixel n These results assume a L 1 trigger rate 100 KHz n n n Estimates are very naïve There is not a large amount of buffering in the readout system n n n Possible with the existing Silicon RODs. Poor performance expected when the bandwidth limit is approached Operation above design luminosity will require a reduction in trigger rate No detailed simulation effort at the module level A high intensity testbeam program, suggests that the module operation will be (barely) OK. The readout of the present FE chip degrades at high occupancy n n Critical occupancy is expected to be ~2 -3 times design luminosity Significant hit losses will occur in the FE chips, independent of L 1 rate.

Readout configuration n n Assume Silicon ROD (or similar) is used Modularity driven by

Readout configuration n n Assume Silicon ROD (or similar) is used Modularity driven by s-link throughput n n n 160 Mb/s implies 8 FE per ROD Only small changes to the ROD BOC should deal with n n n 160 Mbit/s single links (presently 80 Mbit/s) Improved link protocol 64 RODs (4 VME crates) needed n n Currently 192 (9 VME crates) 25 -50% increase in the total ROD count

From K. Einsweiler SLAC ROD upgrade n n n There is some thinking in

From K. Einsweiler SLAC ROD upgrade n n n There is some thinking in the upgrade community about a longer-term ROD improvement program (in fact it would replace the complete ROD/ROL/ROS system). This is very interesting, and worth following for the SLHC. This approach is very much optimized towards high speed data movement, with embedded general purpose processing which could be very flexibly deployed to perform initial analysis on the data streams. However, the present Silicon ROD has evolved to meet both standard DAQ requirements, and the more complex requirements of Calibration. The ROD supports both the DAQ functions of formatting and monitoring data transmitted from the detector, and the TTC functions of trigger and command transmission. During normal data-taking, the TTC function is largely a transparent broadcast. However, during calibration, the TTC and DAQ functions are tightly coupled as specialized commands are sent to individual modules and specific analysis is performed on the returning data in a tight loop. On the ROD, this requires close coordination of the Master (fixed point) DSP that oversees control functions and command/trigger sequences and the four Slave (floating point) DSPs that analyze returning data using a total of 1 GB of local memory for histogramming purposes. The parallelism provided by the ROD farm (528 FP DSPs and 132 GB of memory) is essential for rapid characterization of the 80 M channel Pixel Detector. The present ROD is complex and imperfect, but after many years of work, it does what we need. The IBL does not seem to require major improvements in the ROD/ROS.

Physics rates and trigger n Detailed rates and menus for LHC n n n

Physics rates and trigger n Detailed rates and menus for LHC n n n Available at 1031 and 1032 How do they scale? Understand requirements n Including pile-up and event size effects

Level 1 rates at 1031 n Total estimate without overlap is ~12 k. Hz

Level 1 rates at 1031 n Total estimate without overlap is ~12 k. Hz

Level 1 rates at 1032 n Total estimate without overlap is ~46 k. Hz

Level 1 rates at 1032 n Total estimate without overlap is ~46 k. Hz

Trigger menus n n Initial prototype for 1031 Description takes more than 40 pages

Trigger menus n n Initial prototype for 1031 Description takes more than 40 pages n Includes algorithm sequences at HLT

Some trivial notes n SLHC physics requirements a moving target n n Lack of

Some trivial notes n SLHC physics requirements a moving target n n Lack of quantitative results n n Some level 1 estimates available Trigger menu for 1031 already too complex n n n Need initial LHC results Need to simplify, will grow with luminosity More complex triggers probably needed at level 1 Need for tools and MC simulations n To have quantitative assessments

Physics requirements to TDAQ n To exploit physics potential of SLHC n Triggers for

Physics requirements to TDAQ n To exploit physics potential of SLHC n Triggers for discovery physics n n Triggers for precision measurements n n n High p. T objects (with similar thresholds as for LHC) Use more exclusive / multi-object selection to control rate Monitor and calibration triggers n n (Very) high p. T objects (thresholds increased wrt LHC) Low to high p. T thresholds (will be pre-scaled) Conditions at 1035 will impact trigger rates n n n Higher rate for fixed threshold and efficiency Trivial increase by corresponding increase in luminosity Further increase due to less effective isolation criteria, fake rate n Due to the 80 – 500 interactions per crossing

Calo trigger (extrapolation) n If we could maintain the same performance as at lower

Calo trigger (extrapolation) n If we could maintain the same performance as at lower L: n L 1 Trigger Thresholds for O(10 k. Hz) rate: n n n L 1 Trigger Thresholds for O(1 k. Hz) rate: n n ~60 Ge. V inclusive isolated EM trigger ~25 Ge. V isolated EM pair ~300 Ge. V inclusive jets ~100 Ge. V ETmiss Saturation rate (250 Ge. V/tower) > 100 Hz Maybe OK for high ET physics. But they are optimistic: n These are trigger thresholds, not ET at which trigger is efficient n n SLHC pileup will degrade performance Can't simply raise all thresholds n Need EW scale triggers still

Muon trigger n Single muon trigger n n n No rate control above ~20

Muon trigger n Single muon trigger n n n No rate control above ~20 Ge. V Rely on combined trigger? Are there possible improvements? Track trigger? n n Already critical at 1034 Big impact on ID upgrade Simulation studies needed

Montecarlo status n n n Is current detector a good approximation? No big problems

Montecarlo status n n n Is current detector a good approximation? No big problems with Muon and LArg changes Most problems coming from ID n n Upgrades implies a lot of channels more There are not enough free channel IDs n n n Geometry is not yet defined n n Implies ID remapping Very difficult to change geomtry Comparison of options needed Private versions of simulation n Non-trivial problems to run it

Full simulation n G 4 produces one event at a time n n With

Full simulation n G 4 produces one event at a time n n With current geometry Digitization able to go up to 1034 n n n Takes few minutes per event Limited by memory usage Timing does not scale with luminosity

ATLFAST-II n Two modular components n Fast. Calo. Sim n n Shower parametrization Only

ATLFAST-II n Two modular components n Fast. Calo. Sim n n Shower parametrization Only calorimetry (no level 1) No time information (offline event overlay not possible) FATRAS (ATLFAST-IIF) n n Readout geometry and custom physics modules Muon system digitized n n Inner detector still not digitized n n Time information available, trigger info No trigger, only in-time pileup Rough timing n n ATLFAST-II (400 Min. Bias events) ~3. 5 h ATLFAST-IIF (400 Min. Bias events) ~15 m

Time scales (unofficial) n Upgrade detector geometry n n n Digitization / full simulation

Time scales (unofficial) n Upgrade detector geometry n n n Digitization / full simulation n Needs decisions and manpower 6+ months from integrated geometry descriptions Can run 1034 some of the time, 1035 perhaps this year Some tricks to get pileup exist, but would need validation ATLFAST-II n n Simulation of in-time pileup exists Limited trigger modules n n L 1 calo ready for validation - month-or-so time scale? Out of time pileup could be many months away

Dataflow system

Dataflow system

ROS PC and ROBin n The ROS PC is a critical point of the

ROS PC and ROBin n The ROS PC is a critical point of the system n Hosts the ROBin n n ROB memory limits the possible level-2 and EB latency Data received through 3 s-link (160 MB/s) A GB ethernet interface is available Event data read out from ROB on request n n Limitation on the rate of level-2 requests Data and delete requests via PCI n Alternative path is the ROBin GB ethernet interface

ROS performance n Limitations from n n L 2 requests EB requests

ROS performance n Limitations from n n L 2 requests EB requests

General limitations of DF n ROB input limited by s-link n n n S-link

General limitations of DF n ROB input limited by s-link n n n S-link throughput is 160 MB/s Each exceeding link generates backpressure ROS is limited by output n L 2 and EB n n n Data requests Data throughput Data size n n TDAQ can handle larger data volumes Actual limitation is Tier-0 (300 MB/s)

Possible improvements n New ROS MB and CPU n Improve the limits due to

Possible improvements n New ROS MB and CPU n Improve the limits due to high data requests n n n PCI Express ROBin n n Current bandwidth 256 MB/s Requires PC upgrade n n n Measured gain ~30% Cost 1. 5 k. CHF / MB Gain >30% Cost 2 k. CHF (PC) + 4*3. 0 k. CHF/ROBin = 14 k. CHF Switch based scenario n n n Use the ROBin GB ethernet to handle requests Tests on-going Requires more ROBins?

The SLAC ROD (1) n n An alternative approach Motivations: n Future running will

The SLAC ROD (1) n n An alternative approach Motivations: n Future running will require more bandwidth n n n Larger events (more detectors, more luminosity) Larger data fraction to L 2 for efficient triggering ROD maintenance is an issue n Hard to find ROD components

The SLAC ROD (2) n Proposed improvements n n Unique ROD Operating system (not

The SLAC ROD (2) n Proposed improvements n n Unique ROD Operating system (not just DSP!) Calibration data at high readout rate Higher bandwidth per “read-out link” n n Full EB at 100 KHz n n n Increased ability to handle hot “read-out links” To get better HLT selection Merge L 2 and EF Address ROS and ROS upgrades together

Components (1) n Reconfigurable Cluster Element (RCE)

Components (1) n Reconfigurable Cluster Element (RCE)

Components (2) n Cluster Interconnect (CI) board

Components (2) n Cluster Interconnect (CI) board

Components (3) n ATCA Read Out Crate

Components (3) n ATCA Read Out Crate

Proposed upgrade n Read-out module (ROM) n Contains 8 (16? ) RCE's n n

Proposed upgrade n Read-out module (ROM) n Contains 8 (16? ) RCE's n n Can read-out and pre-process data from front-ends Could be used as upgraded ROD n n Plus L 1. 5 triggering? Can buffer the data and present it to L 2/EF over Ethernet Could be used as upgraded ROS Proposed for B-layer…

Possible Atlas ROM

Possible Atlas ROM

Possible Atlas ROC

Possible Atlas ROC

Brainstorming on DAQ/HLT n Focused on architecture and infrastructure n n Mainly for HLT

Brainstorming on DAQ/HLT n Focused on architecture and infrastructure n n Mainly for HLT Called by Andre Anjos n Nov 19 th 2008 n n Appendix to the DAQ week Jan 22 nd 2009

Current TDAQ architecture 3 data networks Simplify the design Keep flexibility 2 types of

Current TDAQ architecture 3 data networks Simplify the design Keep flexibility 2 types of processing units Merge L 2 end EF on the same PU 7 types of applications Queues and timeouts

Key points n Simplify HLT n n n Automatic load balance Testing and deployment

Key points n Simplify HLT n n n Automatic load balance Testing and deployment in a single application Simplify networking n n Merging L 2 and EF networks Makes possible incremental event building n n Latency and ROB size? Simplify configuration n Fewer application type Fewer queues Fewer timeouts

Some other aspects n Timing n n Phase-I – 2012 -2013? Adiabatic approach n

Some other aspects n Timing n n Phase-I – 2012 -2013? Adiabatic approach n First prototype by Mario Bondioli (Irvine)

Personal notes n Requirements: n Dynamic system n n n Process control n n

Personal notes n Requirements: n Dynamic system n n n Process control n n n Online latency measurements needed Dynamic thresholds and timeouts HLT must be a state machine Allocated resources must be de-allocated Machine resources n Multithreading to n n n Share resources Improve scalability with new machines Current “offline approach” is not correct n n Lack of control Bad use of resources Dependencies are complex and in the wrong direction Start from an online framework !

Conclusioni n Quale puo’ essere il coinvolgimento dei gruppi italiani? n n n Interessi

Conclusioni n Quale puo’ essere il coinvolgimento dei gruppi italiani? n n n Interessi attuali Tempi Sinergie da sfruttare al massimo n n n Aspetti HW e SW da considerare congiuntamente Interazioni tra le parti da valutare accuratamente Cerchiamo fin da ora di compattarci il piu’ possibile