The LHC orbit and tune feedback system Diogo
The LHC orbit and tune feedback system Diogo Alves BE-BI-SW 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves
Outline • LHC beam-based feedbacks: an overview • Main changes from LHC run 1 to run 2: another overview • Testbed • Dashboard • Coming changes • Plans for the future, lessons learned & conclusions 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (2)
Outline • LHC beam-based feedbacks: an overview • Main changes from LHC run 1 to run 2: another overview • Testbed • Dashboard • Coming changes • Plans for the future, lessons learned & conclusions 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (3)
Feedbacks … why do we need them? Why is orbit correction needed? • Quadrupole magnet misalignments → dipolar deflections • Perturbations vary with time (ground motion, earth tides and optics changes) • Mix of random and reproducible components Why is tune correction needed? • Tune changes during dynamic phases (energy and optics changes) • Wrong tune values at injection and during the ramp can lead to high losses • Feedback most useful during the ramp at energies < 2 Te. V 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (4)
Beam Position Monitors Inputs and Outputs Tune Meas. Systems 12 > 2000 BPM > 250 Quad > 500 Dipole Q Dipole Magnets Beam-based feedback controller Quadrupole Magnets 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (5)
Data flow 67 FECs UDP @ 25 Hz → ← 1 Hz Dipole UDP 25 Hz acq BPM LHC FB Service Unit LHC FB Controller UDP @ 25 Hz → ← 1 Hz (non-FESA) Quad Q 10/03/2016 4 FECs UDP @ < 12. 5 Hz TCP settings & diagnostics (FESA) Settings management CMW Monitoring Acquisition & Logging LDB BI day - LHC orbit and tune feedback system - D. Alves (6)
Outline • LHC beam-based feedbacks: an overview • Main changes from LHC run 1 to run 2: another overview • Testbed • Dashboard • Coming changes • Plans for the future, lessons learned & conclusions 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (7)
Fundamental changes from LHC run 1 to run 2 • Hardware • 32 -bit, 4 core Intel G 5 s → 64 -bit, 24 core Intel Gen 8 s • Configuration • Real-time SLC 5 (32 -bit) OS → Real-time SLC 6 (64 -bit) OS • Software • Controller • 32 -bit → 64 -bit port • Service Unit • 32 -bit → 64 -bit port • FESA 2 → FESA 3 (RDA 2 → RDA 3) • People • R. Steinhagen + M. Andersen → S. Jackson + D. Alves 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (8)
Porting Service Unit to FESA 3 • Code functionality and content (> 35 k lines of code) was mostly unknown in advance • 79 FESA “actions” • 19 supporting classes • Issue tracking (JIRA) for managing all changes • One issue (at least) was created for the porting of each class/action • Code commits are tagged with the issue number, this allows for automatic cross-referencing of issues and SVN commits • At the time we started porting (July 2014) the FESA migration tool was not mature enough hence all the porting was done by hand • Code reviews • 3 -4 months added up for the “complete” porting 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (9)
Overview of main changes already in place • Controller • Re-engineered orbit trigger delivery mechanism (now using CTR) • Improved logging mechanism (CO log viewers now usable) • Previously hardcoded cfg → XML-based cfg • Added CPU profiling info (exposed in service unit) • Added stats on IIR and Sync orbits (exposed in service unit) • Code infrastructure for incorporating DOROS data • Continuous build and unit testing • Service unit • Re-engineered settings transaction mechanism • Greatly improved system stability 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (10)
Outline • LHC beam-based feedbacks: an overview • Main changes from LHC run 1 to run 2: another overview • Testbed • Dashboard • Coming changes • Plans for the future, lessons learned & conclusions 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (11)
Testbe d 67 FECs UDP @ 25 Hz → ← 1 Hz Dipole UDP 25 Hz stream BPM UDP @ 25 Hz → ← 1 Hz LHC FB controller TCP settings & diagnostics Quad Q 10/03/2016 LHC FB Service Unit Simulate the LHC 4 FECs UDP @ < 12. 5 Hz BI day - LHC orbit and tune feedback system - D. Alves (12)
Release candidates under test Testbe d UDP 25 Hz stream (non-FESA) Why? • Testing without beam • BI & OP developers can test changes • Reproduce operational problems and validate fixes • OP+BI can accept (or not) changes • Document system behavior • Particularly useful in a code inheritance context 10/03/2016 LHC FB Service Unit LHC FB Controller TCP settings & diagnostics (FESA) CMW LHC feedback testbed (Java + JUnit) Dipole BPM Quad UDP RT I/O Q Settings management + Acquisition ASSERT that the FBs behave as specified! BI day - LHC orbit and tune feedback system - D. Alves (13)
Agile Scrum • Very focused team work • Achieving a complex task incrementally by completing and building on smaller tasks • Close and cooperative interaction including daily standup meetings • Breakdown of work into “sprints” (3 -4 weeks) • Breakdown of sprints into tasks (1 -5 days) • JIRA used to track tasks • Automatic sprint/task tracking integrated into JIRA’s Agile scrum board 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (14)
Outline • LHC beam-based feedbacks: an overview • Main changes from LHC run 1 to run 2: another overview • Testbed • Dashboard • Coming changes • Plans for the future, lessons learned & conclusions 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (15)
Dashboard – why? • Gather statistics on the relative offsets of the arrival of measurement data • Inspect RT I/O (UDP packets) • Display summarized status information on the latency and content of the input data • Have the means to rapidly diagnose feedback problems during commissioning and operation in a non-intrusive way (i. e. without compromising the feedback system’s real -time behavior) 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (16)
Dashboard - Architecture 67 FECs UDP @ 25 Hz → ← 1 Hz Dipole UDP 25 Hz stream BPM LHC FB Dashboard Controller UDP @ 25 Hz → ← 1 Hz (FESA) (non-FESA) Quad Q 10/03/2016 TCP settings & diagnostics LHC FB Service Unit CMW (FESA) 4 FECs UDP @ < 12. 5 Hz BI day - LHC orbit and tune feedback system - D. Alves (17)
Dashboard – RT I/O inspection 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (18)
Outline • LHC beam-based feedbacks: an overview • Main changes from LHC run 1 to run 2: another overview • Testbed • Dashboard • Coming changes • Plans for the future, lessons learned & conclusions 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (19)
Main changes coming soon … to an LHC near you • Collect BPM calibrations (BI-QP request) • Expose in Service Unit • Logging purposes • Offline Analysis Framework (see B. Kolad’s presentation) • Timing-based Function Player for driving FB settings • Homogenize operational procedures for configuring LHC systems • Setup tune ref, orbit ref, optics (and more) using preprogrammed waveforms as a function of time 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (20)
Function Player for FB Settings 67 FECs UDP @ 25 Hz LHC FB Function Player UDP @ 25 Hz → ← 1 Hz Dipole (FESA) UDP 25 Hz stream BPM LHC FB Service Unit LHC FB Controller UDP @ 25 Hz → ← 1 Hz (non-FESA) Quad Q 10/03/2016 Drives & Monitors TCP settings & diagnostics (FESA) a a a Configure & Monitor Central timing 4 FECs UDP @ < 12. 5 Hz BI day - LHC orbit and tune feedback system - D. Alves (21)
Outline • LHC beam-based feedbacks: an overview • Main changes from LHC run 1 to run 2: another overview • Testbed • Dashboard • Coming changes • Plans for the future, lessons learned & conclusions 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (22)
Outlook • Commission Function Player • Continue to enhance the testbed • Increase code coverage • Integrate DOROS data • Re-engineer optics loading mechanism • Split of service unit into 2 FESA classes OP sets priorities! 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (23)
Lessons learned and conclusions • Documentation • Software practices • • Unit-testing Functional testing (testbed) Continuous build Coding conventions Code reviews Issue tracking (JIRA) Agile methodologies Use/develop diagnostic tools • Close collaboration with OP 10/03/2016 • Confidence in making changes • Confidence in deploying new versions • Support from OP • Software practices instrumental in the smooth restart and operation of the LHC beam-based FBs • Apply software practices in other BI-SW projects BI day - LHC orbit and tune feedback system - D. Alves (24)
Team work! Past: R. Steinhagen, G. Sivatskiy and M. Andersen 5 2 6 9 4 1 8 7 3 OP-LHC • L. Di Giulio (1) • K. Fuchsberger (2) • J. Wenninger (3) 10/03/2016 BI-SW • • D. Alves (4) S. Jackson (5) B. Kolad (6) I-D. Rodis (7) CO-DO • V. Baggiolini (8) • J. Pedersen (9) BI day - LHC orbit and tune feedback system - D. Alves (25)
Backup slides 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (26)
GROUPED 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (27/15)
Why is orbit correction needed? Orbit is disturbed by dipolar deflections mainly due to: • Dipole field errors Spread in bending angles due to manufacturing defects or dynamic tolerance. They can be static or vary with the field itself. In general rather reproducible from one machine cycle to the next. • Quadrupole magnet misalignments This is the dominant perturbation in the LHC. Varies with time, i. e. ground motion, earth tides and optics changes. Mixture of random and reproducible components. 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (28)
Why is tune correction needed? • During commissioning, MDs and special physics runs Feedbacks help establish feedforward corrections. Most critical during dynamic phases where the energy and optics are changed. • Once the cycle is setup The main tune uncertainties concern injection and the first part of the ramp. Tune feedback is most useful during the latter to fine-adjust the feedforward corrections at energies < 2 Te. V. 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (29)
Porting Service Unit to FESA 3 • Code functionality and content was mostly unknown in advance (> 35 k lines of code) • 49 server actions • 30 real-time actions • 19 supporting (common) classes • Issue tracking (JIRA) for managing all changes imposed by the porting process • One issue (at least) was created for the porting of each class / action • Code commits are tagged with the issue number, this allows for automatic crossreferencing of issues and SVN commits • At the time we started porting (July 2014) the migration tool was not mature enough hence all the porting was done by hand • Many issues with inherent 32 -bit assumptions in original code • Handling of arrays changed from FESA 2 to FESA 3 • Many data-consistency issues with settings-properties setting acquisition-fields (and viceversa) • 3 -4 months added up for the “complete” porting 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (30)
Dashboard - Architecture 67 FECs UDP @ 25 Hz → ← 1 Hz UDP 25 Hz stream LHC FB Controller (non-FESA) TCP settings & diagnostics LHC FB Service Unit (FESA) 4 FECs UDP @ < 12. 5 Hz 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (31/15)
Function Player for FB Settings 67 FECs UDP @ 25 Hz → ← 1 Hz UDP 25 Hz stream LHC FB Service Unit LHC FB Controller (non-FESA) TCP settings & diagnostics (FESA) LHC FB Function Player (FESA) 4 FECs UDP @ < 12. 5 Hz 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (32/15)
Parallel projects • Testbed for the LHC beam-based feedbacks • Joint collaboration with CO-DO and OP-LHC • Ensure that changes do not change system behaviour • Reproduce operational problems & validate new features • Acceptance mechanism for Sw changes • Documentation • Dashboard for the LHC beam-based feedbacks • Gather statistics on the relative offsets of the arrival of measurement data • Inspect RT I/O in a user-friendly way • Display summarized status information on the synchronism and content of the input data • Have the means to rapidly diagnose problems with the feedbacks during commissioning and operation in a non-intrusive way (i. e. without compromising the feedback system’s real-time behavior) 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (33/15)
Java-based Domain Specific Language (DSL) 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (34)
Automated build for feedback tests 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (35/15)
Dashboard – RT I/O status Bpm. Dab. Temperature 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (36/15)
Changes coming soon to an LHC near you • BPM calibration collection (BI-QP request) • Expose in Service Unit • Logging purposes • Offline Analysis Framework • Timing-based Function Player for driving FB settings • Homogenize operational procedures for configuring LHC systems • Setup tune ref, orbit ref, optics (and more) as a function of time • Triggered manually over CMW, or via MTG events • Generic FESA-class will handle all use cases • Main functionality developed and unit-tested outside FESA 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (37)
Main changes coming soon … to an LHC near you • Collect BPM calibrations (BI-QP request) • Expose in Service Unit • Logging purposes • Offline Analysis Framework • Timing-based Function Player for driving FB settings • Homogenize operational procedures for configuring LHC systems • Setup tune ref, orbit ref, optics (and more) using preprogrammed waveforms as a function of time 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (38)
Outlook • Commission controller with DOROS data • Basic Sw infrastructure already developed • Delicate configuration changes • Regenerate bootstrap files from MAD-X • Will require thorough testing • Re-engineer optics loading mechanism • Split of service unit into 2 FESA classes • OP sets priorities! 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (39)
Still on the table • Re-write of Controller in FESA • Removal of root serialisation mechanism between service unit and controller • Loading of the optics via CMW rather than executing external Java process from service unit 10/03/2016 BI day - LHC orbit and tune feedback system - D. Alves (40/15)
- Slides: 40