HLLHC CMS Upgrade QA Plan Carol Wilkinson HLLHC

HL-LHC CMS Upgrade QA Plan Carol Wilkinson HL-LHC CMS Associate Project Manager NSF FDR Dry Run December 13, 2018 C. Wilkinson

U. S. QA Program The primary goal of Quality Assurance for U. S. CMS deliverables is to ensure that the CMS experiment achieves the science requirements and goals listed in the document Science Requirements for U. S. CMS HL-LHC (CMS-doc 13337). Major components of the U. S. CMS QA Program are: § Quality Assurance Plan (CMS-doc-13093) § Subproject (L 2) Quality Assurance Plan Appendices § Subproject (L 2) QA Activities Spreadsheets § Experienced, dedicated, and pro-active technical leads § Assigned FNAL QA Coordinator for U. S. CMS effort § QA oversight by CERN, CMS management, and U. S. CMS management NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 2

Quality Assurance Plan Show revised cover page NSF FDR Dry Run - Dec. 13, 2018 Key Elements • Defines Quality Assurance expectations from international CMS through Fermilab to participating institutions in the U. S. • Assigns roles and responsibilities for QA oversight • Outlines the work process controls and QA validation to ensure that CMS achieves the stated science requirements C. Wilkinson 3

Section 1: Introduction and Purpose § QA is an integral part of the design, fabrication, and construction of the U. S. CMS Upgrade Project § CMS has a formal review and approval process and hierarchy for setting requirements, determining QA activities, and providing acceptance § All components and subproject deliverables must meet CMS approved science and engineering technical and programmatic requirements § All work performed by U. S. CMS draws on guidelines and criteria set out in the Fermilab Quality Assurance Manual, except where CMS requirements take precedence NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 4

Section 2: Scope and Applicability § Applies to all U. S. CMS Project activities funded and undertaken by § National Science Foundation (NSF), and § Department of Energy (DOE) § Controlled document § Approved & signed by the PM (DOE), Deputy PM (NSF), Fermilab Chief Project Officer, and U. S. CMS QA Coordinator § Periodic review and updates NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 5

Section 3: CMS Project Organization International CMS Organization § Overall responsibility for CMS QA is held by CERN and the international CMS collaboration § The detector will be installed and run @ CERN § The CMS Technical Coordinator (TC) – Austin Ball, appointed by CERN, holds overall responsibility for CMS QA activities § Works with the CMS Upgrade Coordinator (UC) on QA for the subdetector upgrade design and fabrication § TC and UC report to the CMS Spokesperson and the CMS Management Board § The Upgrade Coordination Group (UCG) - international leaders for each of the upgrade subdetector projects § Each CMS collaborator/participating institution is responsible for day-today QA practices that meet subdetector QA processes NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 6

Section 3: CMS Project Organization U. S. CMS Organization (w. r. t International CMS) § Each CMS collaborator is responsible for day-to-day QA and performance that meet processes developed by each of the subdetectors § The U. S. CMS HL-LHC Upgrade Project is integrated into the organization and follows CMS QA methodology and standards § In most cases, the day-to-day CMS processes are compatible with the Fermilab QAM. § If there is a discrepancy between CMS and Fermilab, the CMS process supersedes that of Fermilab. NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 7

Section 4: CMS Integrated QA Review and Approval Process § CERN has a formal, multi-step approval and verification process for all LHC experiments § Described in the CERN document LHC Experiments Phase II Upgrades Approval Process § Follows recognized large project management methods (e. g. ISO 21500(2012)), with phases, reviews, and decision points detailed in the document. § QA procedures are developed collaboratively, approved by CMS subdetector managers, coordinated by CMS TC, enforced by series of reviews § U. S. CMS HL-LHC L 2 s implement these QA procedures in their subproject making sure also to comply with U. S. project requisites § Reviews are called by the CMS TC in conjunction with the UC and include non-CMS technical experts. NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 8

QA Project Reviews § Reviews are scheduled at four stages of the project: § § Stage 1 – Initial Design Stage 2 – Baseline Design Stage 3 – Final Design/Start of Construction Stage 4 – Installation and Commissioning § Each review stage includes a QA component § § § Technical or engineering design reviews provide QA planning and design validation during R&D and Preproduction activities Acceptance reviews, including readiness reviews, provide QA during production and installation activities Ad Hoc reviews called as-needed basis by the U. S. Project or the CMS TC NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 9

Section 5: Roles and Responsibilities CMS and CERN management § CERN LHCC/UCG § Approve plans for QA as part of Step 2 baseline design approval and at Step 3 for detailed implementation approval. § CMS TC and Electronics Coordination (EC) Leads § Keep up-to-date drawings and ensure inter-compatibility between CMS subcomponents and LHC infrastructure. § Participate in planning/coordination of QA activities and metrics. § Call for reviews of all subprojects leading to Step 3 § Maintain technical documentation in the CMS EDMS or Doc. DB document systems, including specifications and QA procedures. § CMS Upgrade Coordination Lead § Calls for and conduct the CMS internal reviews leading to Step 2 § Works with the CMS TC for reviews leading to Step 3. NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 10

Section 5: Roles and Responsibilities, cont. For CMS and CERN Subdetector personnel § CMS Subdetector Leads § Oversight and management of each integrated detector subsystem, including efforts from all contributors. § CMS Subdetector QA Managers (assigned by CMS) § Coordinate of QA processes across all participating institutions for that subdetector. § Responsible for defining or approving assembly/test procedures for each component or subassembly § Maintain common data-base and tracking tools for grading, performance matching, and history tracking NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 11

Section 5: Roles and Responsibilities, cont. For U. S. Management and subproject personnel § Project Manager (and deputies) § Ultimate QA responsibility for U. S. scope § Project Scientist § Support planning and provide review of the Quality Tests and Inspections developed by the WBS Level 2, 3, & 4 managers/leads § Works with Subproject Leads and QA Coordinator to ensure technical requirements are met § Subproject Leads (WBS L 2, L 3, L 4) § Responsible for QA for their scope of work § QA Coordinator § Provide planning support & review/surveillance of participating institution QA procedures § QA representatives at participating institutions NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 12

U. S. CMS HL-LHC Upgrade Organization QA Responsibilities HL-LHC CMS Detector Upgrades Project U. S. CMS UPGRADE ADVISORY BOARD U. S. CMS COLLABORATION BOARD Project Manager V. O’Dell (Fermilab) Chair: M. Chertok Deputy: M. Klute Chair: Deputy: Deputy PM (NSF) A. Ryd (Cornell) M. Narain S. Eno Deputy PM (DOE) V. Papadimitriou MANAGEMENT TEAM PROJECT CONTROLS, FINANCE, ADMIN. Project Controls Lead: Scheduler: Finance (DOE): Finance (NSF): 402. 2: Outer Tracker L 2 Manager: S. Nahn Dep. P. Merkel 402. 3: Barrel Calo. L 2 Manager: C. Jessop NSF FDR Dry Run - Dec. 13, 2018 Assoc. PM (cost, schedule, risk): L. Taylor Assoc. PM: C. Wilkinson Project Scientist: C. Hill Systems Engineer: J. Dolph CMS HL-LHC Liaison: P. Rumerio Education and Public Outreach: S. Rappoccio ESH&Q: T. J. Sarlina W. Freeman S. Rogers J. Teng W. Franklin 402. 4: Endcap Calo. L 2 Manager: J. Mans Dep. H. Cheung 402. 5: Muons L 2 Manager: A. Safonov 402. 6: Trigger/DAQ L 2 Manager: J. Berryhill Dep. R. Cavanaugh, K. Ulmer C. Wilkinson 402. 7: TFPX L 2 Managers: J. Alexander K. Ecklund 402. 8: MIP Timing L 2 Manager: C. Neu Dep: F. Chlebana, D. Stuart 13

Section 6: Overview of U. S. CMS QA U. S. QA deals with day-to-day work process control and oversight for production. It creates/follows detailed QA plans and procedures, verifies vendor and collaborator capability and compliance, and performs acceptance testing. § Follows guidelines from the Fermilab Quality Assurance Manual, with details of processes specifically applicable to U. S. CMS. § Applies to all work performed at U. S. CMS collaborating or participating institutions. § Is general in nature due to differences in the type of deliverables for each subproject as well as the methods of interaction between subproject and CMS § The specifics for each U. S. CMS L 2 subproject QA activities are given in the Appendix to the QAP. NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 14

Section 6: Graded Approach Fermilab uses a graded approach to an appropriate level of formality for the severity of negative impact on ES&H, cost and schedule, or performance. Four graded ESH&Q Risk levels are defined (See QAP for examples): § ESH&Q Level 1 – Critical potential impacts, requiring a disciplined set of actions. § ESH&Q Level 2 – Major potential impacts, justifying a balanced set of actions. § ESH&Q Level 3 – Minor potential impacts, justifying a flexible approach. § ESH&Q Level 4 – Negligible potential impacts, justifying a minimal approach. Adjust the QA planning, process controls, and documentation to fit the risk level. NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 15

Section 6: Qualifications and Training § U. S. CMS Project Managers and subproject leads responsible for ensuring that team members are adequately trained and qualified, based on. § U. S. CMS leads have varying level of experience and expertise with respect to QA § Most are selected based on demonstrated technical skills and experience as well as past project management and CMS experience § Some are already knowledgeable about formal QA § On-the-job training provided through interactions with the CMS QA Managers and U. S. QA Coordinator § Majority of participating institutions have past CMS experience/expertise § Remainder have demonstrated capability NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 16

Section 6: Requirements § Deliverables must comply with Technical Requirements flowdown from science goals and requirements. § Requirements established at TDR and finalized at EDR, per CMS review and approval process. § Technical and programmatic requirements documented for each subsystem § Activities/deliverables follow CMS codes and standards § EU codes and standards followed with some exceptions o U. S. destination tooling, o Electronics testing at U. S. sites o Software § U. S. leads responsible for ensuring appropriate codes and standards are used at all U. S. institutions NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 17

Section 6: Work Processes and Controls § Design process is a collaborative effort across CMS, controlled by the CMS review and approval process § Requires close coordination to ensure that engineering/design satisfies the Technical Requirements of the experiment. § Determines QA/QC verification and inspection criteria of fabricated products and installations. § Production Quality Assurance includes activities (tests, inspections, analyses, etc. ) which will demonstrate actual compliance with requirements. § Fabrication work performed according to approved instructions and procedures. § Subsystem activities documented in QA Activity spreadsheets linking QA to technical and programmatic requirements NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 18

Section 6: Change and Document Control § Change Control over requirements, scope, cost and schedule, drawings, risk assessments, and project level documents implemented close to PDR/CD-1. § U. S. CMS HL-LHC Configuration Management Plan § CMS TC has ultimate responsibility to keep up-todate documents, drawings § CERN Engineering Design Management System (EDMS) § Each subproject has a system of maintaining and tracking issue reports, travelers and test records, and other project documents, § Doc. DB and CERN EDMS are the project storage sites. NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 19

Section 6: Additional Topics § Software QA § Follow QAM section on software development § Procurements § Follow the participating institutions procurement policies § Ensure QA /QC activities and responsibilities established by time of award § Non-conforming or counterfeit items § Follow QAM on labeling and quarantining § Management and/or Independent Assessments § Quality Improvements and Lessons Learned programs NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 20

Section 7: Participating Organizations Participating U. S. CMS institutions with DOE or NSF funded scope must follow QA plans that satisfy CMS Subdetector requirements and the QAP. § The U. S. L 2 and L 3 subproject leads work with institutional representatives and the US CMS QA Coordinator to create QA plans per the graded approach in this QAP. § The institutional QA plan should cover all work managed by the institution, including subawards to vendors or other participating institutions. § Local QA responsible contact designated for each site § Participating institutions staff are responsible for verifying that the QA plans are compatible with any local institutional QA requirements § The subproject L 2 lead and the US CMS QA Coordinator review and approve the institutional QA plans and monitor/verify compliance. § Site visits may be required for QA plan approval and monitoring NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 21

Section 7: Site Visits/Audit Forms § Standardized forms for Site Field Visit Audit Checklists and Reports NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 22

QAP Subproject Appendices The QAP appendix contains high level descriptions of each L 2 U. S. CMS subproject, including: § Short description of the types of deliverables (designs, hardware, software, test results, etc. ) § Subdetector organization and communication methods within CMS and U. S. CMS § Short description of the types of QA activities (electronic prototyping, simulations or other modeling, material testing, procurement, assembly, QC, performance testing, etc. ) § List of participating institutions § Management of non-conforming parts § Document and Record keeping NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 23

Example: Trigger/DAQ Appendix Summary § Design, production, and performance testing of trigger electronics and DAQ hardware and firmware § Primary responsibility for deliverables, but with tight interfaces to entities responsible for trigger inputs and outputs § Independently manages its work while meeting requirements § Jointly funded by NSF and DOE § Ten participating DOE institutions and six NSF § Strong history of CMS work § QA activities: vendor/assembler pre-qualification, QC inspections, integrated testing using simulated data and demonstrators NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 24

Trigger/DAQ - Scope NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 25

Trigger/DAQ – Organization & Interfaces NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 26

Trigger/DAQ – Design Validation NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 27

Trigger/DAQ - Production Verification NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 28

Trigger/DAQ – Participating Institutions 3 NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 29

Trigger/DAQ – Documentation 5 Document/Record Storage: Trigger/DAQ documents are stored on the CMS Doc. DB site that is accessible to authorized project stakeholders. Hardware deliverables data and testing results are tracked in a spreadsheet, stored on the CMS Doc. DB site. Issues and handling of nonconforming parts are also tracked in the same spreadsheet. Testing/commissioning/production software and firmware tracked in github repositories. The CMS Technical Coordinator has ultimate responsibility for managing all Trigger/DAQ documentation. NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 30

QA Activities Spreadsheets § Activity Titles and summary descriptions § Assigned responsibilities/contacts § Links to technical engineering and/or scientific requirements § Some activities and technical requirements still being finalized during remaining design phase § References to procedures, related hardware, training, calibrations § Working documents expected to evolve and mature over time § Ultimate goal to combine with subproject technical requirements to complete science flowdown NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 31

Ex: Trigger/DAQ QA Spreadsheet § Activity titles, WBS, responsibilities, and links to technical requirements NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 32

Ex: Trigger/DAQ QA Spreadsheet (cont. ) § Activity descriptions, procedures, calibrations, records, training NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 33

Summary § Met the requirement to have a Quality Assurance Plan § Quality Assurance resources have been assigned with defined roles and responsibilities § hierarchy of Quality Assurance controls to U. S. participating institutions § Established the process for capturing flowdown from technical requirements to QA Activities § Documentation in process § On track to be ready for the Final Design Review in January NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 34

Back-up slides C. Wilkinson

Biographical Sketch § CMS Upgrade QA Coordinator (T. J. Sarlina) § Assistant Radiation Safety Officer for Meson Department (1979 -1982) § Fermilab Senior Safety Officer for Research Division and Particle Physics Division (1982 -2002) § Project Scheduler (2002 -2008) § CDF Upgrade Project, DO Upgrade Project, Minerva, Dark Energy Camera. § Project Manager at Fuel Tech, Inc. (2008 -2010) § Air Pollution Control Projects (power plants and refineries) in Hong Kong, Guangzhou, Liaoning Province. Austin, TX and Seattle, WA. § Associate Project Manager for ESH and QA for NOv. A (2010 -2014) § Constructed and commissioned Near Detector at Fermilab and Far Detector in Ash River, MN. § Fermilab Quality Assurance Manager (2014 -2017) § Transitioned the Fermilab QA Program from consultant led to internally owned. § Fermilab Quality Assurance Specialist (2017 -present) § Supporting the Fermilab QA Program under Jemila Adetunji, IERC QA Coordinator, CMS US HL-LHC QA Coordinator NSF FDR Dry Run – Dec. 13, 2018 C. Wilkinson 36

Relevant i. CMS Management § CMS UC is ultimately responsible for scientific/technical requirements § In practice, delegated to subdetector PMs § CMS TC is ultimately responsible for QA for technical & programmatic requirements § Delegated to subdetector PMs for implementation for non-safety reqs. § TC enforces through reviews § Formally delegated to LEXGLIMOS (CERN safety professional) for safety reqs. NSF FDR Dry Run - Dec. 13, 2018 UC = Upgrade Coordinator TC = Technical Coordinator C. Wilkinson

i. CMS Approval Steps § i. CMS collaboration QA practices are embedded in the formal review and approval process described in the LHC Experiments Phase II Upgrades Approval Process [CERN LHCC-2015 -007]. § The following steps are required for each CMS subdetector in the Upgrade: § Step 1: Initial Design § review overall scope and cost for the entire upgrade program for each experiment, retaining the possibility for different options which may depend on technical issues and/or on funding availability. Approve readiness to proceed to Step 2. Step 1 is documented in the CMS Upgrade Technical Proposal [Ref-7] and Scope Documents [Ref-8]. § Step 2: Baseline Design § review and approve Technical Design Reports and QA plans for each subdetector. This documents the baseline scope, cost and schedule for the subsequent change control process. § Step 3: Final Design / Start of Construction § review and approve the final design and the production of the major detector components, verifying that they meet the requirements and are compatible with the installation plan. Establish follow-up reviews/approvals for installation readiness. § Step 4: Installation and Commissioning § review and approve the installation and commissioning of the major detector components. Evaluate the capability of the integrated detectors to provide the expected performance. Review and approve readiness for operations. § Each of these steps includes review/approval at the CMS level, followed by review/approval by CERN LHCC/UCG and RB. NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson approval of the TDRs => we are roughly here

Recommendations from CD-1 § Revise the ISM and QAP to accurately document the process for receipt, review, concurrence, coordination, and oversight of project specific plans and activities prior to the issuance of any contract instrument. § Develop a clear plan for identification and documentation of codes, standards, requirements, and timing for inclusion. § (The process for identification and documentation of requirements and standards for engineering and design is not clearly defined for European Union and U. S. standards and code compliance, this should be developed soon as designs and engineering specifications are being written. ) NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 39

Site Field Audit Checklist NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 40

Site Field Audit Checklist NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 41

Site Field Audit Checklist NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 42

Site Field Audit Report NSF FDR Dry Run - Dec. 13, 2018 C. Wilkinson 43