DESI Spectroscopic Pipeline Adam S Bolton University of

DESI Spectroscopic Pipeline Adam S. Bolton, University of Utah DESI Data Systems Lead Scientist DESI pipeline L 3 manager Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline)

DESI Spectroscopic Pipeline • Core mission for DESI Data Systems • Translate raw spectroscopic data into redshifts • A central item for DESI experimental success: optimal software needed to extract full information content delivered by DESI hardware, and to enable most precise & accurate cosmological analysis • Builds on proven design and team experience from SDSS, BOSS, and e. BOSS pipelines • Solid plan for construction will deliver working DESI pipeline system at first light Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 2

Statement of the Problem multiple exposures (Most counts are night-sky photons) Wavelength Translate raw spectroscopic CCD data into spectra, classification, and redshifts Fiber number Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 3

Pipeline Calibration Elements Pixel-Level Calibration Bias subtraction, gain correction, pixel-response correction, badcolumn masking, cosmic-ray rejection, pixel error estimation Fiber-Flat Calibration Spectral trace positions, fiber-to-fiber throughput corrections, crosssectional profile determination (integral constraint on PSF) Arc Frame Calibration Wavelength solution, spectrograph pointspread function calibration Additional Calibration Monochromatic calibration systems, sky lines for calibration tweaking to science frames, instrument priors from calibration library or hardware design WBS items 1. 8. 4. 1, 1. 8. 4. 2, 1. 8. 4. 5 Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 4

Extraction & Sky Subtraction • Extraction using 2 D spectrograph PSF (New for DESI) Bolton & Schlegel, 2010 Comparison of extraction methods Raw Data Row-by-row residual 2 D PSF-based residual • Forward-model sky subtraction (Key for DESI ELGs!) Before Adam S. Bolton After DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 1. 8. 4. 1, 1. 8. 4. 2, 1. 8. 4. 5 5

Fluxing and Coaddition Theoretical stellar spectra in physical flux units matched to spectrophotometric star spectra in count units to determine fluxing vectors Adam S. Bolton Multiple arms and exposures combined with a single coadd model spectrum, along with coadded errors and resolution matrices DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 1. 8. 4. 1, 1. 8. 4. 2, 1. 8. 4. 5 6

Classification & Redshifting • Minimum-chi-squared forward modeling - Basis of archetypes or PCA eigenspectra Multiple template classes (ELG, LRG, QSO, star) Objective model comparison, errors, & confidence Bolton et al. 2012, AJ, 144 (Note: S/N not representative of DESI!) Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 1. 8. 4. 1, 1. 8. 4. 2, 1. 8. 4. 5, 1. 8. 4. 8 7

Pipeline System Diagram Hardware Parameters Database Raw Science CCD Frames Legend Raw Calibration Frames Biases Darks Flats Arcs Processes 2 D PSF Calibration Auxiliary Data Pre-Processing Outputs Calibration Archive Pixel-Level Calibration Meta-Data Inputs Master Biases Badpix Masks Master Darks Pixel Flats 2 D PSF Models & Priors Extraction Guider Data Sky Subtraction Extracted Frames Photometric Data Spectrophoto Models Spectrophoto Priors Single-Frame Fluxing Fluxed Frames Coaddition Global Fluxing Fluxed Spectra Single-Class Measurements Spectrum Templates & Priors Adam S. Bolton 1 D Outputs Classification & Redshifting DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 1. 8. 4. 1, 1. 8. 4. 2, 1. 8. 4. 3, 1. 8. 4. 5, 1. 8. 4. 8 8

Software Heritage • SDSS-I & -II (2000 -2008): - Groundbreaking redshift survey with fully automated calibration and analysis software Proof of BAO experimental technique - Successful adaptation of SDSS pipeline code to a more demanding S/N regime at higher redshifts Best current BAO cosmology results • SDSS-III BOSS (2008 -2014): • e. BOSS (2014 -2020): - Pushing to even more ambitious (and DESI-like) observations Significant new code development in DESI-applicable general form Great continuity of pipeline design & team from SDSS through DESI Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 9

DESI Pipeline WBS 1. 8. 4 Spectroscopic Pipeline Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 10

DESI Pipeline WBS 1. 8. 4 Spectroscopic Pipeline 1. 8. 4. 1 Develop initial algorithms 1. 8. 4. 2 Integrate and improve algorithms 1. 8. 4. 3 Testing infrastructure 1. 8. 4. 4 (L 2) Prelim. pipeline end-to-end on DESI sims 1. 8. 4. 5 Refine pipeline and algorithms 1. 8. 4. 6 (L 3) Pipeline progress review 1. 8. 4. 7 (L 3) Pipeline running at DESI scale 1. 8. 4. 8 Spectroscopic templates 1. 8. 4. 9 Large-scale structure catalog pipeline 1. 8. 4. 10 Spectroscopic data model 1. 8. 4. 11 Spectro pipeline development support 1. 8. 4. 12 Proto. DESI support Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 11

DESI Pipeline WBS 1. 8. 4 Spectroscopic Pipeline 1. 8. 4. 1 Develop initial algorithms 1. 8. 4. 2 Integrate and improve algorithms 1. 8. 4. 3 Testing infrastructure 1. 8. 4. 4 (L 2) Prelim. pipeline end-to-end on DESI sims 1. 8. 4. 5 Refine pipeline and algorithms 1. 8. 4. 6 (L 3) Pipeline progress review 1. 8. 4. 7 (L 3) Pipeline running at DESI scale 1. 8. 4. 8 Spectroscopic templates 1. 8. 4. 9 Large-scale structure catalog pipeline 1. 8. 4. 10 Spectroscopic data model 1. 8. 4. 11 Spectro pipeline development support 1. 8. 4. 12 Proto. DESI support Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 12

DESI Pipeline WBS 1. 8. 4. 1 Develop initial algorithms - Nov 2013 - Sep 2014 Prototype code DESI pipeline software Baseline algorithms & structure from BOSS pipeline Uniform adoption of Python and C/C++ for new code Completed “codefest” for intensive development “Spec. EX” 2 D PSF calibration prototype code “Harp” scalable parallel extraction prototype code Preliminary pipeline applied to BOSS data (L 4) “Specter” DESI raw-data simulation/extraction suite “Quicksim” DESI spectrum simulation suite “Redmonster” modular archetype-redshift software Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 13

DESI Pipeline WBS 1. 8. 4. 1 Develop initial algorithms - Nov 2013 - Sep 2014 “Harp” parallel extraction prototype code Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 14

DESI Pipeline WBS 1. 8. 4. 1 Develop initial algorithms - Nov 2013 - Sep 2014 “Spec. EX” 2 D PSF calibration prototype code Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 15

DESI Pipeline WBS 1. 8. 4. 1 Develop initial algorithms - Nov 2013 - Sep 2014 Preliminary pipeline applied to BOSS data (L 4) Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 16

DESI Pipeline WBS 1. 8. 4. 1 Develop initial algorithms - Nov 2013 - Sep 2014 “Redmonster” modular archetype-redshift software Adam S. Bolton Delta-chi-squared 25 km/s < vdisp < 800 km/s Simultaneous spectrum fit to multiple arms and exposures (BOSS DR 10 data) 0. 6289 < z < 0. 6317 DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 17

DESI Pipeline WBS 1. 8. 4 Spectroscopic Pipeline 1. 8. 4. 1 Develop initial algorithms 1. 8. 4. 2 Integrate and improve algorithms 1. 8. 4. 3 Testing infrastructure 1. 8. 4. 4 (L 2) Prelim. pipeline end-to-end on DESI sims 1. 8. 4. 5 Refine pipeline and algorithms 1. 8. 4. 6 (L 3) Pipeline progress review 1. 8. 4. 7 (L 3) Pipeline running at DESI scale 1. 8. 4. 8 Spectroscopic templates 1. 8. 4. 9 Large-scale structure catalog pipeline 1. 8. 4. 10 Spectroscopic data model 1. 8. 4. 11 Spectro pipeline development support 1. 8. 4. 12 Proto. DESI support Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 18

DESI Pipeline WBS 1. 8. 4. 2 Integrate and improve algorithms - Sep 2014 - Nov 2015 Prioritized improvement of existing elements Prototype pixel-flat, fiber-flat, and wavelength cal. Automation of inter-component connections Extractions & redshifts working on DESI sims (L 4) Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 19

DESI Pipeline WBS 1. 8. 4 Spectroscopic Pipeline 1. 8. 4. 1 Develop initial algorithms 1. 8. 4. 2 Integrate and improve algorithms 1. 8. 4. 3 Testing infrastructure 1. 8. 4. 4 (L 2) Prelim. pipeline end-to-end on DESI sims 1. 8. 4. 5 Refine pipeline and algorithms 1. 8. 4. 6 (L 3) Pipeline progress review 1. 8. 4. 7 (L 3) Pipeline running at DESI scale 1. 8. 4. 8 Spectroscopic templates 1. 8. 4. 9 Large-scale structure catalog pipeline 1. 8. 4. 10 Spectroscopic data model 1. 8. 4. 11 Spectro pipeline development support 1. 8. 4. 12 Proto. DESI support Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 20

DESI Pipeline WBS 1. 8. 4. 3 Testing infrastructure - Jun 2015 - Jan 2016 Schedule overlap with code improvement work Development of pipeline performance metrics Pipeline QA test development & completion (L 4) Engineering spectrograph support Preliminary end-to-end pipeline on DESI sims (L 2) Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 21

DESI Pipeline WBS 1. 8. 4 Spectroscopic Pipeline 1. 8. 4. 1 Develop initial algorithms 1. 8. 4. 2 Integrate and improve algorithms 1. 8. 4. 3 Testing infrastructure 1. 8. 4. 4 (L 2) Prelim. pipeline end-to-end on DESI sims 1. 8. 4. 5 Refine pipeline and algorithms 1. 8. 4. 6 (L 3) Pipeline progress review 1. 8. 4. 7 (L 3) Pipeline running at DESI scale 1. 8. 4. 8 Spectroscopic templates 1. 8. 4. 9 Large-scale structure catalog pipeline 1. 8. 4. 10 Spectroscopic data model 1. 8. 4. 11 Spectro pipeline development support 1. 8. 4. 12 Proto. DESI support Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 22

DESI Pipeline WBS 1. 8. 4. 5 Refine pipeline and algorithms - Feb 2016 - Dec 2017 Extensive refinement of all pipeline elements Focus informed by previous work Parallelization, integration, and deployment Pipeline progress review (L 3, early) Interface with WBS 1. 8. 3 (Survey) for “quicklook” extraction pipeline development (1. 8. 3. 3. 5) Production extraction code finished (L 4, mid) Production redshift code finished (L 4, mid) Full pipeline running at scale (L 3, late) Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 23

DESI Pipeline WBS 1. 8. 4 Spectroscopic Pipeline 1. 8. 4. 1 Develop initial algorithms 1. 8. 4. 2 Integrate and improve algorithms 1. 8. 4. 3 Testing infrastructure 1. 8. 4. 4 (L 2) Prelim. pipeline end-to-end on DESI sims 1. 8. 4. 5 Refine pipeline and algorithms 1. 8. 4. 6 (L 3) Pipeline progress review 1. 8. 4. 7 (L 3) Pipeline running at DESI scale 1. 8. 4. 8 Spectroscopic templates 1. 8. 4. 9 Large-scale structure catalog pipeline 1. 8. 4. 10 Spectroscopic data model 1. 8. 4. 11 Spectro pipeline development support 1. 8. 4. 12 Proto. DESI support Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 24

DESI Pipeline WBS 1. 8. 4. 8 Spectroscopic templates - Oct 2013 - Dec 2017 ELG, LRG, QSO, and Star classes Ongoing with pipeline software development Preliminary, Improved, and Production versions Close coordination with redshift algorithm work Multiple contributors enlisted (J. Moustakas, L 4) ‣ ‣ Synthetic stellar spectra (C. Allende-Prieto) Synthetic galaxy spectra (C. Conroy) Empirical ELG spectra from DEEP 2 (J. Moustakas) Quasar templates from SDSS (A. Slosar) Simulate spectra for all classes (L 4, fall 2014) Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 25

DESI Pipeline WBS 1. 8. 4. 8 Spectroscopic templates - Oct 2013 - Dec 2017 Empirical ELG spectra from DEEP 2 (J. Moustakas) Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 26

DESI Pipeline WBS 1. 8. 4 Spectroscopic Pipeline 1. 8. 4. 1 Develop initial algorithms 1. 8. 4. 2 Integrate and improve algorithms 1. 8. 4. 3 Testing infrastructure 1. 8. 4. 4 (L 2) Prelim. pipeline end-to-end on DESI sims 1. 8. 4. 5 Refine pipeline and algorithms 1. 8. 4. 6 (L 3) Pipeline progress review 1. 8. 4. 7 (L 3) Pipeline running at DESI scale 1. 8. 4. 8 Spectroscopic templates 1. 8. 4. 9 Large-scale structure catalog pipeline 1. 8. 4. 10 Spectroscopic data model 1. 8. 4. 11 Spectro pipeline development support 1. 8. 4. 12 Proto. DESI support Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 27

DESI Pipeline WBS 1. 8. 4. 9 Large-scale structure catalog pipeline - Oct 2013 - Dec 2017 Project-controlled software effort for LSS analysis Initial porting of BOSS/e. BOSS “mksample” code Significant scientific activity of clustering WG Coordination with other Data Systems elements: ‣ ‣ ‣ 1. 8. 2 (Target selection) 1. 8. 3. 2 (Fiber assignment) 1. 8. 6. 2 (Cosmological simulations) Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 28

DESI Pipeline WBS 1. 8. 4 Spectroscopic Pipeline 1. 8. 4. 1 Develop initial algorithms 1. 8. 4. 2 Integrate and improve algorithms 1. 8. 4. 3 Testing infrastructure 1. 8. 4. 4 (L 2) Prelim. pipeline end-to-end on DESI sims 1. 8. 4. 5 Refine pipeline and algorithms 1. 8. 4. 6 (L 3) Pipeline progress review 1. 8. 4. 7 (L 3) Pipeline running at DESI scale 1. 8. 4. 8 Spectroscopic templates 1. 8. 4. 9 Large-scale structure catalog pipeline 1. 8. 4. 10 Spectroscopic data model 1. 8. 4. 11 Spectro pipeline development support 1. 8. 4. 12 Proto. DESI support Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 29

DESI Pipeline WBS 1. 8. 4. 10 Spectroscopic data model - Sep 2014 - Nov 2017 Defines intermediate & final pipeline file format Preliminary redshift-template data model complete 1. 8. 4. 11 Spectro pipeline development support - Sep 2015 - Nov 2016 Project support for intensive development effort 1. 8. 4. 12 Proto. DESI support - Jun 2016 - Jan 2017 Reserved effort for proto. DESI Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 30

Level 3 Pipeline Milestones Apr 2014 Sep 2014 Jun 2015 Aug 2015 Dec 2015 Jan 2016 Sep 2016 Mar 2017 May 2017 Nov 2017 Preliminary pipeline applied to BOSS data Simulate DESI ELG, LRG, QSOs Spectral extractions working on DESI sims Redshifts working on DESI sims Prelim. pipeline end-to-end on DESI sims Pipeline performance QA tests written Pipeline progress review Production extraction code finished Production redshift code finished Pipeline running at scale Also note: planning periodic “Data Challenges” to verify progress towards system goals (J. Guy’s talk) Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 31

DESI Pipeline Key Personnel • Adam Bolton (University of Utah) - DESI Data Systems Lead Scientist; DESI pipeline L 3 Manager and lead redshift -code developer; BOSS & e. BOSS pipeline developer; DOE Early Career grant awardee • Stephen Bailey (LBNL) • David Schlegel (LBNL) • Ted Kisner (LBNL) • Julien Guy (LPNHE Paris / LBNL) • John Moustakas (Siena College) • Other significant subsystem contributors DESI Data Systems L 2 Manager; BOSS & e. BOSS pipeline developer DESI cognizant Project Scientist; SDSS & BOSS pipeline developer Lead DESI extraction code developer; HPC expert Lead DESI calibration code developer Lead DESI spectral template developer Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 32

DESI Pipeline Staffing Levels Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 33

Risk Items • PSF calibration from arc lines: - R 8 -01: Is PSF stable in time? R 8 -02: Do arc lines sample it well? - Extent of risk being identified early Lab-spectrograph & proto. DESI tests Lab tests show fiber optical stability Arc calibration work in BOSS/e. BOSS Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 34

DESI Pipeline Summary • Core mission for DESI Data Systems • Translate raw spectroscopic data into redshifts • A central item for DESI experimental success: optimal software needed to extract full information content delivered by DESI hardware, and to enable most precise & accurate cosmological analysis • Builds on proven design and team experience from SDSS, BOSS, and e. BOSS pipelines • Solid plan for construction will deliver working DESI pipeline system at first light Adam S. Bolton DESI CD 1 BO 3. 2 WBS 1. 8. 4 (Spectro Pipeline) 35