Ignacio Sevilla Noarbe CIEMAT Madrid on behalf of

Ignacio Sevilla Noarbe CIEMAT (Madrid) on behalf of the DES Collaboration From photons to catalogs

• Cosmological survey in visible/near IR light using 4 complementary techniques to Blanco 4 -meter at CTIO characterize dark energy: I. Cluster Counts II. Weak Lensing III. Large-scale Structure Credit: NOAO IV. Supernovae • Two multiband (photometric) surveys: 5000 deg 2 griz. Y to 24 th mag AB griz 10 -30 deg 2 repeat (SNe) • Build new 3 deg 2 FOV multi-CCD camera, Data management system, improve Blanco facilities 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 2/29

SDSS-II DES Area 10000 5000 Nb. of CCDs 22 62 Resolution 120 Mpix 570 Mpix Raw data/night 200 GB/night 300 GB/night Catalog size 18 TB 100 TB (est. ) Total data volume 60 TB 4 PB (est. ) Credit: Kotwani et al. (2010) 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 3/29

Transfer Process Archive Distribute 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 4/29

Transfer CTIO NCSA 300 GB/night in 18 h Process Archive Distribute 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 5/29

Transfer Process Orchestration: NCSA HPC nodes Archive Distribute 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 6/29

Transfer Process Archive Results Archive nodes Oracle DB Distribute 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 7/29

Transfer Process Archive Distribute Through web portals 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 8/29

l NCSA Team l l l l Bob Armstrong Bill Baker (Cristina Beldica) Ankit Chandra Greg Daues l Shantanu Desai Michelle Gower Robert Gruendl Joel Houston Wayne Hoyenga l Chit Khin Kailash Kotwani Terry Mc. Laren Don Petravick (Project Manager) Yuxuan Yang Institut d’Astrophysique l l Emmanuel Bertin l Fermilab Team l l Munich Team l l l Gurvan Bazin Art Carlson Joe Mohr (Project Scientist) SWG Development links l l Mike Jarvis + WL SWG (WL pipeline) John Marriner + SNe SWG (Diff Imaging) Molly Swanson + LSS SWG (Survey Masks) I. S. + LSS SWG (cosmic rays, satellites) SWG Testing links l l 11/08/2011 Huan Lin Svetlana Lebedeva Nelly Stanfield Douglas Tucker Clusters- Sarah Hansen, Wayne Barkhouse LSS- I. S. APS-DPF 2011 DESDM I. Sevilla 9/29

Exposure consists of 62+ CCD images – 570 Mpix - 3 deg 2 Survey is ~150, 000 100 -sec exposures over 525 nights Auxiliary CCDs record images for autoguiding and calibration X 300 + calib. = RAW DATA we send this to NCSA 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 10/29

Credit: NOAO Correct for cross-talk among CCDs. Correct for bias levels, non-uniformities, other optical and electrical effects. DETRENDED DATA 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 11/29

We need reference star catalogs Full focal plane is fit to single solution Correct optical distortion at focal plane Credit: E. Bertin We use Sextractor, SCAMP software by E. Bertin 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 12/29

Use standard star fields at different angles in the sky (X) with known fluxes (m_std and color_std) and relate with instrumental flux (m_inst): input output Make big least squares solution for a, b, k; apply results to science images. REDUCED DATA 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 13/29

At this point we have, for every night, approx. 300 exposures corrected by: • Instrumental effects • Absolute position • Absolute photometry This is the nightly processing. We store these in the archive (+ auxiliary images, info). 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 14/29

Go deeper; calibrate better BUT Point spread function is inhomogeneous: each exposure has different quality single exposure single exposure COADDED DATA 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 15/29

Find PSF in each image Homogeneize per coadd tile 0. 77 ’’ 0. 94 ’’ 1. 32 ’’ 0. 94 ’’ PSF HOMOGENEIZED COADDED DATA 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 16/29

It takes 30 x more time for coaddition with respect to nightly: • PSF has to be extracted • PSF has to be homogeneized • Actual addition of image and recomputation of errors Additionally: • Global photometric calibration among all images of the season We store these in the archive (+ auxiliary images, info). 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 17/29

This step is performed with the SExtractor package E. Bertin Position, shape and photometry is calculated. CATALOGS 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 18/29

Hundreds of millions of objects with hundreds of columns each: 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 19/29

Hundreds of millions of objects with hundreds of columns each: 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 20/29

Hundreds of millions of objects with hundreds of columns each: 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 21/29

Photometric redshift pipeline: take fluxes in five bands -> estimate redshift Photoz pipeline DESDM mag_band_g = 20. 7 (neural network) mag_band_r = 19. 2 mag_band_i = 18. 5 … + errors, other estimates (this only one kind of photo-z! More estimations foreseen) Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear. Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena. 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 22/29

Photometric redshift pipeline: take fluxes in five bands -> estimate redshift Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear. Eliminate instrumental signature DESDM Obtain true shape (intrinsic galaxy shape+ shear) WL pipeline local PSF Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena. 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 23/29

Photometric redshift pipeline: take fluxes in five bands -> estimate redshift Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear. Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena. Credit: Pan-STARRS 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 24/29

Produce cosmological simulations Process through atmosphere, detectors, include ‘nasty’ stuff 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 25/29

Galaxies Stars 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 26/29

BCS Images of First SPT Clusters Real data sets from the Blanco Cosmology Survey, SPT, SCS (Mosaic 2 camera). Large scale management with SDSS data. Credit: Blanco Cosmology Survey 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 27/29

The Dark Energy Survey (next talks!) will make use of a large, scalable data management system to process and archive raw images and science ready data products. Acceptance of the system is underway (results end of year). Tests on real DES data expected for first months of 2012. Raw and reduced images to be released yearly, catalogs at midpoint and end of survey. Community pipeline getting ready for usage of the DES camera starting May 2012. 11/08/2011 APS-DPF 2011 DESDM I. Sevilla 28/29

11/08/2011 APS-DPF 2011 DESDM I. Sevilla

l Science portal hosted by Brazilian collaborators to offer various analysis pipelines (to run on catalogs, images) and value-added catalogs.

n Development Funding ~$6 M n n n 11/08/2011 $4 million from NSF $1. 78 million (in kind) from NCSA/U Illinois, Fermilab, IAP and Munich $300 K from DES collaboration for Community Pipeline APS-DPF 2011 DESDM I. Sevilla