NCSA Plan for DPP Construction and Operations of

NCSA Plan for DPP: Construction and Operations of the LSST Data Facility Don Petravick, Paul Domagala, Margaret Johnson Joint Technical Meeting March 8, 2017 JTM 2017 • Glendale, CA • March 6 -8, 2017 1

DPP is a kind of crossroads which couples Sci Ops, Obs Ops, Biz Ops, and EPO JTM 2017 • Glendale, CA • March 6 -8, 2017 2

Analogy: Transportation JTM 2017 • Glendale, CA • March 6 -8, 2017 3

JTM 2017 • Glendale, CA • March 6 -8, 2017 4

JTM 2017 • Glendale, CA • March 6 -8, 2017 5

JTM 2017 • Glendale, CA • March 6 -8, 2017 6

JTM 2017 • Glendale, CA • March 6 -8, 2017 7

JTM 2017 • Glendale, CA • March 6 -8, 2017 8

JTM 2017 • Glendale, CA • March 6 -8, 2017 9

Service Management − The way of organizing and managing what the computing and data systems do for you. – It is tied to use cases DPP satisfies – It is used internally within DPP – ITIL is a service-oriented operations framework. – Diverse services in a common model/govenance framework. – Guidance – Concepts + Vocabulary – Processes and roles – Basis for supporting management tooling. JTM 2017 • Glendale, CA • March 6 -8, 2017 10

Map the Requirements onto a Regular Operational Framework Some aspects of technical design JTM 2017 • Glendale, CA • March 6 -8, 2017 Regular planning and management schema 11

LSST/DPP structure Specific Use Cases The structure accommodates all use cases, driven by science requirements. Supports Sci Ops, Obs Ops, Biz Ops, EPO, and General Infrastructure. JTM 2017 • Glendale, CA • March 6 -8, 2017 12

LSST/DPP structure All work specific to satisfaction of a Use Case Support Many Use Cases General Capabilities Three Layer Business Architecture JTM 2017 • Glendale, CA • March 6 -8, 2017 13

LSST/DPP structure Components JTM 2017 • Glendale, CA • March 6 -8, 2017 14

LSST/DPP structure Cross-cutting elements provide for overall management and evolution of the system. Crosscutting JTM 2017 • Glendale, CA • March 6 -8, 2017 15

What’s visible to you, outside of DPP? − The main agreement is in Service catalogs/Service agreements − What are we providing? − What use case(s) does it support? − Who are we doing it for? − SLT – How well do we need to do it? JTM 2017 • Glendale, CA • March 6 -8, 2017 16

Service Level Agreements (SLAs) - - Detailed contract with the customers of a specific service that describes: - Hours of operation & support - Metrics that need to be achieved - Where to find information about the service - How we measure performance - Service history It has an approval process It is under change control JTM 2017 • Glendale, CA • March 6 -8, 2017 17

Example: Spectrograph Commissioned Use Case Support commissioning of the spectrograph. DPP-offered Services Archive the spectrograph data. Extract, transform and load the EFD databases and large file annex. Provide L 1 complete test stand. Reusable Services Deploy instance of archiver. (EFD is its own thing) Batch processing service. Data, Compute and IT Security Services File mirroring between Chile and NCSA; managed database services for file system metadata and for transformed EFD; batch system; authentication and authorization; network-based IT security; disaster recovery for spectrograph data and file annex. ITC Data acquisition at Base; Data Backbone at Base, Data Backbone at NCSA; batch platform. Software L 1 archiving software, EFD ETL software, file management software, workload management system. Mo. U North-south networking; offsite DR. JTM 2017 • Glendale, CA • March 6 -8, 2017 18

Why IT Service Management − Repeatable process − Continuous improvement − Increased quality of service and reliability − Decreased cost − ITSM is to IT services what EVPM is to projects JTM 2017 • Glendale, CA • March 6 -8, 2017 Replace unstructured work patterns with intelligent workflows (Process) IT Service Management Incident Management Problem Management Change Management Release Management Asset Management Service Catalog Knowledge Management IT Operations Management Discovery Service Mapping Event Management Orchestration Cloud Management Configuration Management Database IT Business Management Project Portfolio Management Demand Management Resource Management Application Portfolio Management Financial Management Agile Development Performance Analytics 19

Summary − Operations has defined an organization called “DPP” (soon to be the “LSST Data Facility and Operations”). – One set of cases is pipeline code from Science Operations. – DPP supports every other operational department in some way. – A detailed WBS is created, using a layered architecture and a standard set of operations concepts. − Services are stood up incrementally – Some services are already being provided. – There is no one grand integration. JTM 2017 • Glendale, CA • March 6 -8, 2017 20

Extra Slides JTM 2017 • Glendale, CA • March 6 -8, 2017 21

Constructing Processes • Draw the process, vet with stakeholders, revise. Repeat as necessary • Exercise/stress-test & revise again • Find a good production-quality tool & implement it. JTM 2017 • Glendale, CA • March 6 -8, 2017 22

JTM 2017 • Glendale, CA • March 6 -8, 2017 23

JTM 2017 • Glendale, CA • March 6 -8, 2017 24

A common IT change management process How W. Edwards Deming suggested we do things (yes, he was correct) JTM 2017 • Glendale, CA • March 6 -8, 2017 25