Database Workshop Report Eva Dafonte Perez ITDB Workshop

Database Workshop Report Eva Dafonte Perez IT-DB

Workshop Aims • Database Futures Workshop – 29 th-30 th May, 2 nd edition – https: //indico. cern. ch/event/615499/overview – 74 participants registered • Aims – – Discuss future requirements in the database area Identify common needs between user communities Evaluate new trends & technologies Understand how services should evolve/improve to fulfil new requirements 2

Relational Databases Distributed Databases (DBMS) Object Oriented Databases No. SQL Big Data (OODBMS) Object Relational Databases Analytics (ORDBMS) Relational Model Elastic. Search Postgre. SQL Oracle (RSI) 1970 Time Series Databases (TSDB) 1980 Hadoop My. SQL 1990 Oracle version 2. 3 (accelerator control) Database Futures workshop (1 st edition) 2000 Influx. DB 2010 Hadoop Service Influx. DB Elastic. Search Database On Demand My. SQL, Postgre. SQL 3

Tracks • Requirements for run 3&4 – Data volumes, update and access rates; application criticality; HA, replication needs, business continuity, cloud services, virtualization; machine learning; development and deployment process • Implementations & Technologies – Various database engines and technologies, innovative database applications; comparisons • Going beyond relational – Technologies and use cases: Big Data, Hadoop, Spark, … 4

Requirements for Run 3&4 • Traditional applications will still be key elements in the HL-LHC era • Increase of database applications to index events and analysis applications – Based on relational databases and No. SQL technologies • Overall the relational model is valid • Oracle is the preferred solution for critical applications – – Exceptions: ALICE (“zoo of solutions”), LHCb (mainly My. SQL) Cost-effective platform in terms of functionality and performance Expertise Support is a key factor 5

Requirements for Run 3&4 • Run 3&4 larger insert and update rates = database workload increase – Advance Oracle 12 c features (in-memory, new partitioning, …) • Migration to Oracle 12 c. R 2 in LS 2 – Powerful hardware to improve response times • More difficulties to schedule interventions – Move towards zero downtime – Fast-switching standby services • Alternatives to Oracle for smaller projects to facilitate – Collaboration with other institutes – Open sourcing 6

Implementations & Technologies • New systems/evolution – move towards No. SQL solutions – – – New accelerator logging service (NXCALS) Next generation archiver Next generation for Post Mortem event storage and analysis Conditions data management system for HEP experiments CMS Big Data project 7

8

Implementations & Technologies • Motivation – Scale out – Enable data analytics – Newer technologies more appropriate to solve specific use cases • No antagonism SQL vs No. SQL anymore • Risks in the medium term – Less interest / disappear – Difficult to maintain 9

Implementations & Technologies • Provenance – LHCb bookkeeping, CMS analysis, … – Integrate origin / meta information important for further analysis • Database on Demand – – Supports My. SQL, Postgre. SQL (relational) and Influx. DB (time series) Backup & Recovery, HA, Monitoring updates Working to offer instances in TN Help to use different DBMS • Open source tools available to facilitate migration • DBo. D team can be contacted 10

Going beyond relational • Data Analytics – Hadoop, Spark, Sqoop, Impala, Hbase, Hive and Pig • Centralised Elasticsearch service – Distributed, RESTful search and analytics engine • • Hadoop and Elastic. Search becoming critical to ATLAS Growing interest on Time Series databases – Easier analysis – Improved storage and ingestion rates – Influx. DB use cases: • DBo. D monitoring • IT monitoring • Streams processing – Kafka pilot service use cases: • Accelerator logging service • Computing infrastructure monitoring 11

In general… • Positive feedback on database services by IT • Fruitful discussions • Synergies (even overlaps) – Collaboration – Scope to optimise resources for all • Next similar workshop in 2019 (LS 2) – Given the dynamic nature of the technologies – Many projects in development 12

k n a Th ! u o y 13