USIT TSD Status and TSD API Maria Francesca

USIT TSD Status and TSD API Maria Francesca Iozzi, Leon Du Toit and many more …

TSD IN NUMBERS # projects vs. time 180 160 > 300 VMs (linux and windows, databases) 140 120 100 80 > 550 users > 1 Pi. B data circa 270 projects 60 40 20 0 5/5/14 8/5/14 11/5/14 2/5/15 5/5/15 8/5/15 2 11/5/15 2/5/16 5/5/16

Drift of TSD • Routines and Documentation • Monitoring / Logging (ELK) • Weekly on-duty watcher (high volume of tickets!) • Incident Reports • Operational Logs • Collaboration / communication across groups at USIT 3

TSD offer to the users • Platform for secure storage and analysis • HPC • Data Collection Tools / Nettskjema • Statistical Analysis Tools • Electronic Consent • Secure Data Collection from App (personalized medicine!) • Video and audio support 4

Bulk delivery of TSD to … • Hi. AO (80 projects) • HSØ (130 projects) • Ui. T (80 projects) • NTNU (80 projects) 5

Further developments • Developments driven by use cases • Developments financed by user communities Users want: • • • Fast / volume-intensive ingestion mechanism Data management tools Effective workflows involving video User friendliness Much more…. 6

Future • Reaching soon the limit of the current set-up: • Too many manual operations • More automation and serf-service needed 7

TSD API Architecture, features, roadmap

Architecture TSD project area client proxy api db

Architecture TSD project area client https proxy https api ssl db

Architecture TSD project area trusted client https auth proxy https auth api ssl authnz db

Architecture TSD ipad_1 ipad_2 ipad_n Underernœring Nettskjema nginx postgrest psql

Architecture data collection TSD ipad_1 Underernœring data analysis ipad_2 ipad_n Nettskjema nginx postgrest psql

Architecture Scalability, redundancy nginx project 1 project 2 HA proxy nginx … firewall …

Architecture storage Modularity psql files nginx HA proxy nginx export firewall stream psql

Value for researchers • No longer have to care about implementation details of data transfer or storage - focus on doing science • Enables a complete data workflow, automation • Authenticated, secure and reliable data transfer • Persistent and redundant storage

Value for client developers • an API o/ • Supports development of end-user solutions (both collection and analysis sides) • multiple data types (JSON, files, arbitrary data streams) • self service (client registration, get approved, read the docs and go) • test environment outside of TSD - offers the same features and interface as in production

Roadmap • Improved operational readiness • Automated deployment • Monitoring and alerting • Documentation (for developers and maintainers) • Risk analysis

About export • Data export - with Bank. ID and Nettskjema

Next • Integration with a 3 rd party developer (JSON data) • Files from medical instruments • video streaming from psychological interviews