From the Channel Archiver to the Best Ever

From the Channel. Archiver to the Best Ever Archive Utility, Yet kasemirk@ornl. gov October 2009

Channel Archiver “ASCII” Config. IOC Channel Access config. xml Archive Engine Binary Data Files Data Server XML-RPC CSS-based OPI 2 Managed by UT-Battelle for the U. S. Department of Energy History ~2000: Started by Bob Dalesio ~2003: Index Tools, Data Server ~2007: CSS Client

Problems with old Channel Archiver • Data file format optimized to write many samples quickly – More than 40000/second –. . but we only used maybe 1000/sec –. . and had many ill-configured or duplicate channels • Headaches with data maintenance: – – – – 3 Scripts to restart engines, copy data, update indices. Index time grows with data Stuck when index files reach 2 GB SNS Users faced with ~80 sub-archives No clue what needs fixing after network/power problems No idea who contributes how many samples No way to remove selected channels or time ranges Improving on this means implementing an RDB Managed by UT-Battelle for the U. S. Department of Energy

New Setup IOC Channel Access Archive. Engine config. xml Engine. Config -Import Config. Samples RDB (Oracle/My. SQL) CSS-based OPI 4 Managed by UT-Battelle for the U. S. Department of Energy Other tools for config & samples

CSS Data Browser Handles Both • New URL • Just one ‘RDB’ sub-archive • Old and new data can be combined in one plot 5 Managed by UT-Battelle for the U. S. Department of Energy

Web Interface to Engine Config • Tomcat/JSP/Servlets to view and edit 6 Managed by UT-Battelle for the U. S. Department of Energy – Part of reporting system; somewhat specific to Oracle and SNS

Stats • Host that runs sample engines: – CPU load 45%, zero disk I/O wait, very responsive • Oracle Setup – – Cluster Sample tables partitioned by day 8000 samples/sec peak in write tests Operationally maybe ¼ of that • Better configuration would likely have fewer samples/sec 7 Managed by UT-Battelle for the U. S. Department of Energy

Web Config View: Channel Stats 8 Managed by UT-Battelle for the U. S. Department of Energy

Web Config View: Sample Stats 9 Managed by UT-Battelle for the U. S. Department of Energy

Viewing the Data Almost every software technology can access data in RDB ü Automated weekly performance plots ü Custom reports for - and done by various SNS groups (SNS Beam Instrumentation: Mariano Padilla, D. J. Edwardson) 10 Managed by UT-Battelle for the U. S. Department of Energy

Main User Interface: CSS Data Browser • Automatic switch from min/max/average to ‘raw’ data • This operation is currently quite slow – Old “Network Data Server” ran min/max/avg. on server side – RDB code for now in client. Working on stored procedure. 11 Managed by UT-Battelle for the U. S. Department of Energy Zoom in…

RDB: Perfect except for retrieval speed • Fermilab’s archiver, Jlab’s My. A – Wrapper code around My. SQL to create ‘cluster’, or to store data compressed into BLOBs, i. e. access is not plain RDB. • BNL Ideas: – Gabriele Carcassi mentioned RDDTool – Mauro Giacchini investigates Hypertable All faster than plain RDB for basic read/write. New sample engine and Data Browser could be adapted to them. Still, both loose advantages of plain RDB access to data and configuration 12 Managed by UT-Battelle for the U. S. Department of Energy

Summary, Status • At SNS, BEAUt. Y replaced Channel Archiver in July – Before, parallel operation for ~2 month • About a year of testing, many Oracle setup issues – Oracle cluster setup – Updated partitioning • Next – Stored procedure for retrieval – Data reduction: Replace Oracle partitions of old data with reduced channel/sample count 13 Managed by UT-Battelle for the U. S. Department of Energy