Update on a New EPICS Archiver Kay Kasemir

Update on a New EPICS Archiver Kay Kasemir and Leo R. Dalesio 09/27/99

Outline • • Performance numbers for the data storage and retrieval. Disk Utilization Numbers Current Archiver User Interface for operation and data viewing Code Structure Data Managemente Conclusions

Performance Numbers - Archiving • These tests were performed on a 450 MHz Pentium. • 500 channels changing at 10 Hz and written every 15 seconds uses 20% of the CPU time during collection and 100% of the CPU time for 5 seconds while writing to disk. • 1, 000 channels changing at 10 Hz and written every 15 seconds uses 100% of the CPU continually, but allows the status viewer to run, but more slowly.

Performance - Retrieval • The retrieval tests were done in two stages. – 1) The data file was from the previous 12 months – The data was in over 1, 000 files (early files were made every 4 hours). – These files contained over 3 Gbytes – 2) The data was compressed and started from June 1 st – There were only 3 files - 1 per month. – These files were 1. 3 Gbytes • The retrieval tests were performed on a Sun Ultra 1 running Solaris.

Performance Numbers - Retrieval from over 1, 000 files # RFQHR_KDvr. In. Rfl_1_sig: 09/01/99 2 (0. 07) 09/01/99 00: 30. 000 08/01/99 11 (0. 19) 08/02/99 12: 43: 00. 000 07/01/99 6 (0. 19) 07/01/99 06: 35: 00. 000 06/01/99 9 (0. 27) 06/01/99 12: 40: 30. 000 05/01/99 18 (0. 25) 05/03/99 17: 28: 00. 000 04/01/99 15 (0. 36) 04/05/99 08: 10: 30. 000 03/01/99 85 (1. 12) 03/02/99 09: 59: 30. 000 02/01/99 98 (1. 02) 02/01/99 11: 46: 00. 000 01/01/99 48 (1. 1) 01/04/99 09: 25: 30. 000 12/01/98 63 (1. 16) 12/10/98 09: 31: 00. 000 11/01/98 58 (1. 13) 11/02/98 09: 40: 30. 000 10/01/98 1 (0. 04) 10/21/98 11: 04: 24. 844180846 # RFQHR_KCir. Rfl_3_sig: 09/01/99 1 (0. 12) 09/01/99 00: 00. 666656968 08/01/99 5 (0. 12) 08/02/99 12: 42: 30. 000 07/01/99 47 (0. 29) 07/01/99 06: 34: 30. 000 06/01/99 45 (0. 49) 06/01/99 12: 37: 30. 000 05/01/99 21 (0. 45) 05/03/99 17: 27: 30. 000 04/01/99 28 (0. 67) 04/05/99 08: 09: 00. 000 03/01/99 41 (0. 87) 03/02/99 10: 00. 000 02/01/99 42 (0. 76) 02/01/99 11: 44: 30. 000 01/01/99 43 (0. 82) 01/04/99 09: 25: 30. 000 12/01/98 56 (1. 07) 12/01/98 15: 08: 30. 000 11/01/98 48 (1. 03) 11/02/98 09: 40: 30. 000 10/01/98 1 (0. 06) 10/21/98 11: 04: 14. 427514796 # RFQLR_HPM_K 3_Lvl 06: 03/15/99 12: 05: 59. 287883068 - 09/09/99 02: 57: 30. 916664800 09/01/99 2 (0. 12) 09/01/99 00: 00. 616657362 08/01/99 6 (0. 13) 08/02/99 15: 19: 30. 000 07/01/99 3 (0. 1) 07/01/99 12: 44: 30. 000 06/01/99 1 (0. 07) 06/11/99 15: 04: 30. 000 05/01/99 0 (0. 07) 05/03/99 17: 27: 30. 000 04/01/99 3 (0. 14) 04/06/99 14: 09: 30. 000 03/01/99 0 (0. 05) 03/15/99 12: 05: 59. 287883068 # rp_bp_diag_07_ilock. A: 03/16/99 09: 39. 183333246 - 09/09/99 09: 22: 00. 000 09/01/99 0 (0. 05) 09/02/99 15: 29: 00. 000 08/01/99 0 (0. 07) 08/05/99 07: 02: 00. 000 07/01/99 1 (0. 08) 07/03/99 20: 33: 30. 000 06/01/99 1 (0. 09) 06/11/99 15: 03: 00. 000 05/01/99 0 (0. 08) 05/01/99 14: 31: 00. 000 04/01/99 1 (0. 07) 04/01/99 09: 58: 30. 000 03/01/99 0 (0. 05) 03/16/99 09: 39. 183333246 # RFQHR_KFil. Rmp_1_sig: 10/21/98 11: 04: 24. 844180846 - 09/09/99 09: 22: 00. 000 09/01/99 1 (0. 07) 09/01/99 07: 59: 30. 000 08/01/99 3 (0. 14) 08/02/99 12: 43: 00. 000 07/01/99 2 (0. 09) 07/01/99 06: 35: 00. 000 06/01/99 4 (0. 15) 06/01/99 12: 40: 30. 000 05/01/99 1 (0. 07) 05/03/99 17: 28: 00. 000 04/01/99 4 (0. 2) 04/05/99 08: 10: 30. 000 03/01/99 3 (0. 14) 03/02/99 09: 59: 30. 000 02/01/99 1 (0. 14) 02/01/99 11: 46: 00. 000 01/01/99 2 (0. 12) 01/04/99 09: 25: 30. 000 12/01/98 2 (0. 15) 12/10/98 09: 31: 00. 000 11/01/98 0 (0. 11) 11/02/98 09: 40: 30. 000 10/01/98 0 (0. 06) 10/21/98 11: 04: 24. 844180846

Performance Numbers - Retrieval from 3 files # RFQHR_KDvr. In. Rfl_1_sig: 08/01/99 0 (0. 13) 08/02/99 12: 43: 00. 000 07/01/99 1 (0. 25) 07/01/99 06: 35: 00. 000 06/01/99 0 (0. 04) 06/01/99 12: 40: 30. 000 # RFQHR_KCir. Rfl_3_sig: 09/01/99 0 (0. 14) 09/01/99 00: 00. 666656968 08/01/99 0 (0. 07) 08/02/99 12: 42: 30. 000 07/01/99 0 (0. 16) 07/01/99 06: 34: 30. 000 06/01/99 1 (0. 04) 06/01/99 12: 37: 30. 000 # RFQLR_HPM_K 3_Lvl 06: 09/01/99 0 (0. 13) 09/01/99 00: 00. 616657362 08/01/99 0 (0. 08) 08/02/99 15: 19: 30. 000 07/01/99 0 (0. 06) 07/01/99 12: 44: 30. 000 06/01/99 0 (0. 05) 06/11/99 15: 04: 30. 000 # rp_bp_diag_07_ilock. A: 09/01/99 0 (0. 06) 09/02/99 15: 29: 00. 000 08/01/99 0 (0. 05) 08/05/99 07: 02: 00. 000 07/01/99 0 (0. 05) 07/03/99 20: 33: 30. 000 06/01/99 0 (0. 04) 06/11/99 15: 03: 00. 000 # RFQHR_KCth. V_2_sig: 09/01/99 0 (0. 18) 08/01/99 1 (0. 11) 07/01/99 0 (0. 11) 06/01/99 0 (0. 06) 09/01/99 00: 00. 299990172 08/02/99 12: 42: 00. 000 07/01/99 06: 33: 30. 000 06/01/99 12: 38: 00. 000 # RFQLR_HPM_D 14 A_Fld. Cvt. VALA: 09/01/99 0 (0. 18) 09/01/99 00: 26. 666656320 08/01/99 0 (0. 05) 08/02/99 15: 19: 30. 000 07/01/99 0 (0. 06) 07/01/99 12: 44: 30. 000 06/01/99 1 (0. 06) 06/12/99 15: 31: 30. 534141192 # RFQHR_KFil. Rmp_1_sig: 09/01/99 0 (0. 05) 09/01/99 07: 59: 30. 000 08/01/99 0 (0. 05) 08/02/99 12: 43: 00. 000 07/01/99 0 (0. 06) 07/01/99 06: 35: 00. 000 06/01/99 0 (0. 05) 06/01/99 12: 40: 30. 000 # RFQWTR_A 23 H_slope_S: 09/01/99 0 (0. 07) 09/01/99 07: 58: 00. 000 08/01/99 0 (0. 06) 08/02/99 13: 00. 000 07/01/99 0 (0. 05) 07/01/99 07: 00. 000 06/01/99 0 (0. 05) 06/02/99 10: 36: 00. 000 # RFQWTR_WVF_B_RPM: 09/01/99 0 (0. 28) 09/01/99 00: 00. 616657716 08/01/99 0 (0. 13) 08/02/99 13: 00. 000 07/01/99 0 (0. 13) 07/01/99 07: 00. 000 06/01/99 0 (0. 05) 06/02/99 10: 36: 00. 000 # rp_bp_rf_08_ilock. A: 09/01/99 0 (0. 05) 08/01/99 1 (0. 06) 07/01/99 0 (0. 05) 06/01/99 0 (0. 04) 09/02/99 15: 29: 00. 000 08/05/99 07: 02: 00. 000 07/03/99 20: 33: 30. 000 06/11/99 15: 03: 00. 000

Performance - Retrieval • Retrieval on the second set of data was always within one second and therefore not an issue. • Retrieval from the first set was always reasonable for the first month - under 2 seconds of elapsed time. • Retrieval over longer periods became unacceptable for some portion of the test and then improved as more tests were done. • Perhaps we received more dedicated resources - perhaps the file structure had some problems - we do not understand the results yet. • As shown in previous tests - it is better to have fewer files.

Disk Utilization • Files are created every user configured period specified in hours. Perhaps this should have been expressed in days. • Within a file, the first buffer allocated will hold 64 samples, if this fills the second holds 256 samples and all subsequent buffers will hold 1, 024 samples. • The compression utility moves the data into one month files (currently hardcoded) and allocates buffers that contain up to 4, 000 samples. The data buffers are sized to fit the data exactly. • Every sample contains the value, time stamp, status and severity - nothing is done to save disk space.

User Interfaces to the Archiver • User interfaces are required for control and status from the Archiver as well as archive data viewing. • The first pass of the Archiver had an X windows user interface that was managed in the main line of the code. The limitations of this approach are that the Archiver cannot be started on a machine until the window system is available and this structure does not support management from multiple workstations. • The first archive data viewer (XARR from Jlab) is also X based and calls the retrieval library directly. The utility is UNIX based and requires that the disks containing the archive data be mounted. • Approaches were considered to provide network support for archive engine management and data viewing.

Alternatives Studied

Data Management Screens

Data Management Screens

Data Viewing Screens

Code Structure • Rewritten in C++ • Data and Channel Iterator Classes allow storage and retrieval code to share the same base class. • The viewing and extraction tools shown here use the iterator class and XARR is expected to convert to use it soon. • New data stores, like a relational database, could be used by replacing the Data and Channel Iterators. The data taking engine and retrieval tools would all be reusable for this alternative data storage.

Code Structure - Use the Iterator Class

Data Management • Currently there is a utility that allows a user to split an archive file. • The split utility compresses data by reducing the files to one every 3 months (could easily be configurable) and it compresses multiple data buffers for (64, 256, 1024) into buffers of 4, 000 samples. • True data management is required to all selective data winnowing and algorithmic compression. This work is expected to be completed in the next 12 months. • Supporting statistical compression would require significant changes all tools as each sample currently has just one value - need to allow for mean, median, standard deviation etc….