NLC The Next Linear Collider Project NLC Archiving

NLC - The Next Linear Collider Project NLC Archiving Requirements (Preliminary) Lee Ann Yasukawa 05/25/99

NLC - The Next Linear Collider Project Why does the NLC need a new Archiver? • Since the NLC will be such a complex machine, there needs to be greater flexibility in the data archived than is currently available in order to troubleshoot problems • The archived data for some devices need to be on the scale of milliseconds for a short period of time (seconds) while data for other devices may be on the scale of minutes over a longer period of time (10’s of minutes to hours) • Data viewing from many different control system applications should have a consistent interface: such as archived data, correlation plots data, buffered data acquisitions, configuration files, etc. Lee Ann. Yasukawa 05/25/99 2

NLC - The Next Linear Collider Project Why does the NLC need a new archiver (cont)? • Collaborators will be internationally based so access to data must be accessible from around the world • The amount of data is huge (not including fast-timed based archives): 600, 000 pvs x 80 bytes/pv x 20 sets of data/hour ----------------~1. 0 Gbyte/hour x 24 hours ----------------24 Gbytes/day Lee Ann. Yasukawa 05/25/99 3

NLC - The Next Linear Collider Project Requester and Archiver Requester • Interface between user and archiver • Supports time-based entries, monitorbased entries, triggered entries, timebased + monitor-based entries and fasttimed entries (ring buffers) Archiver • Retrieves pv data at specified rates or monitor intervals and stores the data in files quickly and efficiently • Limited ring buffer capability • Validates requests • Ability to create “other” data files for configuration data, correlated data, etc. • Bookkeeper of archived channels/rate/requesters • Must be able to handle up to ~600, 000 pvs • Limits requests when necessary • Much of the data will come in bursts instead of regular intervals • Generates diagnostic reports Lee Ann. Yasukawa 05/25/99 4

NLC - The Next Linear Collider Project Data Converter and Data Retriever Data Converter • Runs periodically to convert data from fast file save format to fast file access format • Not part of the archiver • Ages and compresses data when necessary • Calculates statistics of compressed data • Keeps master index of data locations Data Retriever • Accepts requests to retrieve data • Validates requests • Utilizes master index to locate data • Can process data in fast file save format and also fast file access format • Can retrieve data from archiver, correlation plots, buffered data acquisitions and configuration files • Sends data back to requester Lee Ann. Yasukawa 05/25/99 5

NLC - The Next Linear Collider Project Browser • User interface for data retrieval • Web based component • Capability to display graphs and histograms • Capability to perform simple calculations (mean, std dev, etc. ) and simple fits • Ability to zoom, pan and lasso graphics • Ability to export data to various output formats (MATLAB, spreadsheets, etc) • Ability to display time-based data, correlated history data, configuration data, etc Lee Ann. Yasukawa 05/25/99 6

NLC - The Next Linear Collider Project Requester/Archiver Questions Requester Questions: • Can anyone make archiver requests? Should there be special privileges for different types of requests (based on rate? Subsystem? ) • Do people based internationally need this capability? Should it be web-based? Archiver Questions? • How closely sync’d in time do the IOCs need to be? • What are the requirements for fast-time based entries (ring buffers)? How many data points should be available? What is the maximum frequency which needs to be sampled (based on hardware requirements)? Lee Ann. Yasukawa 05/25/99 7

NLC - The Next Linear Collider Project Data Converter/Data Retriever Questions Data Converter Questions: • How is data aged? Should statistics be kept on all aged data? If so, what type? (May depend on hardware requirements). • Do users have input as to how much data is kept? • How long should data be kept available online? • How often should data be converted? Once daily? More often? • Performance requirements? Data Retriever Questions: • What are the performance requirements? How much data, how fast so that it is acceptable to users? Lee Ann. Yasukawa 05/25/99 8

NLC - The Next Linear Collider Project Browser Questions • Does the browser need security imposed or can anyone request data? • What are the display performance requirements? How many pvs? How many graphs? Timing? • How sophisticated should the calculation capability be? Can the browser do simple calculations and simple fits and rely on exporting data for sophisticated calculations and fits? • Or, do we want the browser to encompass everything the user wants to do with the data? Should MATLAB (or some other calculation engine) and some spreadsheet program be integrated into the browser? Lee Ann. Yasukawa 05/25/99 9

NLC - The Next Linear Collider Project Technical Challenges • Optimizing algorithms which stores data and accesses data fast and efficiently • Finding storage mediums to handle the volume of data which will be archived and that is capable of meeting performance requirements • Creating a robust system with built-in redundancies which can handle failovers seamlessly • Designing a system which meets requirements now yet is flexible since the requirements will inevitably change over time Lee Ann. Yasukawa 05/25/99 10