Managing Data Quality in a Terabytescale Sensor Archive

Managing Data Quality in a Terabyte-scale Sensor Archive Bryce Cutt, Ramon Lawrence University of British Columbia Okanagan Kelowna, British Columbia, Canada ramon-lawrence@ubc. ca http: //people. ok. ubc. ca/rlawrenc/

Data-Driven Scientific Discovery Modern scientific discovery uses vast quantities of data generated from instruments, sensors, and experimental systems. u. The quality and impact of the research is highly dependent on the quality of the data collection and analysis. Challenges: u. The amount of data required for research is exploding. u. The types and sources of data is increasing and data generated by different experiments or devices must be integrated. These two factors require scientists to be concerned with how their experimental data is collected, archived, analyzed, and integrated to lead to research contributions. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 2

Fundamental Sensor Data Archive Issue Sensors produce vast amounts of data that is valuable for historical as well as real-time applications and analysis. Due to the number of sensors and volume of data collected, manual curation and data validation is difficult. By their nature, sensors are prone to failures, inaccuracies, and periods of intermittent or substandard performance. Despite this, the historical data record should be as clean and accurate as possible despite device limitations. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 3

Key Question (and Answer) Question: How can we achieve high quality historical archives of sensor data? Answer: In addition to operational monitoring of the data archive system, the data stream should be analyzed using metadata properties to detect errors. Operational monitoring – Are the system components and workflow functioning properly? Metadata validation – Does the data stream conform to known ranges? Can data cleansing and correction be performed? Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 4

NEXRAD Archive System Overview Our goal is to provide the science community with ready access to the vast archives and real-time information collected by the national network of NEXRAD radars. [This requires hiding the numerous data management issues. ] We will briefly overview: u. The data collected by the NEXRAD system and its scientific value. u. The current state of NEXRAD data archiving and its use in scientific discovery, including its data quality limitations. u. An extension of the system that uses metadata properties to validate and clean archived data. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 5

NEXRAD System and Generated Data There are over 150 NEXt generation RADars (NEXRAD) that collect real-time precipitation data across the United States. u. The system has been operational for over 10 years, and the amount of collected data is continually expanding. A radar emits a coherent train of microwave pulses and processes reflected pulses. Each processed pulse corresponds to a bin. There are multiple bins in a ray (beam). Rotating the radar 360º is a sweep. After a sweep the radar elevation angle is increased, and another sweep performed. All sweeps together form a volume. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 6

Usefulness of NEXRAD Data Although the NEXRAD system was designed for severe weather forecasting, data collected has been used in many areas including: uflood prediction ubird and insect migration urainfall estimation The value of this data has been noted by a NRC report which labeled it a “critical resource. ” Enhancing Access to NEXRAD Data—A Critical National Resource. National Academy Press, Washington D. C. ISBN 0 -309 -06636 -0, 1999 Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 7

Archiving NEXRAD Data Researchers have two options for acquiring NEXRAD data: u 1) Retrieve RAW data from the National Climatic Data Center (NCDC) tape archive. u 2) Capture real-time data distributed by University Corporation for Atmospheric Research (UCAR) using their Unidata Internet Data Distribution (IDD) system. Acquiring, archiving, and analyzing the data requires significant computational and operational knowledge which makes it impractical for many researchers. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 8

NEXRAD Archive System The NEXRAD archive system is a NSF-funded project that aims to simplify the analysis of NEXRAD data for researchers. The NEXRAD archive: u. Collects and archives RAW data from the real-time stream. u. Analyzes and indexes data for retrieval by metadata properties. u. Performs data cleansing such as removing ground clutter. u. Allows researchers access to historical and real-time data in RAW form. u. Provides an analysis workflow system that will generate derived products (such as rainfall maps) using the RAW data, known algorithms, and researcher parameters. The NEXRAD archive is hosted at the University of Iowa and the development is done in conjunction with NCDC, Unidata, and Princeton University. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 9

NEXRAD Archive Architecture u Files are added from real-time stream or from other sources. u Metadata extractor produces XML description of each data file used for indexing. u Clients can access archive directly using C library and their own program. ð All data files are web accessible. ð Metadata directory can be queried using web services interface. u Most clients use pre-constructed web workflow system and do not access RAW data. u Data and metadata will be replicated at a supercomputing center and eventually at NCDC. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 10

User/Client’s View “Find all the 2002 storms over the Ralston Creek watershed with mean areal precipitation greater than X mm, and with a spatial extent of more than Z km 2, with a duration of less than N hours. I want the data in Geo. TIFF. ” Metadata Archive “Find all the 2002 storms over the Ralston Creek watershed with mean areal precipitation greater than X mm, and with a spatial extent of more than Z km 2, with a duration of less than N hours. I want the data in Geo. TIFF” Query Metadata Archive User/Client Distributed Data Archive Get URIs Get data Web Services (NCDC, Iowa, etc. ) HTTP Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 11

NEXRAD System Current Statistics The NEXRAD archive system: u. Has been running for over 2 years u. Collects data for 30 of the 150 radars u. Has indexed over 8 million radar scans u. Has RAW data that is over 8 TB in compressed form u. Processes real-time data stream of 10 -20 GB/day u. Supports a sophisticated workflow system that produces derived data products (e. g. rainfall maps) for users on demand u. Has an operational monitoring system (is the archive workflow pipeline functioning properly? ) but only simple data validation checks Question: What is the quality of the data being archived? Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 12

Archive Monitoring System We developed a new archive monitoring system that: u. Explicitly tracks all archive workflow events in logs that are stored and queried using a database u. Detects data corruption using metadata properties as well as pipeline failures u. Produces reports on a web interface to simplify the task of administration of the archive The monitoring system was developed and operated separately from the main archive to compare performance and to prevent issues with the operational system. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 13

Archive System with Monitor Basic archive workflow components unchanged except for logging: u Converter – translate RAW form to compressed RLE u Metadata Extractor – analyze data properties/check for inconsistencies u Loader – load metadata into database and files onto web servers Monitoring system: u Loads XML log records from each archive component into DB. u Provides metadata ranges for checking data validity. u Tracks files through pipeline (lineage) and handles corrupt files. u Has separate log database that is accessed using web front-end. u Can restart any workflow software. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 14

Validating Sensor Data using Metadata Operational ranges of data produced by sensors is commonly known. u. For example, the timestamp of sensor readings for the radars should be close to the current time. Reflectivity readings are within known ranges given weather conditions. Monitoring system provides these operational ranges to the metadata extractor component that can verify that data is within accepted ranges. Data outside ranges causes files to be dropped if not recoverable, fixed if possible (date changes), or flagged as warnings of potential corruption otherwise. The goal is to get as much data through the pipeline as possible, but make sure compromised data is flagged. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 15

Monitoring System Implementation The monitoring system required that each workflow component be changed to use XML log records instead of separate files. Each XML log record is loaded into a Postgres database by a log processor. The log processor and logging is separate from the archive system to ensure that logging does not slow the archiving. As a RAW file proceeds through the pipeline, log events for it are recorded at each stage. Files that do not make it through the pipeline are not “lost” to the archive as before. Administrator has a web front-end to control archive processes, monitor events on per file or per process level, and track operational characteristics across the entire workflow. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 16

Monitor System Administrator Interface Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 17

Operational Results A duplicate NEXRAD archive system that included the monitoring system processed two radars in parallel with the live system for six months. Key results: u. Data errors occur in about 5% of input files. Expectation was less than 1% given sophistication of sensors. ð One radar had data corruption for a two week period that went unnoticed in the live system as radar was indicating good operational status. ð Most errors were not fixable, but significant # of correctable date errors. u. Administrator time reduced dramatically compared to manual log investigation. u. The cost of logging a sensor stream is high. Storing log records in a database is a bottleneck and must be separated from archive. (Database loading is also a bottleneck in archive itself. ) Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 18

Future Work and Conclusions Archiving sensor data is going to be an increasing challenge. u. Ensuring high quality archives requires more than operational monitoring and should also include data validation using metadata properties. Live archive system is being updated to use monitoring system. The bottleneck in archiving and monitoring sensor data is the database system. Monitor should be separated from archive. u. Loading the metadata into the archive takes an order of magnitude longer than generating it. u. Metadata is growing beyond the capabilities of a single database. It will be replicated and distributed for performance and political reasons. u. Logging to the database provides easy access to information, but you must be aware of performance issues. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 19

Project Participants The University of Iowa (Lead) u. W. F. Krajewski (PI) u. A. A. Bradley, A. Kruger, R. Lawrence Princeton University u. J. A. Smith (PI) u. M. Steiner, M. L. Baeck National Climatic Data Center u. S. A. Delgreco (PI) u. S. Ansari UCAR/Unidata Program Center u. M. K. Ramamurthy (PI) u. W. J. Weber Research supported by NSF ITR Grant ATM 0427422: “A Comprehensive Framework for Use of NEXRAD Data in Hydrometeorology and Hydrology”. Ramon Lawrence Managing Data Quality in a Terabytescale Sensor Archive – SAC 2008 Page 20

Managing Data Quality in a Terabyte-scale Sensor Archive Bryce Cutt, Ramon Lawrence University of British Columbia Okanagan Kelowna, British Columbia, Canada ramon-lawrence@ubc. ca http: //people. ok. ubc. ca/rlawrenc/ Thank You!