Ground Systems Data in the World Data Center

Ground Systems Data in the World Data Center Archives Justin Mabie (NGDC) and Anatoly Soloviev (GCRAS) 2 July 2011 1

Ground System Data Holdings Solar Indices Ionosonde Magnetometer 2

Old School Data in Climate Controlled Storage Some are neat, some are a mess, some are so old our computers can’t read them. Boxes and folders and files – oh my! 3

In the NGDC Warehouse: To get space in the warehouse, we spend countless hours bringing the data together on a single medium and creating inventories and metadata records. In the past, these data have been migrated to film or packed away in boxes. Modern digital data is packed away neatly in our ADIC tape library system, which is cheaper, less labor intensive, and does not require a large climate controlled warehouse. 4

Geomagnetic Data Ingest System NGDC workflow for geomagnetic observatory data 5

Solar Data Ingest System NGDC workflow for solar observations

ISO Metadata Workflow Metadata is a real challenge. This workflow shows how NGDC ingests metadata, performs quality control, ports it to the ISO metadata standard, loads it into databases, and disseminates it to our customers.

Ionosonde – User Services Real Time Applications Users Include NWS SWPC USAF Weather Agency Research Lab Northwestern Research Associates Australia Ionospheric Prediction Services Univ. MA, Lowell Center for Atmos. Research Space Environment Technologies

GRAILS/ORACLE Databases for Metadata NGDC is developing databases so our users can make use of our metadata in ways we can’t think of. We traditionally focus on the needs of the environmental scientist – what about the instrument and hardware developers, the social scientist, the politicians? For further information, contact: geomag. obs@noaa. gov 9

XML – Metadata Format 10

CDMP – Climate Data Modernization Project CDMP Data Recovery for Magnetometer, Solar, Ionosonde, and Airborne Imagery Example – Geomag: Title: CDMP - Geomagnetism Volume: Approximately 4 TB per year of data Storage: ADIC tape library Dissemination: Reduced resolution images will be put on FTP site; full resolution data available by request. Notes: v. Data volume is very large (many terabytes) v. Full filenames are included to reduce NGDC overhead (lesson learned from earlier projects) STATUS: All efforts De-Funded due to fiscal crisis. Procedure is in place, providing an opportunity for future funding. v. Holdings range in date from 1957 – 1983 for Geomag. v. NGDC film magnetograms were on track for completion in 10 years. Solar data is near completion, and Ionosonde data is aprroximately 10% modernized. 11

Geomagnetic and Solar Indices Bulletins Title: Geomagnetic Indices Bulletin Description: Monthly publication containing Kp, Ap, Cp, An, As, Am. http: //ngdc. noaa. gov/stp/geomag/indices. html Solar Indices bulletin contains Sunspot Number, Sunspot Prediction and Solar Radio Flux data. DATA SETS • Printed for research purposes Media: online only NGDC Holdings: 1996 – Present URL: ngdc. noaa. gov/stp/geomag/geoib. html ftp. ngdc. noaa. gov/STP/SOLAR_DATA/Publications/ 12

Geomag Observatory Metadata Form Page 1/5 Page 2/5 Return XML Page 13

! oon s g omin C RSS feeds to notify users of product changes What is RSS? RSS is a way for a user to automatically retrieve information from selected websites and aggregate it. The feeds (in XML format) are collected with an RSS reader. Readers for RSS include email applications such as Mozilla Thunderbird and fancier readers such as Google Reader. Subscribers to RSS feeds maintain their privacy. How will RSS feeds be used at NGDC? Users will select RSS feeds for specific data types. Subscribers will get occasional automatic alerts for: format changes reprocessed data sets new data sets 14

Thank You! 15