Metadata Requirements for EOSDIS Data Providers Siri Jodha

Metadata Requirements for EOSDIS Data Providers Siri Jodha Singh Khalsa khalsa@colorado. edu HDF-EOS Workshop II SJSK 1

Topics • Why metadata is important • Types of metadata in HDF-EOS files • Required metadata • How metadata is encoded and delivered HDF-EOS Workshop II SJSK 2

What is Metadata? • Metadata is information that identifies and characterizes an information product. • Sometimes called “data about data” HDF-EOS Workshop II SJSK 3

Users Need Metadata • Metadata is needed to answer questions such as: - What time and location does this data apply to? - Why type of instrument and processing produced the data? - What other inputs were used to generate the data? - What QA has been performed on this data? - Who do I contact if I have questions about this data? HDF-EOS Workshop II SJSK 4

Metadata is Essential • Large data archive systems cannot function without metadata. • Metadata is used to keep track of such things as: - where the data is what type of operations are possible on the data whethere any access restrictions on the data how individual data files are logically grouped into “collections. ” HDF-EOS Workshop II SJSK 5

Key Concepts • A granule is the smallest aggregation of data that is independently described and inventoried by the ECS. A granule consists of 1 or more physical files. • A collection is a logical grouping of granules. • The ECS Data Model allows for: - “Core” attributes - “Product-Specific” Attributes (PSAs) SJSK 6

Types of Metadata • Metadata in HDF files - stored as global text attributes • Types of Metadata used in HDF-EOS files: - Structural Metadata - Core Metadata (inventory, can include PSAs) - Archive Metadata (non-searchable, product-specific) • Collection level metadata - core and product-specific HDF-EOS Workshop II SJSK 7

Required Metadata • Origins of metadata requirements: - what is required to archive and retrieve files - what is required to provide search and other services on data - what is federally mandated (FGDC) • There are 287 attributes in the ECS data model - only a subset are used for any given product - 101 are applicable at the granule level HDF-EOS Workshop II SJSK 8

Metadata Coverage • Science Data that are delivered for archiving in ECS must meet what is called the Intermediate level of metadata coverage. This involves as few as: - 31 collection level attributes - 4 granule level attributes • Compliance at this level is not enforced by the system. HDF-EOS Workshop II SJSK 9

Collection-Level Metadata for Intermediate Coverage - Short. Name Long. Name Collection. Description Version. ID Archive. Center Revision. Date Version. Description Collection. State Maintenanceand. Update. Frequency ECSDiscipline. Keyword ECSTopic. Keyword ECSTerm. Keyword ECSVariable. Keyword Contact. Organization. Name Role HDF-EOS Workshop II - Spatial. Coverage. Type Point. Latitude Point. Longitude Time. Type Date. Type Temporal. Range. Type Precisionof. Seconds Endsat. Present. Flag Calendar. Date Timeof. Day Guide. Name Data. Center Document. Version Document. Updated Title Document. Created SJSK 10

Granule-Level Metadata for Intermediate Coverage • There are only four granule-level metadata attributes required: - Short. Name - Version. ID - Size. MBECSData. Granule - Production. Date. Time • Short. Name and Version. ID are identical to the collectionlevel attributes with these names. • For granules coming into ECS, Size. MBECSData. Granule and Production. Date. Time are supplied by the system upon insertion. HDF-EOS Workshop II SJSK 11

How is Metadata Supplied? • Collection-level metadata is carried in an Earth Science Data Type (ESDT) Descriptor file. • Granule-level metadata is defined in the descriptor file and populated using a Metadata Configuration File (MCF). • Granule-level metadata is delivered in the HDFEOS granule *or* in a populated MCF accompanying a non-HDF granule. • The DAAC where a collection will reside is responsible for descriptors and ingest routines. HDF-EOS Workshop II SJSK 12

Metadata Work Flow for External Data Providers HDF-EOS Workshop II SJSK 13

Metadata Resources on the Web • ECS Metadata Homepage http: //ecsinfo. hitc. com/metadata. html • Metadata Works (ESDT Descriptor Tool) http: //et 3 ws 1. HITC. COM/metadata_works/ • EOSDIS Information Architecture http: //spsosun. gsfc. nasa. gov/Info. Arch. html • Federal Geographic Data Committee http: //www. fgdc. gov/ SJSK 14

Q&A w/ Experts Panel • Q: “If you are a new data provider, how do you get your data into an HDF-EOS granule, given the bewildering array of utilities and tools available? What is the simplest solution for this? ” • A: The recommended solution is to obtain the HCR package, which includes the HDF-EOS and HDF libraries. For populating the required metadata in the granule, obtain the Metadata/Time Toolkit_MDT. The steps would be: 1. Write an HCR and use the tools to turn this into a skeletal HDF-EOS granule. (This step is optional). 2. Use the HDF-EOS library to create a granule. (If starting with a skeletal HDF-EOS file generated from an HCR then plain HDF calls can be used to insert data into the granule ). 3. Use Toolkit_MDT calls to insert metadata into the granule. This requires generation of an MCF in ODL. Metadata_Works is available for doing this. As an alternative, a simple HDF call can be used to attach minimum metadata (in ODL) to an HDF file. Note: if the data are going to reside in a DAAC, or in an archive that must be interoperable with ECS, you will need to generate collection-level metadata. Metadata_Works is the recommended tool for this. SJSK 15