Tutorials on Data Management CC image by bonus

Tutorials on Data Management CC image by bonus on Flickr Lesson 7: Metadata What is Metadata

Lesson Topics • Definition of metadata • Examine information included in a metadata record • Examples of metadata standards and how to choose • Illustrate the value of metadata to data users, data providers, and organizations CC image by Alec Couros on Flickr • Describe the utility of metadata for a variety of scenarios What is Metadata

Learning Objectives After completing this lesson, the participant will be able to: • Define science metadata • Give examples of metadata that you are likely to encounter in the ‘real world’ (i. e. , outside of a research context) • Identify and list the types of information typically included in metadata records for environmental datasets • Identify 3 reasons metadata is of value to data users, data developers, and organizations • List 3 uses for metadata, beyond discovery of data • Identify and describe factors that may determine which metadata standards are most appropriate for a given dataset What is Metadata

The Data Life Cycle Plan Collect Analyze Integrate Assure Discover Describe Preserve What is Metadata

What is Metadata CC image by ISAS on Flickr CC image by kukkurovaca on Flickr CC image by acordova on Flickr CC image by SEDAC on Flickr CC image by Justin See on Flickr CC image by CIMMYT on Flickr Data Collection

From Field Notes to Datasets Average Temperature of Observation for Each Species Average Temperature Standard Deviation Number of Observations Minimum Temperature Maximum Temperature Northern Red-legged Frog 4. 4 --- 1 4. 4 Tailed Frog 7. 0 3 4 10 Arizona Toad 10. 0 --- 1 10 10 Strecker's Chorus Frog 10. 5 2. 0 11 9 16 Oregon Spotted Frog 11. 0 15. 5 2 0 22 New Jersey Chorus Frog 11. 5 4. 5 17 3 22 Wood Frog 12. 5 5. 5 897 0 28. 8 Spring Peeper 13. 2 5. 6 569 -1 32 Red-legged Frog 13. 3 5. 9 16 4 27 What is Metadata

CC image by Heather Kennedy on Flickr From Datasets to Published Papers What is Metadata

What is a Dataset? • Definition: A collection of data • Generally datasets can be defined as: o Spatial – a collection of logically related features arranged in a prescribed manner such as GIS map layers, water features, etc o Tabular – a file, spreadsheet, data in a table o Many tabular datasets are inherently “spatial”, e. g. water-quality samples associated with stream collection sites • Elements in a dataset can include: o Values, measures, points, coordinates, conditions, qualities, frequencies, or attributes that are a result of an observational study What is Metadata

Working with Data • When you provide data to someone else, what types of information would you want to include with the data? • When you receive a dataset from an external source, what types of details do you want to know about the data? What is Metadata

Working with Data • Providing data: o o Why were the data created? What limitations, if any, do the data have? What does the data mean? How should the data be cited if it is re-used in a new study? • Receiving data: o o o o What are the data gaps? What processes were used for creating the data? Are there any fees associated with the data? In what scale were the data created? What do the values in the tables mean? What software do I need in order to read the data? What projection are the data in? Can I give these data to someone else? What is Metadata

What is Metadata? • • • WHO created the data? WHAT is the content of the data? WHEN were the data created? WHERE is it geographically? HOW were the data developed? WHY were the data developed? What is Metadata Photo by Michelle Chang. All Rights Reserved Metadata is: Data ‘reporting’

Metadata in Real Life Author(s) Boullosa, Carmen. Title(s) They're cows, we're pigs / by Carmen Boullosa Place New York : Grove Press, 1997. Physical Descr viii, 180 p ; 22 cm. Subject(s) Pirates Caribbean Area Fiction. Format Fiction What is Metadata CC image by USDAgov on Flickr CC image by Mskadu on Flickr • Metadata is all around…

What is a Metadata Standard? • A Standard provides a structure to describe data with: o o Common terms to allow consistency between records Common definitions for easier interpretation Common language for ease of communication Common structure to quickly locate information • In search and retrieval, standards provide: o Documentation structure in a reliable and predictable format for CC image by ccarlstead on Flickr computer interpretation o A uniform summary description of the dataset What is Metadata

CC image by I like on Flickr What Does a Metadata Record Look Like? What is Metadata

The Value of Metadata Data developers Data users Metadata helps… Organizations What is Metadata

Concerns About Creating Metadata CC image by waterlilysage on Flickr Even if the value of data documentation is recognized, concerns remain as to the effort required to create metadata that effectively describe the data. What is Metadata

Concerns About Creating Metadata Concern Solution workload required to capture accurate robust metadata incorporate metadata creation into data development process – distribute the effort time and resources to create, manage, and maintain metadata include in grant budget and schedule readability / usability of metadata use a standardized metadata format discipline specific information and ontologies ‘profile’ standard to require specific information and use specific values What is Metadata

What is the Value to Data Developers? • Metadata allows data developers to: o Avoid data duplication o Share reliable information o Publicize efforts – promote the work of a scientist CC image by US Embassy Guyana on Flickr and his/her contributions to a field of study What is Metadata

What is the Value to Data Users? • Metadata gives a user the ability to: information from both inside and outside an organization o Find data: Determine what data exists for a geographic location and/or topic o Determine applicability: Decide if a data set meets a particular need o Discover how to acquire the dataset you identified; process and use the dataset What is Metadata CC image by ASEE on Flickr o Search, retrieve, and evaluate data set

What is the Value to Organizations? • Metadata helps ensure an organization’s investment in data: o Documentation of data processing steps, quality control, definitions, data uses, and restrictions o Ability to use data after initial intended purpose o Offers data permanence o Creates institutional memory • Advertises an organization’s research: o Creates possible new partnerships and collaborations through data sharing What is Metadata CC image by mambol on Flickr • Transcends people and time:

Time of data development Information Entropy DATA DETAILS Specific details about problems with individual items or specific dates are lost relatively rapidly General details about datasets are lost through time Retirement or career change makes access to “mental storage” difficult or unlikely Accident or technology change may make data unusable Loss of data developer leads to loss of remaining information TIME What is Metadata (From Michener et al 1997)

DATA DETAILS Information Entropy Sound information management, including metadata development, can arrest the loss of dataset detail. TIME What is Metadata

A Closer Look: The Utility of Metadata • Metadata can support: o data distribution o data management o project management • If it is: o considered a component of the data o created during data development o populated with rich content collect classify derive planimetric imagery meta analysis charette alternative committee review meta PLAN meta What is Metadata

Data Distribution via Metadata metadata publication data ery v disco data ls porta What is Metadata

Distribution: Data Discovery • The descriptive content of the metadata file can be used to identify, assess, and access available data resources. IDENTIFY • keywords • geographic location • time period • attributes What is Metadata ASSESS • use constraints • access constraints • data quality • availability/pricing ACCESS • online access • order process • contacts

Distribution: Metadata Publication • A metadata collection can be published to the internet via: o website catalog o web accessible folder (WAF) o Z 39. 50 metadata clearinghouse o metadata service o geospatial data portal Internet / Internet User Query What is Metadata Collection Intranet Dataset

Distribution: Geospatial Data Portals • Examples of metadata search portals: o Data. gov http: //www. geo. data. gov o Metacat • Repository for data and metadata • http: //knb. ecoinformatics. org/index. jsp o US Geological Survey • USGS Core Science Metadata Clearinghouse: http: //mercury. ornl. gov/clearinghouse o Arc. GIS Online • ESRI sponsored national geospatial data portal http: //www. geographynetwork. com What is Metadata CC image by RGB 12 on Flickr • Federal e-gov geospatial data portal

Example: USGS Core Science Metadata Clearinghouse What is Metadata

Data Management via Metadata nce a n e t Main pdate &U very o c s i D -use e R & What is Metadata Data Accountabil it y Data Liability

Management: Maintenance and Update • Metadata records can used to track data provenance accuracy • Data Maintenance: o Are the data current? • Do we have data older than ten years? • was before some political or geophysical event that resulted in significant change? o Are the data valid? • prior to most current source data • prior to most current methodologies • Data Update: o Contact information o Distribution policies, availability, pricing, URLs o New derivations of the dataset What is Metadata

Discovery: Data Reuse If you create metadata, you can find your own data What is Metadata CC image by Oceanit Daily Photo on Flickr If you create metadata, other people can discover your data

Management: Data Discovery & Reuse o o o CC image by NASA Goddard Spece Flight Center on Flickr • Find your data by: themes / attributes geographic location time ranges analytical methods used sources and contributors data quality Discoverable data is usable data! What is Metadata

Management: Data Accountability • Metadata allows you to repeat scientific process if: o methodologies are defined o variables are defined o analytical parameters are defined INPUT • Metadata allows you to defend your scientific process: o demonstrate process o increasingly GIS-savvy public requires metadata for consumer information RESULTS What is Metadata

Management: Data Accountability • Metadata is an exercise in data accountability. It requires you to assess: o What do you know about the dataset? o What don’t you know about the dataset? o What should you know about the dataset? Are you willing to associate yourself with the metadata record ? What is Metadata

Management: Data Liability • Metadata is a declaration of: Purpose What to o the originator’s intended application of do… the data What not to do… Use Constraints o inappropriate applications of the data Completeness o features or geographies excluded from the data Distribution Liability o explicit liability of the data producer and assumed liability of the consumer What is Metadata

Project Management via Metadata ct e j o r P ing n n a l P Project Coordination What is Metadata Project Monitoring Contract Deliverables

Project Management: Project Planning • Metadata records can serve as a project design document: o descriptions & intent of project o geographic and temporal extent of project o source data of project o attribute requirements of project • Benefits: o expectations are clearly outlined o metadata is integrated into the process o provides a medium to record progress What is Metadata

Project Management: Project Monitoring • Use metadata to monitor: Monitoring requires that the metadata be actively maintained and reviewed! milestones o data development status o QA/QC assessments o needed changes in approach time What is Metadata

Project Management: Project Coordination • Metadata can be a means to improve communications among project participants using common: o o o descriptions & parameters keywords, vocabularies, thesauri contact information attributes distribution information • If reviewed regularly by all participants, metadata created early and updated during the project improves opportunity for coordinating: o source data o analytical methods o new information What is Metadata

Project Management: Contract Deliverables • As a key component of the data, metadata should be part of any data deliverable • For quality metadata from a deliverable, the record should provide: o o o Citation information Data quality information Accurate geospatial information Clearly defined entities and attributes Distribution information What is Metadata

Image courtesy of Viv Hutchinson Choosing Metadata Standards What is Metadata

Multiple Metadata Standards Exist: Examples • Dublin Core Element Set o Emphasis on web resources, publications o http: //dublincore. org/documents/dces/ • FGDC Content Standard for Digital Geospatial Metadata (CSDGM) o Emphasis on geospatial data o Biological Data Profile (BDP) of the CSDGM o Profile to the CSDGM emphasis on biological data (and geospatial) o http: //www. fgdc. gov/metadata/geospatial-metadata-standards • ISO 19115/19139 Geographic information: Metadata o Emphasis on geospatial data and services o http: //www. fgdc. gov/metadata/geospatial-metadatastandards#fgdcendorsedisostandards What is Metadata

Multiple Metadata Standards Exist: Examples • Ecological Metadata Language (EML) o Focus on ecological data o http: //knb. ecoinformatics. org/eml_metadata_guide. html • Darwin Core o Emphasis on museum specimens o http: //rs. tdwg. org/dwc/index. htm • Geography Markup Language (GML) o Emphasis on geographic features (roads, highways, bridges) o http: //www. opengeospatial. org/standards/gml What is Metadata

Comparing Metadata Standards EML FGDC Title Abstract Entity Description Entity Type Definition Intellectual Rights Use Constraints What is Metadata

Choosing a Metadata Standard • Many standards collect similar information • Factors to consider: o Your data type: • Are you working mainly with GIS data? Rastor/vector or point data? Do you have biological or shoreline information in your dataset? - Consider the FGDC Content Standard for Digital Geospatial Metadata with one of its profiles: the Biological Data Profile or the Shoreline Data Profile. • Are you working with data retrieved from instruments such as monitoring stations or satellites? Are you using geospatial data services such as applications for web-mapping applications or data modeling? - If so, then consider using the ISO 19115 -2 standard • Are you mainly working with ecological data? - Consider Ecological Metadata Language (EML) What is Metadata

Choosing a Metadata Standard • More Factors to consider: o Your organization’s policies: do they state which standard to use? o What resources are available to create metadata? Examples of Tools: • • FGDC CSDGM: - Mermaid (NOAA) http: //www. ncddc. noaa. gov/metadata-standards/mermaid/ - Metavist (Forest Service) http: //ncrs. fed. us/pubs/viewpub. asp? key=2737 - TKME (USGS) http: //geology. usgs. gov/tools/metadata/tools/doc/tkme. html EML: - Morpho (http: //knb. ecoinformatics. org/morphoportal. jsp) • ISO: (http: //www. fgdc. gov/metadata/iso-metadata-editor-review) - XML Spy or Oxegyn - Cat. MD o Other factors: Availability of human support; instructional materials; use of controlled vocabularies; output formats What is Metadata

Summary • Metadata is documentation of data • A metadata record captures critical information about the content of a • • • dataset Metadata allows data to be discovered, accessed, and re-used A metadata standard provides structure and consistency to data documentation Standards and tools vary – select according to defined criteria such as data type, organizational guidance, and available resources Metadata is of critical importance to data developers, data users, and organizations Metadata can be effectively used for: o data distribution o data management o project management • Metadata completes a dataset. Creating robust metadata is in your OWN best interest! What is Metadata

The full slide deck may be downloaded from: http: //www. dataone. org/education-modules Suggested citation: Data. ONE Education Module: Metadata. Data. ONE. Retrieved Nov 12, 2012. From http: //www. dataone. org/sites/all/documents/L 07_Metadata. p ptx Copyright license information: No rights reserved; you may enhance and reuse for your own purposes. We do ask that you provide appropriate citation and attribution to Data. ONE. What is Metadata