RDAs Recently Endorsed Outputs September 16 2015 Agenda

RDA’s Recently Endorsed Outputs September 16, 2015

Agenda § § § Introduction Data Foundation and Terminology – PID Information Types Practical Policy Data Type Registries Questions 2

Data Foundation and Terminology - talking the same language – Peter Wittenburg, Gary Berg-Cross, Raphael Ritz

Summary of the Problem 4 § What is the problem? § Data organizations (DOrg) and ideas about it are all different § We are all speaking different languages, wasting time and misunderstanding each other in any project involving data § Different DOrgs make data discovery and integration very time consuming, inefficient and thus expensive § Different DOrgs prevent us developing maintainable support software § Who is impacted (specific domains, professions, etc. )? § All efforts to integrate data, for example in federations, BDA projects, etc. § What are the ramifications of not having the problem resolved? § Combining data of all sorts across different origins (projects, repositories, disciplines, etc. ) is a nightmare and requires a lot of curation and transformation before the actual scientific analysis can start

Highlights of the Effort and Deliverables § Working Group structure (how many members, diversity of experience, geographies, etc. ) § DFT WG had 60 members coming from almost all regions § Members came from different types of institutions and disciplines § DFT WG included relative newcomers up to members with much experience from data intensive projects § DFT WG produced § a list of core terms essential to harmonize conceptualization of data organizations § a graphical model relating the terms § a set of auxiliary documents including many use cases to demonstrate the bottom-up approach and research of the WG § a Term Tool (using Semantic Media Wiki) to store definitions and allow editing, classification and discussion of terms (which is also open for other groups) 5

Active Contributors to the Work Institute/Project Country/ Region Domain CNRI US IT Research and Systems U Cardiff UK IT Research and Systems AWI DE Oceanography & Environment MPG DE Research Organisation EUDAT EU Data Infrastructure CLARIN EU Linguistic Research Infrastructure EPOS EU Earth Observation Res. Infrastructure ENES Int World Climate Res. Infrastructure ENVRI EU Environmental Res. Infrastructure Data. One US Environmental Infrastructure ESSD/RENCI US Earth Science System Data NCGEN/RENCI US Clinical Genomics Europeana EU Humanities Infrastructure Data. Cite/EPIC Int PID Infrastructures DICE US IT Research and Systems CAS CN Earth Science Model ADCIRC/RENCI US Ocean and Storm modeling 6

Impact of the Deliverable 7 § Who was impacted by deliverable? § The European data infrastructure EUDAT is federating data from many discipline repositories where each data collection has a different data organization. If integration is not simply done at physical level (file structures), this heterogeneity makes it very costly to integrate all data to enable re-purposing and to make it accessible at different repositories. § The Technology Director of the international CLARIN project said: § Very handy to have a lingua franca when discussing research infrastructure architectures § It was good to be involved as adopting community from the start of the work § Similar experiences are made by US, Chinese etc. colleagues that work on large scale data integration. Integration work is special and thus does not scale. Even the integration of a simple database of animal voices of the world (11 TB) requested the development of special scripts to extract metadata, relations, rights etc. in addition to the data files § Harmonization would reduce integration time by large factors and had already great effects on interaction efficiency and integration.

Endorsements/Adopters and how have they used the deliverable 8 § Our adopters § The early adopters are to a certain extent those who have these dramatic problems in data integration such as EUDAT, CLARIN, etc. § Their approach was aligned with the progress of the WG discussion. All their repository setups adhere now to the DFT model and their interaction with different communities are based on it: central is the Digital Object, that is described by metadata, is associated with a Persistent ID and whose instances are stored in trustful repositories (see simplified diagram) y d. B persistent ID rence digital object is. Described. By metadata te is. Refe en res p e is. R bitstream In red o t is. S repository isa § Also several other projects, for example from humanities, health, bioinformatics, neuroinformatics and atmosphere research adopted the basic & simple model and the terminology.

Endorsement/Adoption 9 Institute/Project Country/ Region Domain CNRI US IT Research and Systems U Cardiff UK IT Research and Systems MPG DE Research Organisation EUDAT EU Data Infrastructure CLARIN EU Linguistic Research Infrastructure EPOS EU Earth Observation Res. Infrastructure ENES Int World Climate Res. Infrastructure ENVRI EU Environmental Res. Infrastructure ESSD/RENCI US Earth Science System Data NCGEN/RENCI US Clinical Genomics DICE US IT Research and Systems ADCIRC/RENCI US Ocean and Storm modeling Deep Carbon Project US Environmental/Athmospheric Research Note: There may be more projects/institutes that have endoresed or adopted the DFT model without noticing us.

How You Can Endorse 10 § Who could use the DFT Terminologies? § The vocabulary is openly available for everyone who wants to run a project including those with large data collections § The organization should be strictly compliant to the model to guarantee independence and thus easy re-purposing of all components § The vocabulary is openly available for everyone who is working in a data federation project integrating data from different sources or who wants to re -purpose data for data intensive science § Projects could use the DFT WG model as a common reference model to design transformations § Projects could use the suggested terminology to achieve quick, mutual understanding § Software developers can adopt this basic model to make sure that their software can be used by almost everyone adhering to state of the art principles

How You Can Endorse 11 § How to access and use them § Take the “Core Terms and Model” document which provides the final model and the corresponding terms and apply it in your project § In case of questions § Read the supplementary documents to understand conceptualization and background for choices § Meet the WG co-chairs and experts at a plenary § Contact the WG co-chairs § Contribute to the now functioning DFT IG (email, wiki, Term Tool) § Send a request to the RDA Europe support team (email, wiki) (references see last slide)

Next Steps 12 § Are there plans to further evolve this deliverable? § Yes, since the WG just focused on the basic set of core terms, and additional RDAS WGs are completing work so there is much more out there where terminology harmonization would help substantially § We also see the need to consider the dynamics of the field and to be ready to adapt current definitions and perhaps even the model § Is there an IG or WG that individuals can join on a related topic? § Yes, a follow-up DFT Interest Group has been established and will meet at Plenary 6 § A larger scope of integrated work is being discussed as part of the Data Fabric IG

Contact Information § Who can individuals contact to learn more about this deliverable? § DFT WG: https: //rd-alliance. org/groups/data-foundation-and-terminology-wg. html § DFT IG: https: //rd-alliance. org/groups/data-foundations-and-terminology-ig. html § Te. D-T Term Definition Tool: http: //smw-rda. esc. rzg. mpg. de/index. php/Main_Page § RDA EU Support Team: dmp@europe. rd-alliance. org 13

PID Information Types: Towards PID interoperability Tobias Weigel (DKRZ / University of Hamburg) Tim Di. Lauro (Data Conservancy / Johns Hopkins University)

Summary of the Problem 15 § Move from management of files towards management of objects § How does object management scale with increasing numbers? § How do we further automate our processes? § Issues independent from particular disciplines, repositories, management approaches § Understanding the most elemental characteristics of digital objects – for machine agents and human users § Facilitate interoperability across PID systems and simplify PID record usage § Not addressing these key challenges is likely to lead to insular solutions and reiteration of efforts

Highlights of the Deliverables 16 § More than 50 group members from EU/US/AU § A lot of technical expertise and community experience § Key deliverables (cf. summary report): § § Conceptual insights on types and their possible structures Practical type examples geared towards diverse use cases Openly licensed API specification and Java-based prototype Approach for using a general type registry IDENTIFIER Verification service properties size checksum timestamps aggregation version license format Size: Format: Checksum: Date: Size: Checksum: Format: License:

Impact of the Deliverable § Some initial types have been registered, making it possible to explore further applications § Information on how to register new types available in the report § Registration relies on the Type Registry § Incited plans in communities and projects about concrete applications § PIDs and typing increasingly seen as a crucial component to decouple management of objects from contents § Simplify client access to data across domains, implementations and changes in information models § More lightweight access to information on less accessible objects 17

Endorsements/Adopters 18 § Adopters can be: § Communities who can use existing types and share custom types, as well as build tools and services that exploit them § PID service providers who can offer a typing service as added value beyond registration and resolution, increasing PID interoperability Adopter Category Country Scope / Goal ENES Community Int. IPCC AR 6 data management DCO-DS/RPI Community US Enhancing existing PID usage EUDAT Community/Service provider EU Added-value service to various disciplinary communities MGI/NIST Community US Automation of data type conversions EPIC Service provider EU CNRI Service provider US DONA Service provider Int. Generic added-value service

How You Can Endorse 19 § Make use of existing types, invent your own and please tell us about it! § Follow-up RDA WGs on Collections and Data Typing will continue the work on concrete types. The PID Interest Group is also a good place to provide general feedback. § Specification and prototype source code are openly available § Possible development by EUDAT, DCO, ENES and others as interested adopters § Offer by PID service providers as a service beyond registration and resolution § Contribution to a unified type registry is encouraged

Next Steps and Contact Information § PID Information Types WG § https: //rd-alliance. org/groups/pid-information-types-wg. html § PID Interest Group § https: //rd-alliance. org/groups/pid-interest-group. html § PID Collections candidate WG § https: //rd-alliance. org/groups/pid-collections-wg. html § https: //rd-alliance. org/pid-collections-p 6 -bof-session. html § Data Typing Bo. F § https: //rd-alliance. org/data-typing-p 6 -bof-session. html § personal contact: weigel@dkrz. de 20

Working Group Practical Policy based on slides and latest documents from the PP WG chaired by Reagan Moore, Rainer Stotzka

Summary of the Problem Computer actionable policies are used to § enforce data management § automate administrative tasks § validate compliance with assessment criteria § automate scientific data processing and analyses Practical Policy Assertion or assurance that is enforced about a (data) collection (data set, digital object, file) by the creators of the collection Users motivated by issues related to scale, distribution 22

Policy Templates 23 § Practical Policy members represented § 11 types of data management systems § 30 institutions § 2 testbeds § i. RODS Renaissance Computing Institute, Data. Net Federation Consortium – DFC § GPFS Institute of Physics of the Academy of Sciences, CESNET Garching Computing Centre – RZG § Published two documents § Moore, R. Stotzka, C. Cacciari, P. Benedikt, “Practical Policy Templates” February, 2015, http: //dx. doi. org/10. 15497/83 E 1 B 3 F 9 -7 E 17 -484 A-A 466 B 3 E 5775121 CC. § Moore, R. Stotzka, C. Cacciari, P. Benedikt, “Practical Policy Implementations”, February, 2015, http: //dx. doi. org/10. 15497/83 E 1 B 3 F 9 -7 E 17 -484 A-A 466 B 3 E 5775121 CC.

Production Environments 24 § Computer actionable rules to enforce: § Preservation standards § Authenticity, integrity, chain of custody, arrangement § Data management plans § Collection creation, product generation, publication, storage, archives § Data distribution § Replication, content distribution network § Publication § Descriptive metadata, time dependent access controls § Processing pipelines § Workflow execution

Endorsements/Adopters § Distributed data management environments § EUDAT Data Policy Manager § B 2 SAFE use case § § International Neuroinformatics Coordinating Facility Institut national de physique nucléaire et de physique des particules New Zealand BESTGRID Data. Net Federation Consortium § § § NSF data management plans Odum Institute preservation archive The i. Plant Collaborative genomics data grid Science Observatory Network digital library SILS Life. Time Library Hydro. Share § NOAA National Climatic Data Center § NASA Center for Climate Simulations 25

Applications 26 § Policy-based collection management § § § Purpose for assembling the collection Properties required to support the purpose Policies that control when and where the properties are enforced Procedures that execute operations controlled by the policies Persistent state information that is generated by the procedures Periodic assessment criteria that verify compliance § RDA Publications § Policy templates § Constraints, operations, required state information § Policy implementations § Computer actionable rules to automate policy enforcement

Next Steps and Contact Information 27 § Data Fabric Interest Group § Policies to support § Federation § Interoperability § Data Foundations and Terminology Interest Group § Vocabulary for policy management § Interoperability testbeds § EUDAT § http: //eudat. eu/data-access-and-reuse-policies-darup § National Data Service § http: //www. nationaldataservice. org § Data. Net Federation Consortium § http: //datafed. org

Data Type Registries Larry Lannom, CNRI Daan Broeder, Meertens Institute, KNAW

Summary of the Problem § § § 29 Data sharing requires that data can be parsed, understood, and reused by people and applications other than those that created the data How do we do this now? § For documents – formats are enough, e. g. , PDF, and then the document explains itself to humans § This doesn’t work well with data – numbers are not self-explanatory § What does the number 7 mean in cell B 27? Data producers may not have explicitly specified certain details in the data: measurement units, coordinate systems, variable names, etc. Need a way to precisely characterize those assumptions such that they can be identified by humans and machines that were not closely involved in its creation Affects all data producers and consumers

Goal of the DTR Effort: Explicate and Share Assumptions using Types and Type Registries 30 § Evaluate and identify a few assumptions in data that can be codified and shared in order to… § Produce a functioning Registry system that can easily be evaluated by organizations before adoption § Highly configurable for changing scope of captured and shared assumptions depending on the domain or organization § Supports several Type record dissemination variations § Design for allowing federation between multiple Registry instances § The emphasis is not on § Identifying every possible assumption and data characteristic applicable for all domains § Technology

Highlights of the Deliverable § § § 31 Confirmation that detailed and precise data typing is a key consideration in data sharing and reuse and that a federated registry system for such types is highly desirable and needs to accommodate each community’s own requirements Deployment of a prototype registry implementing one potential data model, against which various use cases can be tested Involvement of multiple ongoing scientific data management efforts, across a variety of domains, in actively planning for and testing the use of data types and associated registries in their data management efforts Integration with one additional RDA WG (Persistent Identifier Types) and at least one Interest Group (RDA/CODATA Materials Data, Infrastructure & Interoperability IG) Development of a set of questions that require further consideration before a detailed recommendation on data typing can be issued

Impact of Use Case: Process Use Case 32 3 Users 2 1 Federated Set of Type Registries 4 ID ID Type Payload Type Payload 4 Payload Typed Data Terms: … I Agree 10100 Visualization 11010 Rights 101…. Data Set Data Processing Dissemination Services 1 Client (process or people) encounters unknown data type. 2 Resolved to Type Registry. 3 Response includes type definitions, relationships, properties, and possibly service pointers. Response can be used locally for processing, or, optionally 4 typed data or reference to typed data can be sent to service provider.

Endorsements/Adopters § Materials Science Adoption Project § Demo at P 6 § X-ray diffraction use case § normalize data sets resulting from multiple proprietary instruments § Enable a homogenous analysis platform for data consumers to perform their analyses § Deep Carbon Observatory § Goal: given a dataset identifier, discover detailed information about the structure(s) within that dataset, and act accordingly § DTR is a registry used for explicating structures in the form of type records § Facilitate norms of behavior relevant to data curation and re-use § Digital Object Identifier § Given a DOI, what services are relevant and applicable § Having chosen a service, how can a client invoke that service? § Having invoked a service, how can a client process the returned data? 33

How You Can Endorse § Start a new prototype effort § Follow existing prototype efforts § Attend the BOF at P 6 § Join the Data Typing WG when it starts § Try the public prototype at typeregistry. org 34

Next Steps and Contact Information § A follow-up WG is planned: Data Typing § Leverage results of DTR § Collect results from multiple prototypes § Best practices for federation § BOF on Data Typing at P 6: 24 Sept. , Breakout #6 § Proposed Chairs of Data Typing WG § Giridhar Manepalli, CNRI § Simon Cox, CSIRO § Tobias Weigel, DKRZ § Larry and Daan are still around 35