Position Paper for Data Fabric IG Interoperability Infrastructures

Position Paper for Data Fabric IG Interoperability, Infrastructures and Virtuality https: //rd-alliance. org/group/data-fabric-ig. html Gary Berg-Cross, Keith Jeffery, Reagan Moore What Managed processes & Services? What Basic & Flexible Infrastructure Machinery? See https: //rd-alliance. org/group/data-fabric-ig/post/re-rdadatafabric-ig-data-fabric-position-paper-broadendiscussion-berg-1 for discussion & attached file. What principles & methods principles are needed to guide the interaction between services interface, protocol ?

Key Points 2 1. Much of the current DF discussion focuses on a data management & lifecycle view. § Lacks a focus on other important topics § standards & federation mechanisms that are needed to assemble collaborations spanning institutions, data management environments. 2. Interoperability, needs to be a 1 st class concept in the DF conservation § it is fundamentally important for federation & overcoming data-silo generated problems. 3. There are multiple benefits for development of federated & virtualized mechanisms & mathematical descriptions to assist sharing DOs & (digital) knowledge procedures.

Initial ideas for DF IG- Implied Framework is Data Lifecycle View with a Data Management Focus that emerged from the discussions amongst various RDA WG chairs Where is interoperability in a Raw to Citable Data View ? New Groups emerge so Pubs are part of Data Fabric & related analysis 3

Data Fabric Analysis, e. g. components & services in LC via Use CASES How do we come to essential components & services? (guided by use scenarios that need Sharing to include collaboration) This scenario doesn’t show analysis or data sharing via DO manipulation! 4

Make Interoperability a First Class View § Concept of Interoperability: § The extent to which systems and devices can routinely exchange data and services, and interpret that shared data through the shared services § A stronger type of data exchange can include knowledge of the meaning of the data content, usage constraints, and the underlying assumptions. § Bring interoperability (within and cross-domain) aspects into the DF discussions as a ‘first class citizen’ alongside all the other aspects of the research data lifecycle in a domain. 5

Multiple types of DF infrastructure 6 § When an enterprise implements a data management solution, one of multiple types of DFs infrastructure is typically chosen: § Data management –enterprise to build a data repository, manage an § § information catalog, & enforce management & curation policies (but also) Data analysis –enterprise to process a data collection, apply analysis & visualization tools, and automate a processing pipeline. (but also) Data preservation –enterprise to build reference collections and knowledge bases that comprise their intellectual capital, while managing technology evolution Data publication –enterprise to provide descriptive information and arrangement for discovery and access of data collections. Data sharing – controlled sharing of a data collection, shared analysis workflows, and information catalogs - interoperability.

Interoperability mechanisms required for sharing data, information, & knowledge. Composition how the separate components, developed separately, can be made to work together. Minimal set of infrastructure mechanisms & service requirements 7 Different suites of components will have different data fabrics. Brokers Gaps, obstacles and possible incompatibilities Enable reproducible research

Data Sharing Use Cases § EUDAT & the Data. Net Federation Consortium use cases provide some view to help: § Interoperability mechanisms for sharing DOs & (digital) knowledge procedures § An implication is that researcher can re-execute trusted procedures to obtain identical results, making reproducible datadriven research possible § Community driven research collaborations § Seismology – share seismic data, tsunami prediction workflows between research groups § Climate change – share oceanography environmental data, coastal storm surge analyses, hydrology flood analyses, satellite environmental data § Genomics – build a cohort of genomes, predictive models for humans, plants, animals, diseases 8

Expanded Ideas of Federation: 3 versions of federated systems 9 1. Shared name spaces for users, files, and services. 1. Besides a single sign-on, shared name space providing users federated services we want to afford service for virtual collections that span administrative domains. 2. And a shared name space for services enables re-use of procedures across researcher resources. 2. Shared services for manipulating digital objects. 1. Such as shared service through a broker, accessing the service through its access protocol or an encapsulated service in a virtual machine environment, for movement to the local research resources for execution. 3. Third-party (service) access. 1. Posting requests to a 3 rd party, such as a message queue, and eliminate direct communication between the federated system components.

Enhanced Use of Virtualization 10 § Virtual machines, such as in a CLOUD or GRID environment, § Required to manage dynamic resource allocation, scalability, distributed parallelism, energy efficiency and other aspects. § Virtual collections § Required to build research collaboration environments § We need the appropriate level of abstraction for optimum computing environment/middleware behavior. § Too low or prescriptive a level constrains the environment, § too high or abstract a level does not indicate clearly the requirement of the user. § See Triple-I Computing as a concept (Information-Intention-Incentive model proposed by [Schubert and Jeffery, 2014]) and already research projects are addressing the challenges therein.

Analysis and Preliminary Conclusions 11 § We have made a useful start but the DF vision needs to be expanded (also focused for maximum benefit) to § more than a domain of registered DO stored in well-managed repositories § Frame DF & its services broadly as data use & applications taking into account the available context or environment. § This doesn’t minimize good data management practices and services which are necessary and deserve support, but are not sufficient to address the challenge for interoperability. § For enhanced, semi-automated interoperability we need to consider: § Improved metadata for data in context with enhanced semantics § Leveraging the emergence of a mathematical foundation for federation of data management systems (e. g. work by Hao Xu).

So…. DF, VRE must be integrated § Including datasets, SW services, resources (computers, detectors…), users § Composed as workflows documented mathematically and ideally created autonomically § Achieved through metadata describing the elements of bullet 1 § Discovery § Contextualisation (relevance, quality, … through relations to organisations, persons, projects, publications etc. and provenance, rights) § Detailed application-specific (to connect software to data at a resource for a user) i. e. schema-level § The key technologies to achieve interoperability as recognised by researchers are: § AAAI § PID § Metadata with formal syntax and declared semantics 12