Virtual Research Environments asaService Pasquale Pagano CNR pasquale

Virtual Research Environments as-a-Service Pasquale Pagano, CNR pasquale. pagano@cnr. it EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano EGI Community Forum 10 -13 November 2015 Bari, Italy www. d 4 science. org 1

Outline Context D 4 Science g. Cube • E-Infrastructure • History • … as a Service Capabilities • Virtual Research Environment • Features • Numbers EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 2

e-Infrastructure An operational combination of digital technologies (hardware and software), resources (data and services), communications (protocols, access rights and networks), and people and organizational structures needed to support research efforts and collaboration in the large EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 3

Genealogy DILIGENT 2004 -2007 Testbed: Virtual Research Environment D 4 Science 2008 -2010 Operational: several use cases (fisheries), g. Cube became an open source project D 4 Science-II 2010 -2012 Operational Ecosystem: use cases (marine biodiversity use cases), D 4 Science born to go beyond project lifetime i. Marine 2012 -2014 EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano Operational HDI: exploit D 4 Science, i. Marine Co. P, >1500 active users www. d 4 science. org 4

D 4 Science operates VREs for … +2000 scientists in 44 countries, integrating +50 heterogeneous data providers, executing +20, 000 processes/month; providing access to over a billion quality records in repositories worldwide, with 99, 7% service availability. D 4 Science hosts +40 VREs EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 5

Born to serve user needs I need to host my applications in a secure and scalable environment I need to maintain my database I need to backup my data I need to securely delivery my data to a set of known people I want to offer a flexible sharing, storage, reporting, search and retrieval tool I need to manage and analyze data I need to manage the full data life-cycle from import to validation, curation, harmonization and publication I need to offer to my team a powerful tool to manage code-lists I need to reduce the costs of data maintenance of my dept. I need to access authoritative data I need to simplify the access to my data I need to mash-up statistical and geospatial data I need to analyse my big datasets I need to validate my datasets and provide a standard access to them EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org

Distinguishing capabilities of the e-infrastructure D 4 SCIENCE EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 7

The D 4 Science infrastructure Hybrid Data Infrastructure combining over 500 software components into a coherent and centrally managed system of hardware, software, and data resources EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 8

D 4 Science enables e-infrastructure by. . . Integrating geographically distributed computing infrastructure Providing service allocations, deployment, monitoring, and operation Overcoming administrative boundaries Operation Built on SLAs Exploiting private and commercial providers Ensuring uniform resource and data access EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano Support monitoring, auditing, reporting, and notification Trust Privacy, governance, and attribution Security, trusted network www. d 4 science. org 9

Storage as Service to host and maintain data Database High-availability Standard Ready-to-use Cloud Storage Scalable Reliable Secure EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano Geographical DB Policies Standard Privacy and Attribution www. d 4 science. org 10

Applications as a Service to curate and manage data Metadata Generation Geospatial Data Biodiversity Data Statistical Data Textual Data Harmonization Disambiguate Validate Integrate and Consistency Check EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano Data Exchange OGC protocols Darwin. Core SDMX Dublin. Core www. d 4 science. org 11

Computing as Service to process and extract knowledge Scalable Elastic Easy to Manage Assignment of Computing Across Boundaries Assignment of Processors Tailored Virtual Research Environment EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano Heterogeneous High Throughput Map-Reduce Parallel R www. d 4 science. org 12

Computational Engine Not another cloud computer platform but • a platform where executions can be repeated, compared, discussed, logged Not another computational engine but • a platform where interdisciplinary tools and services can be easily contributed by the communities EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 13

Two exploitation models Dispatcher Tools (R, Java, …) must be uploaded to the storage Executable is deployed on the worker nodes assigned to the VRE Data are made accessible to the worker nodes according to the specification provided Monitoring, accounting, failures management, partial re-execution, sharing, and repeatability are granted Application Framework Predefined data splitting models are provided A large array of models and algorithms can be exploited to define custom workflows Large array of algorithms to compare results are provided EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 14

Virtual Research Environment to access, share and collaborate Share Database Tables Workflow Files Communicate Post Favourite Connection EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano Organize Dynamic Secure Policy Driven www. d 4 science. org 15

Virtual Research Environment • a distributed and dynamically created environment • where subset of resources (data, services, computational, and storage resources) • regulated by tailored policies (e. g. data encryption with VRE specific key, quota on service calls and storage usage, …) • are assigned to a subset of users via interfaces • for a limited timeframe • at little or no cost for the providers of the participatory data einfrastructures L. Candela, D. Castelli, P. Pagano (2013) Virtual Research Environments: An Overview and a Research Agenda. Data Science Journal, Vol. 12 EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 16

VRE Definition Metadata Applications Simple and effective process to define a new environment Data Configuration EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 17

Logical View Applications vs Services Applications Configuration Physical View Hardware Data Registry Software, Tools, Services EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano Data www. d 4 science. org 18

Application Bundles https: //www. gcube-system. org/catalogue-of-applications Apps. Cube To develop applications interfacing g. Cube facilities Geos. Cube To properly access, consume and produce geospatial information Biol. Cube To aid modelling and analysing of distribuition data, comparing checklists, and producing maps Stats. Cube To assist tabular data validation, data enrichment ad efficient analytical tools EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano Connect. Cube To facilitate data publication with appropriate tools including semantic technologies Ice. Cube To support deployment, operation & mgmt of a g. Cube-based infrastructure www. d 4 science. org 19

VRE Exploitation Exploited for Public VREs (used to offer an application environment to a subset of users of a community) and Private VREs (used for experiments, data access and preparation, and data analytics) Fully operational VRE available in one hour Software deployment and hardware setup completely hidden Evolving needs of its users completely supported EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 20

Entity as Resource Entity • Server, Storage • Container • Software • Data As a resource • • Publication/Discovery Lifecycle management Failure management Authorization-accounting As a service • Access • Orchestrate • Reference Software as Resource: transforms servlets-based applications/services in e-Infrastructure resource Container as Resource: transforms standard servlets-based container in e-Infrastructure resource Federated Sources as Resource: transforms external DBs and Repositories in e-Infrastructure resource Algorithm as Resource: for any new algorithm, model, procedure, workflow, … it is possible to manage policies and assign dedicated Hardware and Storage resources Dataset and single product as Resource: for any dataset, map, timeseries, code list, …. It is possible to manage policies and monitor their exploitation EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org

Smart. Gears “a set of Java libraries that turn Servlet-compliant containers and applications into infrastructure resources, transparently. ” g. Cube Wiki turn software and containers into resources what does it mean ? EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 22

Smart. Gears [cont. ] Software-as-Resource Container-as-Resource software and nodes we can discover § use without hardcoded knowledge monitor and control § take actions when not operational dedicate to user groups Actual Solution Zero constraints human solutions § not practical, often impossible automated solutions § local enabling software, remotely controlled management tasks § compile and publish descriptions § track and change status § enforce policies § change policies, assign roles EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 23

g. Cube: One stable open-source platform g. Cube enables the D 4 Science HDI Statistics form openhub. net/p/g. Cube EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 24

Multi-tenant Delivery Model Infrastructure as a Service • Dynamic deployment • Hosting • Resource Lifecycle • Monitoring • Accounting • Security Software as a Service Platform as a Service • VRE • Biol. Cube • Connect. Cube • Geos. Cube • Stats. Cube • Feather. Weight. Stack • Smart. Gears • Application. Support. Layer • SOA 3 EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 25

References / Links D 4 Science: http: //www. d 4 science. org Policies • https: //wiki. d 4 science. org/D 4 Science_Deployment_and_Operation: _Policies Procedures • https: //wiki. d 4 science. org/D 4 Science_Deployment_and_Operation g. Cube: http: //www. gcube-system. org Catalogue of Applications § https: //www. gcube-system. org/catalogue-of-applications Software Key Features § https: //wiki. gcube-system. org/GCube_Features Developer Guide § https: //wiki. gcube-system. org/Developer%27 s_Guide Feather. Weight. Stack • https: //wiki. gcube-system. org/Featherweight_Stack § Smart. Gears • https: //wiki. gcube-system. org/Smart. Gears § g. Cube APIs • https: //wiki. gcube-system. org/GCube_Application_Programming_Interface § Administration Guide § https: //wiki. gcube-system. org/Administrator%27 s_Guide EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 26

Thank you for your attention QUESTIONS? EGI CF 2015 - Virtual Research Environments as-a-Service – P. Pagano www. d 4 science. org 27