Rosetta HUB for AWS mass adoption in Higher

Rosetta. HUB for AWS mass adoption in Higher Education and research. Pervasive cloud, data science, machine learning, big data and HPC education.

Rosetta. HUB for Amazon Web Services 80 higher education institutions including 4 among the top 10 universities in the World 20, 000 students, educators and researchers 16 Countries including the UK, Ireland, France and Germany 100% Automation of onboarding, resources and consumption monitoring and users management Fully automated digital university model

Rosetta. HUB Overview

Governance, Federation and Management for AWS Account Management Automated users enrollment and processing AWS Accounts full life cycle management Full integration with Liferay's accounts, organizations and roles Full mapping of organizational hierarchy and responsibilities. Seamless accounts limits management and traceability. Integrated ticketing system Budget Control and optimization Costs and resources real-time monitoring and control Management of user budgets' and AWS permissions Safeguards and cost optimization Seamless Spot market management AWS grants full life cyle management Compliance enforcement and cloud access sandboxing Automated AWS accounts limits management Detailed auditing and reporting Rosetta. HUB management Web Services E-learning and E-research platform Universal Workbench and notebooks at scale Rosetta. HUB collaborative Workbench Jupyter servers Unlimited RStudio servers, Shiny Apps servers Zeppelin servers, Daa. S Ubuiquitous sharing and real-time collaboration Easy sharing of all Rosetta. HUB artifacts with users, groups or organizations Access to the Rosetta. HUB publishing and sharing platform for e-Learning and e-research Big Data, HPC and Deep Learning made easy Rosetta. HUB-managed Elastic Map Reduce clusters Rosetta. HUB nvidia-docker-based virtual environments for deep learning Rosetta. HUB-managed Cfn. Cluster and Alces Flight HPC clusters Rosetta. HUB spreadsheets for technical computing End-to-end reproducible e-learning and eresearch Management platform for Docker containers Edu. Ops and Res. Ops platform Convergent Ia. C, containers and technical computing APIs Software Development Kits: Python, Java, R, C# and js sdks Office integration: Word and Excel Add-ins Rosetta. HUB meta-cloud and technical computing APIs Programmable hybrid-kernel (R/Python/Java/Scala) Reactive programming framework. Full auditability Detailed auditing and reporting End-to-end traceability Integration capabilities Dedicated Rosetta. HUB portal Dedicated publishing and sharing platform SAML/Oauth 2. 0/LDAP/Active Directory integration Ssl certificates management platform Programmable emailing platform Advanced scheduling

Rosetta. HUB, state of the art governance and management platform for AWS

The building blocks of AWS democratization Rosetta. HUB provides every student and every educator with an account on a social collaboration portal. Each portal account is linked to a private AWS account created, managed and monitored by Rosetta. HUB. The portal makes advanced AWS capabilities easy to understand operate by students and educators. It also makes all cloud artifacts easy to share. Rosetta. HUB fully automates the onboarding processes and gives institutions flexibility on budget allocation.

End-to-end monitoring, management and audit The institution’ Central Point Of Contact (CPOC) and educators can monitor on realtime (*) the students’ interaction with AWS and the portal. The CPOC can manage students: adjust their budgets, their rights on AWS, their resources allowances, etc. The CPOC can create sub-organizations and assign roles to colleagues for a multi-tenant management of students. System administrators can generate reports on users activities and cloud usage. They can measure and assess effectiveness of the use of cloud resources. Repositories of pedagogic cloud artifacts can be prepared and shared with students. (*). .

Students and educators dashboards The Rosetta. HUB students and educators dashboards display an access button to the AWS console as well as access keys for programmatic access to AWS. It provides detailed aggregated real-time information about the resources being used on AWS, the budget amount left and the estimated overall hourly cost. Students and educators can request: 1. Limit increase to access higher capacity machine instance types (eg. p 2. *, p 3. *, g 3. * GPU instances). 2. Access to optional AWS services 3. Budget increase and budget transfer to other users 4. Support 44 AWS Services are accessible by default. Access is available to IAM in a proxied manner to preserve the accounts sandboxing. IAM users and IAM roles can be easily and safely created and managed from the dashboard. (*) Limits and budget requests are automatically processed by the Rosetta. HUB pipelines within a predefined scope. Rosetta. HUB creates and tracks tickets with AWS support.

Cost optimization and safeguards Accounts get automatically disabled and all on-demand EC 2 instances are stopped if the user goes above 100% of his/her budget or if the estimated hourly price exceeds the maximum hourly price. Spot instances are snapshotted then terminated. No data is deleted when a user is disabled. Auto-stop on idle EC 2 instances: the user can set the maximum idle time or disable this feature. By default it is set to 6 hours. Notification emails at 50%, 70%, 90% and 100% of budget consumption. Use of Spot instances is promoted in the Rosetta. HUB launch panels, spot instances are the first choice when launching instances or clusters. Users monitoring panel in the CPOC’s management console

Full technical and compliance integration Institutions, educators and students take no financial risks as all AWS accounts are guaranteed by Rosetta. HUB acts as a procurement adapter: It allows Higher Education institutions and research laboratories to top-up their Rosetta. HUB institutional account with cloud credits in compliance with their regulatory frameworks and administrative constraints. A dedicated Rosetta. HUB infrastructure can be fully integrated with the institution’s Information system.

Rosetta. HUB, Next generation e-research and e-learning platform

Democratic and pervasive data science The Rosetta. HUB platform closes the technology gap between clouds, containers, data science software, realtime collaboration frameworks, social portals and people. The Rosetta. HUB data science platform makes it easy for educators to compose containers-based virtual e-learning environments and for researchers to compose virtual e-science environments. Jupyter, RStudio, Spark, Zeppelin, Shiny Apps, virtual desktops, HPC clusters, etc. can be added to the virtual environments and made accessible in a secure and highly scalable-manner to thousands of students or collaborating researchers.

Defining the meta-cloud: Rosetta. HUB Web Services & managed images Rosetta. HUB delivers : • A docker-based meta-cloud. • A universal data science workbench. • A meta-kernel for data science • A man-cloud and man-data interaction design • A sharing model for cloud artifacts • A SOAP/Restful API with ~1000 functions • SDKs and add-ins • A cloud and data products marketplace. Rosetta. HUB fosters • Usability • Reproducibility • Shareability • Auditability at all layers of interaction between students, educators and researchers and their software tools, infrastructures and peers. Public Cloud Private Cloud Data scientist

One-click access to AWS-powered data science The Rosetta. HUB dashboard displays the cloud and data science related artifacts as customizable icons structured in categories. Rosetta. HUB meta-formations: they enable one-click provisioning and access to fullymanaged complex infrastructures for elearning and e-Research. Rosetta. HUB meta-keys: they map AWS access keys and a default VPC, they allow rapid access to AWS services and they can be shared. Rosetta. HUB meta-images: • Managed: they come with agents to orchestrate all service components and expose a composable virtual workbench to the end user • Semi-managed: they map any EC 2 AMI Rosetta. HUB meta-storages: they map S 3 buckets, EFS or EBS volumes. They can be used as the working or reference volumes for managed instances and clusters.

User-friendly Spark and Hadoop clusters for research and education Launching an EMR cluster can be done in one click by choosing an available formation or by creating a custom formation with custom settings Seamless creation of Hadoop and Spark clusters based on AWS EMR, the Rosetta. HUB smart proxies and the Rosetta. HUB workbench. Support for both on-demand spot. Seamless access to clusters with shells and notebooks including Rosetta. HUB notebooks, Zeppelin, Jupyter, Spark-Notebook, etc. Real-time collaborative access, cluster sharing, security and access control for Hadoop and Spark. Seamless data management, seamless mounting of S 3 and EFS volumes on master and slave nodes. Very rapid big data applications prototyping using the Rosetta. HUB reactive programming frameworks, web applications designers and spreadsheet engines. Access the cluster’s master in the browser from the Rosetta. HUB collaborative workbench

User-friendly managed HPC for research and education Seamless creation of NVIDIA-docker based virtual environments for deep learning on GPU. Seamless creation and access to HPC clusters based on Alces Flight or cfn. Cluster, the Rosetta. HUB smart proxies and the Rosetta. HUB workbench. Real-time eagle-view on resources, billing and hourly cost for HPC clusters. Seamless data management, seamless mounting of S 3 and EFS volumes on master and slave nodes. Extended support for spot and autoscaling. Out-of-the-box cluster security and access control. Notebooks, cluster sharing and real-time collaboration for Alces Flight and cfn. Cluster. Seamless scheduling using cron and rate tasks. Interactive Scientific Web UIs and reactive programming frameworks for HPC clusters. Launching a HPC cluster can be done in one click by choosing an available formation or by creating a formation with custom settings

Rosetta. HUB Res. Ops/Edu. Ops Virtual-labs-as-code

Rosetta. HUB meta-Formations eg. Deep learning assignments Spot Machine EMR Cluster Cloud Keys: AWS Keys SSL certificate Proxy Instance Type Machine Image: Tensorflow GPU Image Machine Pool Instance type: p 2. xlarge EMR Cluster Maximum Bid Price Master Instance type: m 4. large HPC Cluster Reference and Working Volumes Slave Instance type: m 4. large Spot Machine Pool Spot EMR Cluster Rosetta. HUB meta Formation eg. Big data workshop Spot HPC Cluster Proxy Image: Standard CPU Image Reference and Working Volumes

Students and educators persistent workspaces Rosetta. HUB creates for each student and educator a default S 3 storage and a default EFS storage which map an S 3 bucket and an EFS volume Formations are configured with working volumes and reference volumes which can be mappings of EFS, EBS, S 3 or FTP. These are automatically mounted on the EC 2 instances including nodes of HPC and EMR clusters Any public formation that the user launches automatically uses the default user’s EFS as its working volume: Data generated by students and educators is persistent and survives the termination of machine instances The reference volume can by synched at start-up to the working volume EFS, EBS and S 3 Volumes can be automatically mounted on the docker container of the Rosetta. HUB managed instances

Universal collaborative workbench The Rosetta. HUB meta-formations and Images can be used to create Rosetta. HUB Sessions provide access to the universal workbench and they can be shared with a user or a group of users. Users have the same view on the workbench and can collaboratively create and adjust widgets, interact with tools and data. Composable widgets include: • Real-time collaborative consoles, notebooks and code editors on the most commonly used tools for data analysis: R, Pyhton, Scala, RStudio etc. • Applications access (Jupyter, Zeppelin, etc. ) • Real-time collaborative RStudio • Real-time collaborative remote desktop access in the browser. • Data visualization and interaction components such as charts, sliders, buttons.

Meta compute kernels & seamless data management The universal workbench allows the remote interactive control of Rosetta. HUB metakernels created and managed by the Rosetta. HUB docker agents. The Rosetta. HUB meta-kernels are processes merging the virtual machines of Java, R and Python. Meta-kernels allow intercommunication and in-memory transfer of variables from one language to the other Meta-kernels data access is fully managed by Rosetta. HUB. Meta-kernels can be shared as well as their working volumes and reference volumes.

Semi-managed images allow users to easily launch a machine from the Rosetta. HUB web console using their Rosetta. HUB keys Launching semi-managed images can be done in one click from the Rosetta. HUB dashboard Access to the instances is managed by Rosetta. HUB, ie. Rosetta. HUB generates and saves the private keys associated with the instance as well as the password for Windows instances. Users can retrieve their private keys and passwords anytime. Instructions on how to connect to Linux and Windows instances are provided to the user

Rosetta. HUB mass onboarding process

The Rosetta. HUB automated mass onboarding process for AWS: Oxford University Students/Educators register individually at https: //ox. rosettahub. com Students/Educators verify their email addresses by clicking on a link on the verification email sent by Rosetta. HUB Users with emails ending with the institution’s domain get approved automatically and receive an email with credentials after a few minutes Users who register with emails not linked to the institution get approved manually by the CPOC

The Rosetta. HUB automated mass onboarding process for AWS Initial setup for a new institution Institution’s CPOC registers at: https: //www. rosettahub. c om/institutions Set default limits for institution: budgets, budget limits, EC 2 instance perimeters, regions, services etc. Create AWS master account and assign it to the CPOC Enable detailed billing, cost explorer, create Organization Create CPOC’s Rosetta. HUB account Allocate domain name to institution, create dedicated registration website Create CPOC email linked to institution’s domain ending with @subdomain. rosettahub. com Create support ticket to increase AWS Organizations limit Configure CPOC’s AWS account for resources/billing monitoring Configure CPOC’s Rosetta. HUB account with default keys, S 3, EFS Affiliation of students & educators using Excel files CPOC uploads in Excel format lists of students and educators (first name, last name, email, graduation, bio link etc. ) CPOC selects the valid student and educators registrations and clicks process from the Rosetta. HUB users panel After a few minutes users receive their credentials for Rosetta. HUB Affiliation via individual registrations Students/Educators register individually at https: //subdomain. rosettahub. com Students verify their email addresses by clicking on a link on the verification email sent by Rosetta. HUB Users with emails ending with the institution’s domain get approved automatically and receive an email with credentials after a few minutes Users who register with emails not linked to the institution get approved manually by the CPOC

The Rosetta. HUB automated mass onboarding process for AWS Fully automated process for registering students and educators to AWS and Rosetta. HUB Add user to the Rosetta. HUB portal Create email account on RH email server ending with @subdomain. rosettahub. co m Create AWS Sub-Account linked to RH email using AWS Organization Create IAM user with rights based on the institution’s settings (instance types, regions, etc. ) Create Roles for EMR, and Elastic. Beanstalk and service roles for allowed services Add monitoring to each user’s account: Lambda function, Cloudtrail Create RH VPC where all managed RH EC 2 instances will be running Create secondary IAM user for RH keys enabling spot instances access Create user’s default S 3 bucket as well as the RH S 3 storage artifact that maps the bucket Create EFS storage to be used as a default working volume for RH managed instances Send welcome email with user’s credentials for Rosetta. HUB

Rosetta. HUB governance and management platform, modus operandi

The AWS building blocks Rosetta. HUB uses AWS building blocks to harness the AWS platform and make it work seamlessly for research and education. It leverages: Organizations to streamline the affiliation of students and faculty members. IAM to restrict the students and educators’ perimeters of action. Amazon S 3 AWS Lambda AWS Organizations Amazon Cloud. Watch, SNS and Lambda to monitor and control resources and budget consumption in real-time. STS to federate users access to the AWS console. Amazon SNS IAM AWS Cloud. Trail AWS STS

Monitoring and audit at scale Institution master AWS Account A Lambda function is inserted in each AWS account for real-time monitoring. The Lambda function on the master account is triggered a few times per day when a new billing report is made available by AWS. This triggers on Rosetta. HUB computation of all subaccounts usage. Actions are taken to disable sub-accounts which overconsumed. The Lambda functions on sub-accounts are triggered whenever EC 2, RDS, EBS resources are created or updated. They send information about compute and storage resources to the platform which estimates consumption on realtime and disables sub-accounts which exceed their hourly cost limits. Students and Educators AWS Accounts Monitors resources on real-time EC 2, EMR, ECS, RDS, EBS, S 3, EFS. . . Monitors costs on each subaccount

Dedicated Rosetta. HUB Users can authenticate through institutional SAML or Active Directory infrastructures. Registrations’ lifecycle management actions can be triggered programmatically by the Institutional students management system. Notification emails can be customized for the institution and custom Email servers can be used. Cloud resources lifecycle management and sharing actions can be scheduled with cron and rate tasks. A dedicated marketplace can be used as an institutional sharing platform for pedagogic and research artifacts (files and data, virtual labs, machines and containers images, etc. )

Contacts: [email protected] com Rosetta. HUB Website: https: //www. rosettahub. com To register a new institution: https: //www. rosettahub. com/institutions