Beyond Bi G Grid Dr ir Anwar Osseyran

Beyond Bi. G Grid Dr. ir. Anwar Osseyran SARA National High Performance Computing Services Big Grid and Beyond – Amsterdam 120926

Bi. G Grid was a Must ATLAS Computing requirements over time q q q 1995: 100 TB, 107 MIPS : 2001: 1900 TB, 7*107 MIPS : 2007: 70000 TB, 55*107 MIPS 2010 LHC START 2011: 83000 TB, 61*107 MIPS 45% 40% From the start on it was clear that no center could provide ALL computing even for one LHC experiment Similar needs in life science, Astronomy, … Big Grid and Beyond – Amsterdam 120926 15%

Bi. G Grid: a National Effort Partner agreement between Bi. G Grid and SARA already in 2008 National Grid infrastructure (Nikhef, RUG, SARA, Philips, …) To support: q q LHC: Dutch Tier-1 LOFAR: Long Term Archive Life Science: Dutch LSG But also Alfa and Gamma Sciences Co-Development and Support Big Grid and Beyond – Amsterdam 120926

Bi. G Grid: Big Effort 104 resource requests (35 cloud) projects (and counting !) More then 30 scientific communities served q From High Energy Physics: 1000 s scientists q To Dutch Bio Banking Collaboration SARA effort 06/2007 – 12/2012: q Operations: 93 k hours q Support and Development: 31 k hours q Bi. G Grid Investment in HPC eco-system: 4 contribution to Supercomputer Huygens 4 Grid Compute & Storage, Cluster computing, Clouds, Big Data Big Grid and Beyond – Amsterdam 120926

Bi. G Grid @SARA User-Centric Innovation Rapid prototyping and Proof-of. Concept in pilots with concrete use cases (small nr. of users) Pre-production with limited number of users Production for larger user community 3 examples: q HPC Cloud q Beehub storage for science q Hadoop Big Data services Started by SARA, early adopted and co-funded by Bi. G Grid Big Grid and Beyond – Amsterdam 120926

HPC Cloud services in NL Dynamic and flexible selfservice Define your own working environment Define your own HPC cluster Work loads: q scale up workstation loads q scale up complex work flows High User Productivity Big Grid and Beyond – Amsterdam 120926

Bee. Hub: Storage for scientists Integral scalable cloud storage service Easy to use, store and share data, based on Web. DAV For all desktops OS’s & distributed computing infra Integrated with SURFconext Big Grid and Beyond – Amsterdam 120926

Hadoop Big Data Services Open source Apache Hadoop along with a framework for map reduce jobs over data Move processing to the data Seamless scalability 2009: Pilot Hadoop on Cloud 2010: Test cluster available for scientists 2011: Funding for production services Po. C: 20 users 2012: Production started with 72 machines * 8 cores / 8 TB storage / 64 GB RAM, 3 devops, team of consultants Lot of interest from science and business Big Grid and Beyond – Amsterdam 120926

Life after Bi. G Grid… Bi. G Grid the project ends this year (SURF)SARA is committed to continue its effort: q with the current infrastructure partners q With suppliers and support partners q with YOU… We will strive for a seamless transition to the new situation Beyond Bi. G Grid: Sector Clouds and Big Data… Big Grid and Beyond – Amsterdam 120926

Beyond the Grid @CERN Helix Nebula: The Science Cloud Establish a sustainable European cloud computing infrastructure to provide stable computing capacities and services that elastically meet demand Pilot phase : Proof of Concept deployments on 3 commercial cloud providers (ATOS, Cloud. Sigma, T-Systems) Big Grid and Beyond – Amsterdam 120926

Dutch Federative Clouds A Prosumer Approach Sustainability: q Federation of existing data centers. Scalability & Economy of Scale q Maximize utilization factor Security and performance: q Sector specific security in place Community support: q Customer intimacy & sector expertise q Community collaboration and sharing Also for better partnership with Public Clouds Big Grid and Beyond – Amsterdam 120926

To end with: Big Data is Changing Science Big Data is changing science, medicine, business, and technology. A whole new way of science: correlation supersedes causation, coherent models or unified theories… Biggest challenge for science & business is not storing data but how to make sense of it. Big Grid and Beyond – Amsterdam 120926

Also on behalf of Bi. G Grid… Thank YOU Big Grid and Beyond – Amsterdam 120926