CLOUD COMPUTING ARCHITECTURES APPLICATIONS LECTURE 11 SCIENTIFIC APPLICATIONS

CLOUD COMPUTING ARCHITECTURES & APPLICATIONS LECTURE #11 SCIENTIFIC APPLICATIONS FOR COMPUTING CLOUDS. FREELY ACCESSIBLE LARGE DATA SETS. LECTURERS LAZAR KIRCHEV, Ph. D <L. KIRCHEV@SAP. COM> ILIYAN NENOV <ILIYAN. NENOV@SAP. COM> KRUM BAKALSKY <KRUM. BAKALSKY@SAP. COM> 2 May, 2011

OUTLINE Scientific applications characteristics Issues with using current cloud infrastructures for scientific applications Perspectives Scientific for scientific applications which clouds offer projects using cloud computing 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 2

Introduction Cloud computing emerged to meet the needs of large businesses for flexible IT infrastructure Academia, on the other hand, uses for computational/storage purposes HPC and Grids Recently scientists turn towards cloud infrastructures to evaluate if scientific applications can benefit from them 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 3

Introduction Efforts in scientific community to use clouds - US Government funded research projects SDSC funded by NSF Department of Energy project Science clouds Small clouds, made available by several scientific institutions Amazon hosts Public Data Sets on their Elastic Block Store 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 4

Characteristics of scientific applications Data sharing, integration and analysis supported by metadata Very large data sets Very large execution systems (e. g. supercomputers) High performance – HPC and HTC, high system utilization Resource usage – exclusive, space-shared 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 5

Characteristics of scientific applications Consist of parallel jobs with extensive inter-process communication, workflows or bag-of-tasks with no inter-process communication Heterogeneous workloads, bottleneck could be CPU, I/O, memory or network Examples for scientific applications: CARMEN, BLAST, SNFactory 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 6

Issues with cloud infrastructures Clouds are meant to replace the small to medium-size enterprise data centers; these are much less utilized than the systems used for scientific computing Time sharing of resources and virtualization – increase concurrency of users, decrease performance The performance is not enough – performance analysis of four clouds (EC 2, Go. Grid, Elastic Hosts, Mosso) with scientific applications show this 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 7

Issues with cloud infrastructures Lack of dedicated access to the hardware and fine-grained sharing of resources associated with virtualization decreases performance Slow network connections between the virtual machines Optimized for business applications instead of HPC applications 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 8

Issues with cloud infrastructures Cloud environment not suitable for scientific applications (HPC cluster environment expected, or the application should be modified) Abstraction (provided by virtualization) vs. control (over hardware resources) Data access and interoperability – dynamically provisioned resources increases the need to easily integrate distributed data and repositories Difficult management and utilization of the virutalized resources 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 9

Perspectives provided by the cloud infrastructures Cost-effective for scientific computing in comparison with supporting the respective infrastructure in-house users can choose the most effective computing resources suited for their application and budget Appropriate alternative when resources are needed instantly and temporarily Can accommodate large data sets Elastic resource provisioning 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 10

Perspectives provided by the cloud infrastructures Easily provide access to a shared environment, which may evolve Virtual ownership of resources always can access your resources when needed Ease of deployment package the OS, libraries, patches and application codes on an image 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 11

Scientific projects using cloud computing - CARMEN Developed to solve neuroscience problems Deploy a set of generic e-science services (data management, service management, security, workflow enactment) on the cloud build domain-specific services on top 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 12

Scientific projects using cloud computing - Open Science Data Cloud A persistent, distributed storage and computing resource designed to manage, analyze, share, and archive scientific data Managed by Open Cloud Consortium Hadoop DFS and Map. Reduce, Sector, UDT (UDP-based protocol) Eucaliptus elasticity, Images compatible with Amazon’s 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 13

Scientific projects using cloud computing - Azure. Blast Implementation of the BLAST – an algorithm used in bioinformatics Implemented for Windows Azure, Microsoft’s Paa. S solution Azure. Blast is specifically tailored to utilize the Azure infrastructure Therefore it is well supported by Microsoft’s cloud 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 14

Conclusion Current cloud infrastructures are not optimized for scientific computing Performance Network latency Management of virtualized resources However, they are promising Economic efficiency Elasticity When HPC becomes in-scope for clouds performance will improve 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 15

END OF LECTURE #11

The Grid Headline area White space Drawing area 2011 Sofia University “Sv. Kliment Ohridski” > Faculty of Mathematics and Informatics > Cloud Computing Architecture and Applications 18