Accelerate on Windows Desktops HPCS Clouds with Pro

Accelerate on Windows Desktops, HPCS, Clouds with Pro. Active Parallel Suite D. Caromel, C. Delbé Speed: Application + Management: Productivity 1. INRIA Background & Active. Eon’s Mission 2. Pro. Active: Programming Scheduling Resourcing 3. Use Cases: Genomics Finance

1. Background & Mission 2

INRIA OASIS Team q A Research team, about 35 persons q 2004: First Pro. Active User Group q 2009, April: Pro. Active 4. 1, Distributed & Parallel: From Multi-cores to Enterprise GRIDs & Clouds 3 3

Active. Eon and INRIA synergy The best of research and industry! Customer Driven Developments Customers Partners Research Contracts High Added Value Services Located in Sophia Antipolis, between Nice and Cannes, Visitors Welcome! 4

Active. Eon Company Overview (1) q Co-developing, Support for Pro. Active Parallel Suite q Worldwide Customers: Fr, UK, USA Selected Partners: 5 5

Active. Eon Company Overview (2) q Open Source Software Editor Born of INRIA in 2007 q Core mission: Scale Beyond Limits q Co-develops Pro. Active Parallel Suite® q Provides high level services for Pro. Active q Ten years experience in development of parallel computing 6

2. Pro. Active Parallel Suite 7

Pro. Active Parallel Suite Java Parallel Toolkit Multi-Platform Job Scheduler Resource Manager Used in Production everyday: 50 Cores 300 Cores early 2010 Strong Features: q. Java Parallel Programming + Legacy + Integration + q. Portability: Windows, Linux, Mac + q. Versatility: Desktops, Cluster, Grid, Clouds = Perfect Flexibility 8

Pro. Active Parallel Suite Three fully compatible modules Clutch Power: Solid Building Blocks for Flexible Solutions Scheduling Programming Resourcing Scheduling Resource Management 9

Pro. Active Programming 10 10

Groups: Broadcast and Scatter (aka MPI) Broadcast is the default behavior Use a group as parameter, Scattered depends on rankings cg ag JVM s c 1 c 2 c 3 c 3 JVM ag. bar(cg); // broadcast cg Pro. Active. set. Scatter. Group(cg); ag. bar(cg); // scatter cg JVM 11 11 11

Dynamic Dispatch Group Slowest cg ag c 0 c 4 c 2 c 1 c 3 c 6 c 5 c 8 c 7 JVM c 9 Fastest JVM c 0 c 2 c 1 JVM c 3 c 4 c 6 c 5 c 8 c 7 c 9 ag. bar(cg); JVM 12 12 12

Optimizing GUI: IC 2 D 13 13

Optimizing GUI: IC 2 D 14 14

Optimizing GUI: IC 2 D 15 15

Pro. Active Scheduling 16 16 16

Pro. Active Scheduling Big Picture § Multi-platform Graphical Client (RCP) § File-based or LDAP authentication Pro. Active Scheduler § Static Workflow Job Scheduling, Native and Java tasks, Retry on Error, Priority Policy, Configuration Scripts, … Pro. Active Resource Manager § Dynamic and Static node sources, Resource Selection by script, Monitoring and Control GUI, … § Pro. Active Deployment capabilities: Desktops, Clusters, Clouds, … RESOURCES 17

Task. Flow Example : Picture Denoising Split Denoise Merge Denoise Job • with selection on native executable availability (Image. Magik, GREYstoration) • Multi-platform selection and command generation • with file transfer 18

Pro. Active Resourcing 19 19 19

Desktops + Clusters + Clouds 20 20

Versatility: Desktops, Grids, Clouds (EC 2), Multi-Core, VMs (Hyper-V, Xen, VMware, Kvm) Pro. Active Scheduler Pro. Active Resource Manager Static Policy LSF Timing Policy 12/24 Dynamic Workload Policy SSH EC 2 Dedicated resources Desktops Amazon EC 2 21

RESOURCING User Interface 22 22

2 mn Video: Scheduler, Resource Manager 23

3. Use Cases 24

Bio-Tech: Genomic Sequencing 25

Resources set up SOLID machine from 16 nodes Cluster Desktops Nodes can be dynamically added! Amazon EC 2 Clouds 26 26

Scalability: Internal SOLID Cluster to INRIA Cluster q The distributed version with Pro. Active of Mapreads has been tested on the INRIA cluster with two settings: the Reads file is split in either 30 or 10 slices q Use case: matching 31 millions sequences with the human genome (M=2, L=25) Reference point with 16 cores (same as in SOLi. D machine) 4 Time faster from 20 to 100 Speed Up of 80 / Th. Sequential 50 Hours 35 Minutes 27 27

Benchmark: Internal vs. EC 2 cloud only Execution time (min) Cost (US$) Standard PBS config 300 NA Pro. Active Amazon EC 2 340 20 US$ For only $3, 2/hour, the EC 2 setup has nearly the same performances as the local SOLi. D cluster 28

Pro. Active in Finance with Price-It Excel 29

Use case: Price-It and Pro. Active q Price-It® § C++ library developed by Pricing Partners § Pricing solution dedicated to highly complex financial derivatives q Accelerated Price-It® § A collaboration between Pricing Partners and Active. Eon § An implementation based on Pro. Active Scheduling q Specification and constraints § Accelerate Price-It® Excel product ▫ Built on Price-It® library, this product integrates an interface with Excel for input data management and results display § Focus on highly parallelizable Greek computation § Operating system: Windows 30 30

Price-it Sequential Computing C/C++ Price-it® library Price-it® Computation DLL Before distribution 31 31

Price-it Accelerated by Pro. Active C/C++ Price-it® library Price-it® Computation DLL Smooth integration of legacy code! Using distribution Scheduler Price-it® Computation DLL Worker 32 32

Accelerated Price-It performances q q Use case: Bermuda Vanilla, Model American MC Test conditions § One computation is split in 130 tasks that are distributed § Each task uses 300 ko More than 3 times faster with only 4 nodes! Even 6 times faster with 9 nodes! 4 nodes Sequential 5 nodes 6 nodes 7 nodes 8 nodes 9 nodes Distributed 33 33

Windows HPCS Benchmarks 34 34

Overall User Domain and Use Cases q Finance q Life Science and Bio. Tech Use Cases: q Engineering: Accelerating Matlab® and Scilab q IT q Numerical SPMD 35 35

Wrap Up 36

Pro. Active Parallel Suite q. Flexibility Clutch Power q. Portability: Windows, Linux, Mac q. Versatility: Desktops, Grids, Clouds Java Parallel Toolkit Multi-Platform Job Scheduler Resource Manager Free Professional Open Source Software Community and Enterprise Editions q Choose the Edition you need Community Users support Enterprise Professional support www. activeeon. com/downloads 37

Extra Material 38

Active. Eon: Mission, Services, Product Mission: q Accelerate your applications: Development, Deployment, Execution q Optimize your Hardware Infrastructure utilization q Easy and Accessible Parallel Computing & Clouds High-End Services & Support: q Training, Consulting, Integration, Development q Technical Support and Subscription q Partnerships (ISVs, VARs for OEM) Open Source Products: 39

Pro. Active : Active objects JVM A ag = new. Active (“A”, […], Virtual. Node) V v 1 = ag. foo (param); V v 2 = ag. bar (param); . . . v 1. bar(); //Wait-By-Necessity JVM A v 2 v 1 ag A WBN! V Java Object Active Object Future Object Proxy Req. Queue Request Thread Wait-By-Necessity is a Dataflow Synchronization 40 40 40