COMP Superscalar Bringing GRID superscalar and GCM together

COMP Superscalar: Bringing GRID superscalar and GCM together Enric Tejedor Universitat Politècnica de Catalunya etejedor@ac. upc. edu Rosa M. Badia Barcelona Supercomputing Center rosa. m. badia@bsc. es V Pro. Active and GCM User Group, Grid. COMP conference October 21, 2008. INRIA, Sophia Antipolis, France

Outline • • Introduction • • Grid Component Model GRID superscalar Programming model Runtime operation • • • Initialisation Task processing Finalisation and error handling Conclusions V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

1. Introduction V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Basic concepts • • The Grid Component Model (GCM) • • Core. GRID No. E, Grid. COMP Distributed, based on Fractal, intended for the Grid GRID superscalar • • • Framework to ease the development of Grid-unaware applications Simple programming model: Grid as transparent as possible Runtime that optimises the performance of the application (exploiting possible concurrency) V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

A componentised Grid framework • • GRID superscalar is suitable to benefit from the GCM features Componentisation process: identify the inner functionalities of the runtime, and assign each one to a separate component • Result: COMP Superscalar, whose runtime gains in: • Reusability • Flexibility • Deployability • Separation of concerns • Ease of development V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Componentised runtime V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

2. Programming model V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Programming model Parallel Resources (multicore, SMP, cluster, grid) Synchronization, Resource 1 results transfer Sequential Application. . . for (i=0; i<N; i++){ T 1 (data 1, data 2); T 2 (data 4, data 5); T 3 (data 2, data 5, data 6); T 4 (data 7, data 8); T 5 (data 6, data 8, data 9); }. . . Task selection + Resource 2 parameters direction Resource 3 (input, output, inout) . . . Resource N T 10 T 20 T 40 T 30 Task graph creation based on data T 50 Scheduling, T 11 T 21 data transfer, T 41 T 31 task execution precedence T 51 T 12 … V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

COMPSs programming model initialize(f 1); for (int i = 0; i < 2; i++) { gen. Random(f 2); Java application add(f 1, f 2); } Java interface print(f 2); public interface Sum. Itf { @Class. Name(“example. Sum") @Method. Constraints(OSType = "Linux") Implementation Task constraints void gen. Random( @Param. Metadata(type = Type. FILE, direction = Direction. OUT) String f ); @Class. Name(“example. Sum"). . . } V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France) Parameter metadata

Custom Java Class Loader Annotated interface Java app code input Custom Loader inserts calls to COMPSs runtime uses Javassist V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

3. Runtime operation V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Implementation details • Base technologies: • • • Java as programming language Pro. Active: implementation of the GCM model, used to build the components Java. GAT: used for job submission and file transfer V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Initialization • Multicast invocation • Forwarded to all subcomponents TA JM TS init() FM FIP FTM V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Initialization • Multicast invocation • Reduction of return values TA JM TS Reduction FM Reduction FIP FTM V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Task processing (1) • Application submits task • Task Analyser receives the request TA TS T 1 gen. Rd JM T 3 gen. Rd execute. Task(…) f 1 FM FIP f 1 T 2 add f 2 FTM V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France) T 4 add

Task processing (2) • File Information Provider is contacted • File accesses of the task are registered TA JM TS execute. Task(…) FM FIP FTM V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Task processing (3) • Task Analyser discovers dependencies • Dependency-free tasks are sent for scheduling TA TS JM execute. Task(…) FM FIP FTM V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Task processing (4) • Task Scheduler decides where to execute tasks • Job Manager is informed of this decision Task constraints Resource capabilities TS TA execute. Task(…) JM Scheduling algorithm FM FIP FTM V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Task processing (5) • Job Manager checks the necessary file transfers • File Transfer Manager actually performs them TA TS JM execute. Task(…) FM FIP FTM V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France) GAT

Task processing (6) • FTM informs of the end of transfers for a task • Task Grid job, submitted for execution TA TS JM execute. Task(…) FM FIP FTM V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France) GAT

Task processing (7) • Callback for the end of a job is received • The notification is forwarded to the Task Analyser TA TS JM execute. Task(…) FM FIP FTM V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France) GAT

Safe stopping process for the runtime • • COMPSs subcomponents have data dependencies Modification of the Life Cycle Controller • Components stopped in a safe order: TS, TA, JM, FTM, FIP • Otherwise, a deadlock could happen TA TA TS LCC JM FM FIP FTM V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

4. Conclusions V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Conclusions • • COMP Superscalar: componentised version of GRID superscalar • • • GCM-based Hierarchical runtime: separation of concerns Task scheduling, file and resource management Straightforward programming model for Grid-unaware Java applications • Feedback about GCM-Pro. Active • • • Component distribution Performance: no noticeable overhead for a medium number of tasks Component composition and dynamic reconfiguration: Grid IDE V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)

Thank you! Questions? V Pro. Active and GCM User Group. October 21 st 2008, Sophia Antipolis (France)