Grid Flow Workflow Management for Grid Computing Kavita

  • Slides: 22
Download presentation
Grid. Flow: Workflow Management for Grid Computing Kavita Shinde

Grid. Flow: Workflow Management for Grid Computing Kavita Shinde

Outline Introduction n Grid Resource Management n Grid Workflow Management n An Example Scenario

Outline Introduction n Grid Resource Management n Grid Workflow Management n An Example Scenario n Conclusion n

Introduction n Grid. Fow ¨ ¨ given a set of workflow tasks and a

Introduction n Grid. Fow ¨ ¨ given a set of workflow tasks and a set of resources, how do we map them to Grid resources? workflow management systems developed at University of Warwick developed on top of an agent-based resource management system for Grid computing(ARMS) focus is on service-level scheduling and workflow management

Grid Resource Management n Three Layers of resource management system within the Grid. Flow

Grid Resource Management n Three Layers of resource management system within the Grid. Flow system ¨ Grid Resource n high-end computing or storage resource n accessed remotely n Multiprocessors, or clusters of workstations or PCs with large disk storage space ¨ Local Grid n multiple grid resources that belong to one organization n resources are connected with high speed networks ¨ Global Grid n consists of all local Grids

Grid Resource Management n PACE ¨a toolset for resource performance and usage analysis ¨

Grid Resource Management n PACE ¨a toolset for resource performance and usage analysis ¨ takes separate resource and application models as inputs and is able to predict the execution time of a task prior to run time ¨ scalability(execution time vs. level of parallelism) can be determine n helps in preventing over-occupying of resources q useful when trying to interleave sub-workflows as much as possible

Grid Resource Management n Titan ¨ grid resource manager ¨ locates a suitable resource

Grid Resource Management n Titan ¨ grid resource manager ¨ locates a suitable resource set and passes the subworkflow to a local scheduler ¨ utilizes free processors to minimize idle-time and improve throughput ¨ supported by the PACE performance predictive data

Grid Resource Management n ARMS ¨ main component – agent ¨ agent – representative

Grid Resource Management n ARMS ¨ main component – agent ¨ agent – representative of a local grid at a global level of grid resource management ¨ agents cooperate with each other to find the available resources and there characteristics n dispatch requests that can not be satisfied locally to neighboring agents

Grid Workflow Management The implementation of grid workflow management is carried out at multiple

Grid Workflow Management The implementation of grid workflow management is carried out at multiple layers q Tasks n n basic building block of application e. g. . MPI(Message Passing Interface) and PVM(Parallel Virtual Machine) jobs running on multiple processors tasks ¨ Sub-workflows n n a flow of closely related tasks that is to be executed in a predefined sequence on grid resources of a local grid usually significant communication between tasks, but resource conflicts may occur when multiple sub-workflows require the same resource simultaneously ¨ Workflows n a flow of several different sub-workflows

n. Grid. Flow user portal qprovides graphical user interface to compose workflow elements and

n. Grid. Flow user portal qprovides graphical user interface to compose workflow elements and access additional grid services n. LGSS ¨handles conflicts - scheduled subworkflows may belong to different workflows n ARMS ¨represents a local Grid at a global level of Grid resource management, and conducts local Grid sub-workflow scheduling n Globus MDS qprovides information about the available resources on the Grid and their status n Titan qutilizes performance data obtained from PACE for resource scheduling

Grid Workflow Management n GGWM ¨ Simulation n takes place before a grid workflow

Grid Workflow Management n GGWM ¨ Simulation n takes place before a grid workflow is actually executed, workflow schedule is achieved n returns simulation results to Grid. Flow portal for user agreement ¨ Execution n executed according to the simulated schedule ¨ the actual execution may differ - dynamic nature of grid n delays - send back to the simulation engine & rescheduled ¨ Monitoring n provides access to real-time status reports of tasks or subworkflow execution

Global Grid Workflow Management n Scheduling Algorithm ¨ initialize all properties of each sub-workflow

Global Grid Workflow Management n Scheduling Algorithm ¨ initialize all properties of each sub-workflow – null ¨ look for a schedulable sub-workflow n ensure pre- sub-workflows have all been scheduled ¨ configure the start time of the chosen sub-workflow to be the latest end time of its pre- sub-workflows ¨ submit the start time and the sub-workflow to a grid level Agent(ARMS) n finds a suitable local grid using LGSS

Global Grid Workflow Management ¨ ARMS reschedules the less critical sub-workflows ¨ algorithm relies

Global Grid Workflow Management ¨ ARMS reschedules the less critical sub-workflows ¨ algorithm relies heavily on the simulation results of LGSS

Workflow W : a set of subworkflows Si(i=1, …. n) Si and Sn starting

Workflow W : a set of subworkflows Si(i=1, …. n) Si and Sn starting and ending points pi : number of pre- sub-workflows of Si qi : number of post- sub-workflows of Si G: global grid – set of local grids Lj(j=1…. m) k: true if sub-workflow is scheduled else false

Local Grid Sub-Workflow Scheduling n Scheduling Algorithm ¨ very similar to GGWM ¨ has

Local Grid Sub-Workflow Scheduling n Scheduling Algorithm ¨ very similar to GGWM ¨ has to deal with multiple tasks that may belong to different workflows ¨ start time of the chosen task can’t be configured with the latest end time of its pre-tasks directly n n resource conflicts ¨ Executes the task with the higher priority first gives higher priority to a possibly earlier enabled task

Fuzzy Time Operations n n LGSS and GGWM algorithms are implemented using fuzzy timing

Fuzzy Time Operations n n LGSS and GGWM algorithms are implemented using fuzzy timing techniques fuzzy time function – ¨ gives numerical estimate of the possibility that an event arrives at time advantages: can be computed very fast suitable for scheduling time critical applications q they do not necessarily provide the best scheduling solution

 1( ) = 0. 5(0, 2, 6, 7) 2( ) = (2, 4,

1( ) = 0. 5(0, 2, 6, 7) 2( ) = (2, 4, 4, 6) na: possibility distributions of 1 and nb: latest arrival distribution of 1 and nc: earliest enabling time 2 2 operator min – intersection of 1 and 2 nd: ne: nf: operator max – union of 1 and 2 sum of 1 and 2 ¨min(0. 5, 1)(0+2, 2+4, 6+4, 7+6)=0. 5(2, 6, 10, 13)

An Example Scenario n n n W 1, W 2: Workflows L 1, L

An Example Scenario n n n W 1, W 2: Workflows L 1, L 2: Local Grids task A 2 of sub-workflow S 3 from W 1 is being executed S 3 from W 2 is to be scheduled resource conflict between A 3 and A 4 schedule aims to find the e 5( )

An Example Scenario n n task enabling times – from pre-task end times task

An Example Scenario n n task enabling times – from pre-task end times task execution times – from TITAN system supported by PACE functions a 3( )=(3, 5, 5, 7); d 3( )=(5, 6, 7, 8); a 4( )=(0, 3, 3, 5); d 4( )=(10, 12, 14, 16); d 5( )=(2, 5, 6, 9);

An Example Scenario using LGSS s 3( ) = min{(3, 5, 5, 7), earliest{(3,

An Example Scenario using LGSS s 3( ) = min{(3, 5, 5, 7), earliest{(3, 5, 5, 7), (0, 3, 3, 5)}} = min{(3, 5, 5, 7), (0, 3, 3, 5)} = 0. 5(3, 4, 4, 5) s 4( ) = min{(0, 3, 3, 5), earliest{(3, 5, 5, 7), (0, 3, 3, 5)}} = min{(0, 3, 3, 5), (0, 3, 3, 5)} = (0, 3, 3, 5) e 13( )= sum{0. 5(3, 4, 4, 5), (5, 6, 7, 8)} = 0. 5(8, 10, 11, 13)

An Example Scenario e 14( )= sum{latest{0. 5(8, 10, 11, 13), (0, 3, 3,

An Example Scenario e 14( )= sum{latest{0. 5(8, 10, 11, 13), (0, 3, 3, 5)}, (10, 12, 14, 16)} = sum{0. 5(8, 10, 11, 13), (10, 12, 14, 16)} = 0. 5(18, 22, 25, 29) e 24( )= sum{(0, 3, 3, 5)}, (10, 12, 14, 16)} = (10, 15, 17, 21) e 23( )= sum{latest{ (10, 15, 17, 21), 0. 5(3, 4, 4, 5)}, (5, 6, 7, 8)} = sun{0. 5(10, 12. 5, 26, 29), (5, 6, 7, 8)} = 0. 5(15, 18. 5, 26, 29) e 4( )= max{0. 5(18, 22, 25, 29), (10, 15, 17, 21)} = (10, 15, 17, 29)

An Example Scenario e 5( )= sum{(10, 15, 17, 29), (2, 5, 6, 9)}

An Example Scenario e 5( )= sum{(10, 15, 17, 29), (2, 5, 6, 9)} = (12, 20, 23, 38) so S 3 from W 2 will complete on local grid L 1 most likely between 20 to 23 submit this data to GGWM – decides whether the local grid L 1 should be allocated the sub-workflow S 3 from W 2

Conclusion n n the fuzzy timing technique provides a good solution to the conflict

Conclusion n n the fuzzy timing technique provides a good solution to the conflict solving problem arising from grid workflow management issue results indicate that local and global grid workflow management can coordinate with each other to optimize workflow execution time and solve conflicts of interest useful in highly dynamic grid environments large network latencies exists and application performance is difficult to predict accurately needs more flexible cooperation among different grid services and components which challenges security