Parallel Processing SAP AG 2000 Applicationspecific Performance Workshop

Parallel Processing ã SAP AG 2000 Application-specific Performance Workshop /1

Content l Why ? l How ? n Strategies of Parallelization n Distribution of processes on several application servers l Possible Problems n Locking n Deadlocks n System resources l Using parallelization and workload distribution in own programs ã SAP AG 2000 Application-specific Performance Workshop /2

Sequential Work Only one work process is used while other processes are not involved Work Process 11 ã SAP AG 2000 Work Process 2 Application-specific Performance Workshop Work Process 3 /3 Work Process 4

Parallel Work Many processes are working at the same time Work Process 11 Process ã SAP AG 2000 Work Process 22 Process Application-specific Performance Workshop Work Process 33 Process /4 Work Process 4

Parallel Processing for mass data processing l Example n Request: process 500 IDOCs with avg. 1500 Line Items in POSUpload within 8 hours n System environment: 1 DB-Server, 5 App. -Servers with 4 CPUs each n avg. time for processing: about 10 minutes per IDOC n estimated time with sequential processing: about 80 hours Use the available capacities of your hardware and run 20 processes in parallel. Estimated time: about 4 hours ã SAP AG 2000 Application-specific Performance Workshop /5

Advantages l Parallel Processing is the key ã n to achieve a higher throughput n to minimize the required time until full completion of work n to use the available hardware capacities SAP AG 2000 Application-specific Performance Workshop /6

Content l Why ? l How ? n Strategies of Parallelization n Distribution of processes on several application servers l Possible Problems n Locking n Deadlocks n System resources l Using parallelization and workload distribution in own programs ã SAP AG 2000 Application-specific Performance Workshop /7

Example for a Process Chain l Each part must be finished completely before starting the next one ( total time t 1 + t 2 + t 3 + t 4) è Reduce processing time of each single part to reduce the processing time of the whole chain IDOC processing Replenishment t 1 Order t 2 . . . t 3 t 4 Time ã SAP AG 2000 Application-specific Performance Workshop /8

Load distribution on several processes (1) l Start and define parallel processes manually n Divide your work completely into several packages (e. g. by number intervals or site groups) beforehand n Multiple start of a report with different parameters in Background or as Batch Process 1 ã SAP AG 2000 Batch Process 2 Batch Process 3 Application-specific Performance Workshop Batch Process 4 /9 Batch Process 5

Load distribution on several processes (2) l Start one report that starts other processes with fixed packages n Hand over all work to a report n The report divides it completely into suitable packages (e. g. same number of IDOCs) n Further processes are started with fixed packages Dialog/Batch Process 1 Dialog Process 2 ã SAP AG 2000 Dialog Process 3 Application-specific Performance Workshop Dialog Process 4 / 10 Dialog Process 5

Workload Balancing (1) Static distribution beforehand can not consider the real workload. Therefore, time and resources might be wasted before follow-on processes are started. 17 WP 5 18 WP 4 13 WP 3 9 10 WP 2 5 6 WP 1 14 15 19 16 11 12 7 1 8 2 3 4 time waiting active T 1 = Time until Completion Goal: minimize T 1 ã SAP AG 2000 Application-specific Performance Workshop / 11 T 1

Load distribution on several processes (3) l Dynamic Workload Distribution n Hand over all work to a report n The report starts other processes and distributes the work in small packages dynamically (next item after completion of previous) “Distributor” Dialog Process 5 Dialog Process 1 Dialog Process 2 ã Dialog Process 4 Dialog Process 3 SAP AG 2000 Application-specific Performance Workshop / 12

Workload Balancing (2) Dynamic distribution at runtime is oriented to the real workload and so has a better chance to lead to an equal distribution, a higher throughput and thus reduces the time until full completion. WP 5 5 7 WP 4 10 4 WP 3 3 WP 2 9 11 6 12 2 WP 1 18 15 19 13 8 1 17 16 14 time waiting ã SAP AG 2000 T 1 (now) active Application-specific Performance Workshop / 13 T 1 (before)

Workload Balancing (3) The workload distribution can be optimized by processing work items with highest expected workload first. But the improvement must be weighed against the additional effort for analysis and sorting of the work items. WP 5 6 WP 4 10 4 15 2 5 WP 3 17 11 WP 2 8 7 WP 1 18 3 9 16 13 12 1 14 19 time waiting ã SAP AG 2000 T 1 (now) active Application-specific Performance Workshop / 14 T 1 (1 st improvement) approach)

Benefits of dynamic workload distribution l Even distribution of work packages n Oriented to the real processing time not to formal attributes like number ranges or number of work items. l Balanced workload distribution n minimal impact of unforeseen delays in processing of several work items l Note: the program must be designed appropriately (contrary to the other strategies) n ã not all programs are yet designed like that, but currently many transactions are reworked to support this kind of workload distribution. SAP AG 2000 Application-specific Performance Workshop / 15

Content l Why ? l How ? n Strategies of Parallelization n Distribution of processes on several application servers l Possible Problems n Locking n Deadlocks n System resources l Using parallelization and workload distribution in own programs ã SAP AG 2000 Application-specific Performance Workshop / 16

DON’T Application Server Process Process Process Process Process Process Process ã SAP AG 2000 Application-specific Performance Workshop / 17

DO Application Server Process Process Process Process Process Process Process ã SAP AG 2000 Application-specific Performance Workshop / 18

Workload distribution using a. RFC server groups l Usage of a. RFC Server groups ã n ARFC Server Groups can be used to occupy several dialog work processes (DIAGS) on several R/3 instances for parallel processing. n The resource determination is performed at runtime and is based on the current system load. n The quotas which are used to determine the available resources can be configured separately for each instance. n NOTE: The quotas are valid for the R/3 instance and are NOT related to a Server group! SAP AG 2000 Application-specific Performance Workshop / 22

ARFC: Resource determination a. RFC serv. group CALL FUNCTION. . . DESTINATION IN GROUP. . . STARTING NEW TASK 'ABC' Use quotas? Group B yes Message-Server ? Instance 1 Dispatcher-Queue WP 1 WP 2 WP 3 . . . ã SAP AG 2000 1) Dispatcher queue: usage rdisp /rfc_max_queue % Instance 2 and 2) At least rdisp/rfc_max_own_used_wp free dialog work processes and 3) At most rdisp/rfc_max_login % of allowed logins and . . . Application-specific Performance Workshop / 23 Dispatcher-Queue WP 1 WP 2 WP 3 . . .

ARFC Server Group Quotas (RZ 12) percentage absolute value ã SAP AG 2000 Application-specific Performance Workshop / 24

Content l Why ? l How ? n Strategies of Parallelization n Distribution of processes on several application servers l Possible Problems n Locking n Deadlocks n System resources l Using parallelization and workload distribution in own programs ã SAP AG 2000 Application-specific Performance Workshop / 25

Locking l Several processes can block each other by database locks, enqueue locks or semaphore locks (here: a database lock). Update record 1 Commit WP 1 Update record 1 Commit WP 2 Commit Update record 1 WP 3 Time Lock on Record 1: Work process must wait for the release of the lock ã SAP AG 2000 Application-specific Performance Workshop / 26

Database Lock Monitor (DB 01) Screenshot of a system blocked by database locks. All update processes were blocked except one (PID 14. 610) ã SAP AG 2000 Application-specific Performance Workshop / 27

Example: Locking on Number Range Table NRIV l The Number Range Table NRIV is an object of central importance for the whole system: n n n ã The unique ID-Numbers for new objects are taken from there (distinct ranges for different types). Uniqueness must be guaranteed even if several processes access a number range at the same time. Locking problems can decrease throughput. SAP AG 2000 Application-specific Performance Workshop / 28

Number Range / Not Buffered (1) Allocate Number Application Servers § no gaps § chronological order ã SAP AG 2000 Application-specific Performance Workshop Database Server (NRIV) § DB locking system wide § prevents parallelism / 29

Number Range Buffering / Local File (2) R/3 Instance Allocate Number § not continuous, not chronological § reduce DB locking § higher parallelism § gaps occurred can be reported (RSSNR 0 A 1) ã SAP AG 2000 Application-specific Performance Workshop Database Server (NRIV_LOKAL) § DB locking on each R/3 Instance (solution: see OSS note 179224) / 30

Allocate Number Main Memory of R/3 -Instances Allocate Number ã Transmit Number Range Buffering / Main Memory (3) Database Server (NRIV) § parallel processing § numbers can get lost § minimizes data base accesses § no reporting of gaps possible SAP AG 2000 Application-specific Performance Workshop / 31

Maintaining the Number Allocation Transaction SNRO No buffering 1 Main Memory Local File 3 2 1000 for 2. 100 for 3. The configuration affects the number allocation of this object type in general ! ã SAP AG 2000 Application-specific Performance Workshop / 32

Attention ! Legal Reasons might prohibit Number Range Buffering. ã SAP AG 2000 Application-specific Performance Workshop / 33

Content l Why ? l How ? n Strategies of Parallelization n Distribution of processes on several application servers l Possible Problems n Locking n Deadlocks n System resources l Using parallelization and workload distribution in own programs ã SAP AG 2000 Application-specific Performance Workshop / 36

Deadlocks (1) l Processing data in a non-sorted sequence can create deadlocks. Process 1 must wait for Release of the Lock by Process 2 DEADLOCK ! Process 2 must wait for Release of the Lock by Process 1 Database Table Work Process 1 1 8 4 12 16 7 3 11 ã SAP AG 2000 1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16 Application-specific Performance Workshop Work Process 2 / 37 16 9 2 10 5 4 13 3

Deadlocks (2) l Processing data in the same sorted sequence avoids deadlocks. All records are processed COMMIT WORK The lock is released. Process 2 can continue Work Process 1 1 3 4 7 8 11 12 16 ã SAP AG 2000 NO DEADLOCK Database Table 1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16 Application-specific Performance Workshop Process 2 must wait for Release of the lock by Process 1 Work Process 2 2 3 4 5 9 10 13 16 / 38

Content l Why ? l How ? n Strategies of Parallelization n Distribution of processes on several application servers l Possible Problems n Locking n Deadlocks n System resources l Using parallelization and workload distribution in own programs ã SAP AG 2000 Application-specific Performance Workshop / 39

Higher requirements to the system l Using many processes in parallel makes higher demands on system components (CPU, memory, I/O system, network, …). l Any bottleneck will prevent a further increase of the throughput. Eliminate this problem, as additional processes will not have any positive effects. l Have a special look on the CPU load and the configuration of the I/O system on the database. Throughput optimal No of Parallel processes ã SAP AG 2000 Application-specific Performance Workshop / 40

Content l Why ? l How ? n Strategies of Parallelization n Distribution of processes on several application servers l Possible Problems n Locking n Deadlocks n System resources l Using parallelization and workload distribution in own programs ã SAP AG 2000 Application-specific Performance Workshop / 41

Hints for the Design of your own programs (1) l General Recommendation: Design your program for parallel work n even if it is not yet necessary - the day might come. A subsequent redesign takes considerably more time. l Be careful with locks n keep them as short as possible, especially for objects of central importance in the system (remember: you are not alone on the system) n first perform the INSERTS - then the UPDATES (it is less probable that someone else needs access to a new record) n think about a useful granularity for exclusive enqueue locks l Use the same sorted sequences for the processing of data in the whole system to avoid deadlocks n ã ask your colleagues, look at standard programs SAP AG 2000 Application-specific Performance Workshop / 42

Hints for the Design of your own programs (2) l In the initial screen provide fields to configure parallel processing: n definition of a Server Group, n the maximum package size and maximum number of parallel processes n switch on or off parallel processing l Take care to balance workload n collect (and sort? ) the required data in order to define appropriate work packages first (attention: internal tables must not become too large) n then start other processes and distribute the packages to them (just wait for a process to finish its work and pass the next item immediately) l Use local update (“set update task local”) ã n preserves benefit of short locking time on the database n avoids communication overhead to database (tables VBMOD, VBDATA) n end of process can be determined (important for follow-on processes) n But: increases number of database processes modifying data SAP AG 2000 Application-specific Performance Workshop / 43

Useful transactions l ST 05 (Performance Trace) n Optimize the process concerning database operations, enqueues, duration of locks (examination in single user mode !) l SM 66 (system wide work process overview) n Check if load distribution works as expected l ST 06 (OS Monitor) n Watch the system load (CPU load and Memory usage) l DB 01 (database monitor) n Look for exclusive database locks l SM 12 (Lock entries) n Watch the entries of the ENQ Table l RZ 12 (RFC Server Group Maintenance) n ã Define and configure RFC server groups SAP AG 2000 Application-specific Performance Workshop / 50

Monitoring Transactions (sm 66) ã SAP AG 2000 Application-specific Performance Workshop / 51

Some Interesting OSS notes l 179224 (number range buffering on work process level) l 74141 (description of resource management for server groups) l 99284 (load balancing with a. RFC server groups) l 103523 (load balancing with a. RFC server groups) l 142419 (load balancing with a. RFC server groups) l 186704 (error with server group ‘default’) l 195927 (problem with ALE load distribution) l 155345 (better load balancing during IDOC processing) l 201106 (reports to test a. RFC server groups) l 199840 (template for a report with a mutual lock avoiding distribution) ã SAP AG 2000 Application-specific Performance Workshop / 56