Oracle Cache Fusion In Operation Agenda Cache Fusion

Oracle Cache Fusion – In Operation

} Agenda } Cache Fusion – What is it? – Cache Coherency Vs. Cache Fusion – Key Components and terminology } Cache Fusion in operation – Lock Mastering & Resource Affinity – Type of Contentions – Cache Fusion – II – Examples } Instance Crash Recovery in RAC – Key Components in a Instance crash – I Pass recovery – II Pass recovery

Best Place to Bank Best Place to Work Excellence in Service Delivery Cache Fusion – What is it?

} What is it? Oracle introduced the framework of sharing data using private interconnects between the nodes, which was used only for messaging purposes in previous versions. This protocol is Cache Fusion. Data blocks are shipped throughout the network similar to messages, reducing the most expensive component of data transfer, disk I/O, to data sharing. According to the manual: Process that implement Cache Fusion. It maintains the block mode for blocks in the global role. It is responsible for block transfers between instances. The Global Cache Service employs various background processes such as the Global Cache Service Processes (LMSn) and Global Enqueue Service Daemon (LMD). A diskless cache coherency mechanism in Oracle Real Application Clusters that provides copies of blocks directly from a holding instance's memory cache to a requesting instance's memory cache.

} Cache Coherency } According to Manual – The synchronization of data in multiple caches so that reading a memory location through any cache will return the most recent data written to that location through any other cache. Sometimes called cache consistency. } Can We say its something to maintain the resource (block) status, If so, the following two together provides the same for us. – GCS (Global Cache Services) – GES (Global Enqueue Services) In the name of Global Resource Directory

} Now both together ……… } The GCS manages all types of data blocks. Cache coherency is maintained through the GCS by requiring that instances acquire a resource (lock or enqueue on a block) cluster-wide before modifying or reading a database block. The GCS is used to synchronize global cache access, allowing only one instance to modify a block at any single point in time. The GCS, through the RAC wide Global Services Directory, ensures that the status of data blocks cached in any mode in the cluster is globally visible and maintained. } Oracle’s RAC has multi-versioning architecture. This multi-versioning architecture distinguishes between current data blocks and one or more consistent read (CR) versions of a block. A current block contains changes for all committed and yet-to-be-committed transactions. A consistent read (CR) version of a block represents a consistent snapshot of the data at a previous point in time. A data block can reside in many buffer caches under the auspices of shared resources. } In Oracle 9 i RAC, applying rollback segment information to current blocks produces consistent read versions of a block. Both the current and consistent read blocks are managed by the GCS. } To transfer data blocks among database caches, buffers are shipped by means of the high speed IPC interconnect. Disk writes are only required for cache replacement. A past image (PI) of a block is kept in memory before the block is sent if it is a dirty (modified) block. In the event of failure, Oracle reconstructs the current version of the block by reading the PI blocks.

} Background Process and their roles } LMSx – Lock Monitor Services (GCS) } Primarily responsible for shipping the blocks across buffers } Provides/creates a CR image whenever there is cross instance call for a dirtyblcok } LMS must also check constantly with the LMD background process (or our GES process) to get the lock requests placed by the LMD process. } Parameter: GCS_SERVER_PROCESS upto 36 as of 10. 2, Min. cpu_count/2 } LMON – Lock Monitor Process (GES) } LMON Processes manages the global locks & resources. } Reconfiguration of locks & resources when an instance joins or leaves the cluster are handled by LMON ( During reconfiguration LMON generate the trace files) } LMON also provides cluster group services. } LMD – Lock Manager Daemon } LMD process performs global lock deadlock detection local and remote. (GES) } Also monitors for lock conversion timeouts. } Basically maintains the lock queues, traverse through the GES structures } LCK – Lock Process } Manages instance resource requests & cross instance calls for shared resources. } During instance recovery, it builds a list of invalid lock elements and validates lock elements. } DIAG – Diagnostic Daemon – Oracle 10 g - this one new background processes ( New enhanced diagnosability framework). Regularly monitors the health of the instance. Also checks instance hangs & deadlocks.

Best Place to Bank History of Cache Fusion Best Place to Work Excellence in Service Delivery Oracle Release Feature Description Prior to 8. 1. 5 OPS used disk-based pings 8. 1. 5 Cache Fusion I or Consistent Read Server Consistent read version of the block is transferred over the interconnect 9 i Cache Fusion II (write/write cache fusion) Current version of the block is transferred over the interconnect 10 g R 1 Oracle Cluster Ready Services (CRS) CRS eliminates the need for third-party clusterware, though it can be used 10 g R 2 Oracle CRS for High Availability CRS provides high availability for non-Oracle applications

} Key Components in Cache Fusion Ping The transfer of a data block from one instance’s buffer cache to another instance’s buffer cache is known as a ping. Whenever an instance needs a block, it sends a request to the lock master to obtain a lock in the desired mode. If another lock resides on the same block, the master will ask the current holder to downgrade/release the current lock. , this process is known as a blocking asynchronous trap (BAST). When an instance receives a BAST it downgrades the lock as soon as possible. However, before downgrading the lock, it might have to write the corresponding block to disk. This operation sequence is known as disk ping or a hard ping. CR Fabrication When ever there is Consistent read request from any other instance, the holding instance (LMS) has to create a Consistent read image by applying the undo information to the Current Block. Since CR fabrication is I/O expensive which requires a undo into the buffer and apply the undo image etc. Past Image (PI) Blocks PI blocks are copies of blocks in the local buffer cache. Whenever an instance has to send a block it has recently modified to another instance, it preserves a copy of that block, marking it as PI. An instance is obliged to keep Pls until that block is written to the disk by the current owner of the block. Pls are discarded after the latest version of the block is written to disk. When a block is written to disk and is known to have global role, indicating the presence of Pls in other instances’ buffer caches, Global Cache Services (GCS) informs the instance holding the Pls to discard the Pls. With Cache Fusion, a block is written to disk to satisfy checkpoint requests and so on, not to transfer the block from one instance to another via disk. Lock Mastering The memory structure where GCS keeps information about a data block (and other sharable resources) usage is known as the lock resource. The responsibility of tracking locks is distributed among all the instances and the required memory also comes from the participating instances’ System Global Area (SGA). Due to this distributed ownership of the resources, a master node exists for each lock resource. The master node maintains complete information about current users and requestors for the lock resource. The master node also contains information about the Pls of the block.

} Resource Affinity and Dynamic remastering } Each block is mastered in any one of the instance at any given point of time } Resource Master can be changed based on frequency of the block that is requested by other instances – For a period of 10 Mins if an instance request 50 times for a particular resource the requested instance become the master. This is called resource affinity - Block Mastering } In Oracle 9. 2 – documentation describes dynamic remastering – not implemented in code } In Oracle 10. 1 – – work at data file level very high threshold so difficult to test does occur on some customer sites may cause LMON process to crash in 10. 1. 0. 4 } } bug 3659289 - patch available fixed in 10. 1. 0. 5/10. 2. 0. 1 } In Oracle 10. 2 – works at object level – thresholds are relatively low. – Object re mastering is recorded in V$GCSPFMASTER_INFO

} Cache Fusion- Possible Types of Contention } Contention of a resource occurs when two or more instances want the same resource. If a resource such as a data block is being used by an instance and is needed by another instance at the same time, a contention occurs. There are three types of contention for data blocks: } Read/Read contention Read/read contention is never a problem because of the shared disk system. A block read by one instance can be read by other instances without the intervention of GCS. } Write/Read contention Write/read contention was addressed in Oracle 8 i by the consistent read server. The holding instance constructs the CR block and ships the requesting instance using interconnects. } Write/Write contention Write/write contention is addressed by the Cache Fusion technology. Since Oracle 9 i, cluster interconnect is used in some cases to ship data blocks among the instances that need to modify the same data block simultaneously.

} Prior to Cache Fusion (before 8. 1. 5) Write/read contention before Cache Fusion

} Cache Fusion – I aka Consistent Read Server Write/Read contention - CR Block Transfer in Cache Fusion Oracle Introduced a background process called BSP (Block Server process) makes the CR fabrication at the holder’s cache and ships the CR version of the block across the interconnect

} Still need to address Write/Write Contention Write / Write Contention before Cache Fusion – II (before 9 i)

} So now – Cache Fusion – II or Write/Write Cache Fusion current block transfer (from 9 i r 2 )

} Buffer States In Cache Fusion Mode/Role Local Global Null: N NL NG Shared: S SL SG Exclusive: X XL XG SL When an instance has a resource in SL form, it can serve a copy of the block to other instances and it can read the block from disk. Since the block is not modified, there is no need to write to disk. XL When an instance has a resource in XL form, it has sole ownership and interest in that resource. It also has the exclusive right to modify the block. All changes to the blocks are in its local buffer cache, and it can write the block to disk. If another instance wants the block, it will contact the instance via GCS. NL A NL form is used to protect consistent read blocks. If a block is held in SL mode and another instance wants it in X mode, the current instance will send the block to the requesting instance and downgrade its role to NL. SG In SG form, a block is present in one or more instances. An instance can read the block from disk and serve it to other instances. XG In XG form, a block can have one or more Pls, indicating multiple copies of the block in several instances’ buffer caches. The instance with the XG role has the latest copy of the block and is the most likely candidate to write the block to disk. GCS can ask the instance with the XG role to write the block to disk or to serve it to another instance. NG After discarding Pls when instructed by GCS, the block is kept in the buffer cache with NG role. This serves only as the CR copy of the block.

} Example 1: Reading a Block from Disk

} Example 2: Reading a Block from the Cache

} Example 3: Getting a (Cached) Clean Block for Update

} Example 4: Getting a (Cached) Modified Block for Update and Commit

} Example 5: Commit the Previously Modified Block and Select the Data

} Example 6: Write the Dirty Buffers to Disk Due to Checkpoint

} Example 7: Master Instance Crash

} Example 7: What Alert log says abt reconfiguration……. } } } } } } } } } List of nodes: 012 Global Resource Directory frozen * dead instance detected - domain 0 invalid = TRUE Communication channels reestablished * domain 0 valid = 0 according to instance 0 Wed Jun 21 23: 22 2006 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 21 23: 22 2006 LMS 0: 0 GCS shadows cancelled, 0 closed Wed Jun 21 23: 22 2006 LMS 2: 0 GCS shadows cancelled, 0 closed Wed Jun 21 23: 22 2006 LMS 3: 0 GCS shadows cancelled, 0 closed Wed Jun 21 23: 22 2006 LMS 1: 0 GCS shadows cancelled, 0 closed Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Wed Jun 21 23: 22 2006 LMS 0: 2189 GCS shadows traversed, 332 replayed Wed Jun 21 23: 22 2006 LMS 2: 2027 GCS shadows traversed, 364 replayed Wed Jun 21 23: 22 2006 LMS 3: 2098 GCS shadows traversed, 364 replayed Wed Jun 21 23: 22 2006 LMS 1: 2189 GCS shadows traversed, 343 replayed Wed Jun 21 23: 22 2006 Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete

} Crash Recovery – Key Components Ø Redo Threads and Streams Ø Redo Records and Change Vectors Ø Checkpoints ØThread Checkpoint or Local Checkpoint ØDatabase Checkpoint or Global Checkpoint ØIncremental Checkpoint ØBounded Recovery ØBlock Written Record (BWR) ØPast Image (PI) ØCheckpoints and PI Ø I Pass Recovery Ø II Pass Recovery Ø Merge Threads

Cache Fusion - Crash Instance Recovery } The steps for GRD reconfiguration are as follows: Instance death is detected by the cluster manager. Requests for PCM locks are frozen. Enqueues are reconfigured and made available. DLM recovery. GCS (PCM lock) is remastered. Pending writes and notifications are processed. The steps for I Pass recovery are as follows: The instance recovery (IR) lock is acquired by SMON. The recovery set is prepared and built. Memory space is allocated in the SMON Program Global Area (PGA). SMON acquires locks on buffers that need recovery. II Pass recovery steps are as follows: II Pass is initiated. The database is partially available. Blocks are made available as they are recovered. The IR lock is released by SMON. Recovery is complete. The system is available.

} Example 8: Select the Rows from Instance A

} Just for a clear understanding…… } Its time to play ……

} Cross Instance Consistent Read Instance 1 Instance 2 Session 15 LMS 0 SELECT runs, wickets FROM score WHERE team = 'ENG'; Build read consistent version of block 42 Session 27 UPDATE score SET runs = runs + 6 SET runs = runs + 4 SET runs = runs + 2 WHERE team = 'ENG'; segment 5 slot 18: state: 10 wrap#: 4 E 7 dba: 00800777 Undo Header ITL 1 seq: 530 irb 12 xid: 0005. 018. 4 E 7 uba: 800777. 530. 13 uba: 800777. 530. 12 uba: 800777. 530. 13 uba: 800777. 530. 14 slot 0 col 1: ENG col 2: 340 col 2: 350 col 2: 344 col 2: 340 col 2: 344 col 2: 352 col 2: 350 col 3: 1 col 2: 340 col 2: 344 col 2: 352 col 2: 350 12 uba: 5. 1 slot 1 col 1: AUS col 2: 99 col 3: 10 Data Block 42 (copy) Data Block 42 col 3: 340 13 uba 800777. 530. 12 col 3: 1 slot 1 block 42 slot 0 col 3: 344 14 uba 800777. 530. 13 5. 1 block 42 slot 0 col 3: 350 Undo Block 800777

} Commited Block – Block on Disk Session 15 LMS 0 Session 27 22: 9 22: 10 ENG 199 ENG 205 ENG 204 ENG 205 ENG 200 ENG 199 ENG 200 AUS 99 ENG 204 Block 42 Undo Block SELECT runs FROM score WHERE team = 'ENG'; ENG 199 ENG 205 AUS 99 Instance 1 Instance 2 UPDATE score SET runs = 200 WHERE team = 'ENG'; UPDATE score SET runs = 204 WHERE team = 'ENG'; UPDATE score SET runs = 205 WHERE team = 'ENG'; COMMIT;

Committed Block – Block on Buffer Cache } Session 15 LMS 0 Session 27 22: 9 22: 10 ENG 199 ENG 205 ENG 204 ENG 205 ENG 200 ENG 199 ENG 200 AUS 99 ENG 204 Block 42 Undo Block SELECT runs FROM score WHERE team = 'ENG'; ENG 199 AUS 99 Instance 1 STOP Instance 2 UPDATE score SET runs = 200 WHERE team = 'ENG'; UPDATE score SET runs = 204 WHERE team = 'ENG'; UPDATE score SET runs = 205 WHERE team = 'ENG'; COMMIT;

} Uncommitted Block – Block in Buffer cache Session 15 LMS 0 Session 27 22: 10 ENG 199 ENG 200 ENG 204 ENG 205 ENG 200 ENG 199 ENG 200 AUS 99 ENG 204 Block 42 Copy Block 42 Undo Block SELECT runs FROM score WHERE team = 'ENG'; ENG 199 AUS 99 Instance 1 Instance 2 UPDATE score SET runs = 200 WHERE team = 'ENG'; UPDATE score SET runs = 204 WHERE team = 'ENG'; UPDATE score SET runs = 205 WHERE team = 'ENG';

} Uncommitted Block – On Disk Session 15 LMS 0 Session 27 ENG 199 22: 10 ENG 199 ENG 205 ENG 204 ENG 200 ENG 199 ENG 200 ENG 204 ENG 205 ENG 200 ENG 199 ENG 200 AUS 99 ENG 204 Block 42 Undo Block SELECT runs FROM score WHERE team = 'ENG'; UPDATE score SET runs = 200 WHERE team = 'ENG'; UPDATE score SET runs = 204 WHERE team = 'ENG'; UPDATE score SET runs = 205 WHERE team = 'ENG'; ENG 200 ENG 205 ENG 204 ENG 199 SEE SLIDE NOTES FOR ADDITIONAL INFORMATION AUS 99 Instance 1 Instance 2

Q&A

} References: } Oracle 10 g Real Application Clusters handbook – K Gopalkrishnan } Julian Dyke – RAC Presentation } Oracle 10 g RAC Administrators Guide