Automatic Storage Management Julian Dyke Independent Consultant Web
Automatic Storage Management Julian Dyke Independent Consultant Web Version - December 2008 © 2008 Julian Dyke juliandyke. com
Objectives 2 1. Understand how Oracle database files are stored in ASM 2. Calculate how long ASM rebalance operations will take © 2008 Julian Dyke juliandyke. com
Agenda u u 3 © 2008 Julian Dyke ASM Instances ASM Disk Groups u Metadata u Extent Distribution u Rebalancing u Redundancy juliandyke. com
ASM Instances 4 © 2008 Julian Dyke juliandyke. com
ASM Single Instance Architecture OCSSD Daemon Only Oracle Clusterware ASM Instance RDBMS Instance Server Dedicated Storage 5 © 2008 Julian Dyke juliandyke. com
ASM Single Instance Background Processes Oracle 11. 1 DBW 0 LGWR VKTM PSP 0 Fixed Area CKPT MMAN RBAL GMON X 000 Variable Area SMON PMON 6 © 2008 Julian Dyke DIAG ASM Cache DIA 0 juliandyke. com
ASM RAC Architecture Public Network Private Network Oracle Clusterware ASM Instance RDBMS Instance Node 1 Node 2 Node 3 Node 4 Storage Network Shared Storage 7 © 2008 Julian Dyke juliandyke. com
ASM RAC Architecture CLUSTERWARE +ASM 1 PROD 1 TEST 1 Clusterware ASM Instances RDBMS Instances CLUSTERWARE +ASM 2 PROD 2 TEST 2 PROD TEST Database Files 8 © 2008 Julian Dyke juliandyke. com
ASM RAC Instance Background Processes Oracle 11. 1 LMON LMS 0 LMD 0 LCK 0 DIAG DBW 0 VKTM PSP 0 MMAN RBAL LGWR Fixed Area CKPT SMON X 000 Variable Area PMON MARK 9 © 2008 Julian Dyke GMON DIAG DIA 0 ASM Cache KATE juliandyke. com
ASM Disk Groups 10 © 2008 Julian Dyke juliandyke. com
ASM Disk Groups and Disks Disk Group 1 11 Disk Group 2 Disk Group 3 Disk 1 Disk 2 Disk 5 Disk 4 Disk 6 Disk 7 © 2008 Julian Dyke Disk 3 juliandyke. com
ASM Disk Groups and Disks Disk Group 1 Disk 1 12 Disk 4 © 2008 Julian Dyke Disk Group 2 Disk 5 Disk 6 Disk Group 3 Disk 7 Disk 3 juliandyke. com
ASM Disk Groups, Disks and Database Files Disk Group 1 Disk Group 2 Disk Group 3 File 2 File 3 File 1 File 5 File 4 File 6 File 5 Disk 1 Disk 4 Disk 2 13 Disk 3 © 2008 Julian Dyke Disk 5 Disk 6 Disk 7 juliandyke. com
File Extents versus Allocation Units 14 u File Extent u Logical unit of ASM file u Map to allocation units u One to many mapping u Allocation Unit u Physical unit of ASM disk u Oracle 10. 2 and below u Always 1 MB u Can be increased using _asm_ausize u Oracle 11. 1 and above u Variable size u 1 MB, 2 MB , 4 MB, 8 MB, 16 MB, 32 MB, 64 MB © 2008 Julian Dyke juliandyke. com
X$KFFXP u u u 15 Maps file extents to allocation units Only populated in ASM instance Columns include Column Name Description GROUP_KFFXP Disk Group Number NUMBER_KFFXP File Number COMPOUND_KFFXP Disk Group Number || File Number INCARN_KFFXP Incarnation Number PXN_KFFXP Physical Extent Number (within file) XNUM_KFFXP Logical Extent Number (within file) LXN_KFFXP 0=primary, 1=first mirror, 2=second mirror DISK_KFFXP Disk Number AU_KFFXP Allocation Unit Number (within disk) SIZE_KFFXP Size (# allocation units) © 2008 Julian Dyke juliandyke. com
ASM Metadata 16 u Metadata is stored in first 256 files in ASM disk group u Space is initially allocated when disk group is created u Can be subsequently extended u Metadata allocation units are divided into blocks u Each block is 4096 bytes u Block size specifed using _asm_blksize u Metadata files include © 2008 Julian Dyke File# Description 0 Metadata Header 1 File Directory 2 Disk Directory 3 Active Change Directory 4 Continuing Operations Directory 5 Template Directory 6 Alias Directory 9 Attribute directory (optional) 12 Staleness registry (optional) juliandyke. com
ASM Metadata Header Partner Status Table File Directory Disk Header Free Space Table Allocation Table Active Change Directory Continuing Operations Directory Template Directory Alias Directory 17 © 2008 Julian Dyke Metadata Header juliandyke. com
ASM Metadata u Initial Allocation (Single Instance) File# AU 0 u u 18 Description # AUs 0 Disk Header, Free Space Table, Allocation Table 1 1 Partner Status Table 1 1 File Directory 1 2 Disk Directory 1 3 Active Change Directory 4 Continuing Operations Directory 2 5 Template Directory 1 6 Alias Directory 1 42 Active Change Directory u Records changes to metadata u Used during recovery of instance or operation failures Continuing Operations Directory u Maintains state of active operations © 2008 Julian Dyke juliandyke. com
ASM Metadata Block Types 19 Type Description 1 KFBTYP_DISKHEAD 13 KFBTYP_PST_NONE 2 KFBTYP_FREESPC 14 KFBTYP_HASHNODE 3 KFBTYP_ALLOCTBL 15 KFBTYP_COD_RBO 4 KFBTYP_FILEDIR 16 KFBTYP_COD_DATA 5 KFBTYP_LISTHEAD 17 KFBTYP_PST_META 6 KFBTYP_DISKDIR 18 KFBTYP_PST_DTA 7 KFBTYP_ACDC 19 KFBTYP_HBEAT 8 KFBTYP_CHNGDIR 20 KFBTYP_SR 9 KFBTYP_CODBGO 21 KFBTYP_STALEDIR 10 KFBTYP_TMPLTDIR 22 KFBTYP_VOLUMEDIR 11 KFBTYP_ALIASDIR 23 KFBTYP_ATTRDIR 12 KFBTYP_INDIRECT © 2008 Julian Dyke juliandyke. com
KFED Utility u In Oracle 10. 2 and above the kfed utility can be used to inspect and edit the contents of ASM blocks [oracle@server 3 ~]$ $ORACLE_HOME/bin/kfed -h as/mlib ASM Library [asmlib='lib'] aun/um AU number to examine or update [AUNUM=number] aus/z Allocation Unit size in bytes [AUSZ=number] blkn/um Block number to examine or update [BLKNUM=number] blks/z Metadata block size in bytes [BLKSZ=number] ch/ksum Update checksum before each write [CHKSUM=YES/NO] cn/t Count of AUs to process [CNT=number] d/ev ASM device to examine or update [DEV=string] o/p KFED operation type [OP=READ/WRITE/MERGE/NEW/FORM/FIND/STRUCT] p/rovnm Name for provisioning purposes [PROVNM=string] s/eek AU number to seek to [SEEK=number] te/xt File name for translated block text [TEXT=string] ty/pe ASM metadata block type number [TYPE=number] u 20 This utility should only be used under the guidance of Oracle Support © 2008 Julian Dyke juliandyke. com
KFED Utility u For example to dump blocks in aliases directory in DISKGROUP 1 Find group number u SELECT group_number FROM v$asm_diskgroup WHERE name = 'DISKGROUP 1'; Alias directory is stored in file number 6 u SELECT disk_kffxp, au_kffxp FROM x$kffxp WHERE group_kffxp = 1 AND number_kffxp = 6 AND lxn_kffxp = 0; u Disk Allocation Unit 0 49 Find disk name SELECT path FROM v$asm_disk WHERE group_number = 1 AND disk_number = 0; Path /dev/oracleasm/disks/VOL 1 21 © 2008 Julian Dyke juliandyke. com
KFED Utility u Example (continued) u Allocation unit is 1 MB u Block size is 4096 u Therefore there are 256 blocks per allocation unit u Starting block offset = 256 * 49 = 12544 for (( f = 12544 ; f < 12544 + 256 ; f++ )) do kfed op=read blkn=$f dev='/dev/oracleasm/disks/VOL 1' > blk${f} done 22 © 2008 Julian Dyke juliandyke. com
Extent Distribution 23 © 2008 Julian Dyke juliandyke. com
Extent Distribution u Creating a disk group: CREATE DISKGROUP diskgroup 1 EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/VOL 1'; u Dropping a disk group: DROP DISKGROUP diskgroup 1 INCLUDING CONTENTS 24 © 2008 Julian Dyke juliandyke. com
Extent Distribution 1 disk Metadata 0 1 2 3 4 5 6 7 Disk 0 Metadata 25 © 2008 Julian Dyke Data juliandyke. com
Extent Distribution 2 disks Metadata 0 2 4 6 8 10 12 14 Disk 0 Metadata 26 © 2008 Julian Dyke 1 3 5 7 9 11 13 15 Disk 1 Data juliandyke. com
Extent Distribution 4 disks Metadata 0 4 8 12 16 20 24 28 Disk 0 2 6 10 14 18 22 26 30 1 5 9 13 17 21 25 29 3 7 11 15 19 23 27 31 Disk 2 Disk 3 Metadata 27 © 2008 Julian Dyke Data juliandyke. com
Extent Distribution 1 large disk - 1 small disk Metadata 2 5 8 11 0 1 3 4 6 7 9 10 Disk 0 Metadata 28 © 2008 Julian Dyke Disk 1 Data juliandyke. com
Extent Distribution 1 large disk - 3 small disks Metadata 2 7 12 17 1 6 11 16 4 9 14 19 Disk 1 Disk 2 Disk 4 0 3 5 8 10 13 15 18 Disk 0 Metadata 29 © 2008 Julian Dyke Data juliandyke. com
Extent Distribution 2 large disks - 2 small disks Metadata 0 2 6 8 12 14 18 20 Disk 0 1 4 7 10 13 16 19 22 3 9 15 21 5 11 17 23 Disk 1 Disk 2 Disk 4 Metadata 30 © 2008 Julian Dyke Data juliandyke. com
Rebalancing 31 © 2008 Julian Dyke juliandyke. com
Extent Distribution u Adding a disk: ALTER DISKGROUP diskgroup 1 ADD DISK '/dev/oracleasm/disks/VOL 2' REBALANCE POWER 0; u Dropping a disk: ALTER DISKGROUP diskgroup 1 DROP DISK 'DISKGROUP 1_0002' REBALANCE POWER 0; u Rebalancing a disk group: ALTER DISKGROUP diskgroup 1 REBALANCE POWER 1; 32 © 2008 Julian Dyke juliandyke. com
Rebalancing Adding disks - 1 disks to 2 disks 0 1 2 3 4 5 6 7 1 3 5 7 0 1 2 3 4 5 6 7 Disk 0 Metadata 33 STOP © 2008 Julian Dyke Disk 1 Data juliandyke. com
Rebalancing Adding disks - 1 disks to 4 disks 0 1 2 3 4 5 6 7 2 6 1 5 0 4 Disk 1 Disk 2 Disk 3 0 1 2 3 4 5 6 7 Disk 0 Metadata 34 STOP © 2008 Julian Dyke Data juliandyke. com
Rebalancing Adding disks - 2 disks to 3 disks 0 1 2 3 4 5 1 3 5 7 9 11 13 15 17 0 12 4 6 48 10 12 14 16 0 3 2 5 8 11 14 17 Disk 0 Disk 1 Disk 2 Metadata 35 © 2008 Julian Dyke Data juliandyke. com
Rebalancing Adding disks - 2 disks to 4 disks 0 1 2 3 4 5 6 7 20 2 64 6 8 10 12 14 1 5 9 13 0 4 8 12 Disk 1 Disk 2 Disk 3 1 3 5 7 9 11 13 15 Disk 0 Metadata 36 STOP © 2008 Julian Dyke Data juliandyke. com
Rebalancing Dropping disks - 3 disks to 1 disk 0 3 6 1 4 7 2 5 8 Disk 1 Disk 2 1 2 4 5 7 8 Disk 0 Metadata 37 STOP © 2008 Julian Dyke Data juliandyke. com
Rebalancing Moving disks - 2 disks to 2 disks 0 2 4 6 8 10 12 14 Disk 1 Disk 2 0 1 2 3 4 5 6 7 1 3 5 7 9 11 13 15 Disk 0 Metadata 38 STOP © 2008 Julian Dyke Data juliandyke. com
Rebalancing V$ASM_OPERATION u 39 Contains details of ongoing rebalance operations Column Name Data Type GROUP_NUMBER OPERATION CHAR(5) STATE VARCHAR 2(4) POWER NUMBER ACTUAL NUMBER SOFAR NUMBER EST_WORK NUMBER EST_RATE NUMBER EST_MINUTE NUMBER ERROR_CODE VARCHAR 2(44) © 2008 Julian Dyke Estimate of remaining time juliandyke. com
Rebalancing Power Limit 40 u Power limit can be 0 to 11 u 0 disables rebalance operation u 1 to 11 specifies number of ARBn background processes used for rebalance u In Oracle 10. 2 u RBAL manages rebalance operation u Each ARBn background process is allocated a range of 128 allocation units to rebalance u When complete another range is requested u AD lock is taken while an allocation unit is being rebalanced u Rebalance operations take much longer than theoretically necessary. Possible reasons include: u Locking u GES updates with other ASM instances u Updates to RDBMS instance © 2008 Julian Dyke juliandyke. com
Rebalancing Summary 41 u EST_MINUTES column of V$ASM_OPERATION is reasonably accurate u Allow a few minutes for SAN cache to stabilize u Check regularly for changes to estimate u ASM rebalance operations do not affect workload u Locks are only taken briefly u Lock mechanism has changed in Oracle 11. 1 u SAN cache and I/O performance will be affected u In Oracle 10. 2 rebalancing is fastest if u Other ASM instances are shutdown u RDBMS instance is shutdown u Estimated completion time will be affected by: u Use of SAN cache and I/O by rest of workload u Rate of change by applications to blocks in ASM files being rebalanced © 2008 Julian Dyke juliandyke. com
Redundancy 42 © 2008 Julian Dyke juliandyke. com
Redundancy u 43 ASM Supports three levels of redundancy u External Redundancy u Implemented externally using storage layer u Most common configuration in production u Normal Redundancy u Two copies of each extent maintained in separate failure groups u Used with extended clusters u Used occasionally in production e. g. CERN u Increases CPU overhead on servers u High Redundancy u Three copies of each extent maintained in separate failure groups u Very rare in production © 2008 Julian Dyke juliandyke. com
ASM Failure Groups - External Redundancy Disk Group Disk 1 44 © 2008 Julian Dyke Disk 2 Disk 3 juliandyke. com
ASM Failure Groups - Normal Redundancy Disk Group Failure Group 1 Disk 1 45 Disk 2 © 2008 Julian Dyke Disk 3 Failure. Group 2 Disk 4 Disk 5 Disk 6 juliandyke. com
ASM Failure Groups - High Redundancy Disk Group Failure Group 1 Disk 1 46 Disk 2 © 2008 Julian Dyke Failure Group 2 Disk 1 Disk 2 Failure Group 3 Disk 1 Disk 2 juliandyke. com
Normal Redundancy 1 Disk Per Failure Group 1 Failure Group 2 0 1 2 3 4 5 6 7 8 9 10 11 Disk 0 Disk 1 Metadata Data 47 © 2008 Julian Dyke Primary Secondary juliandyke. com
Normal Redundancy 2 Disks per Failure Group 2 Failure Group 1 0 3 4 7 8 11 12 15 16 19 20 23 1 2 5 6 9 10 13 14 17 18 21 22 Disk 0 Disk 1 Disk 2 Disk 3 Metadata Data 48 © 2008 Julian Dyke Primary Secondary juliandyke. com
High Redundancy 1 Disk per Failure Group 1 Failure Group 3 5 0 1 2 3 4 5 6 7 8 9 4 5 0 1 2 3 4 5 6 7 8 9 Disk 0 Disk 1 Disk 2 0 1 2 3 4 Metadata Data 49 Failure Group 2 © 2008 Julian Dyke Primary Secondary Tertiary juliandyke. com
References 50 u Oracle Automatic Storage Management (Oracle Press) u Nitin Vengurlekar u Murali Vallath u Rich Long u What ASM and ZFS Can Do For You u Jason Arneil - Nominet u A Closer Look Inside Oracle ASM u Luca Canali - CERN u Implementing ASM Without HW Raid u Luca Canali - CERN © 2008 Julian Dyke juliandyke. com
Thank you for listening info@juliandyke. com 51 © 2008 Julian Dyke juliandyke. com
- Slides: 51