Host Based Volume Shadowing Update Jeff Friedrichs Volume
Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff. Friedrichs@hp. com
Agenda Review of Recent Features • New Features to look forward to • May 2004
HBMM Host Based Mini Merge Available on V 7. 3 -2 (with TIMA kits) May 2004
What is Mini Merge? • Full merge requires comparing entire shadow set • But only blocks with I/O in progress need to be merged • MSCP Based mini merge – – – • Supported on HSCxx / HSDxx / HSJxx controllers Controller tracks in-progress writes Host can get list of writes from the controller Host Based Mini Merge (HBMM) Selected cluster hosts track recent writes using write bitmap – Bitmap is periodically reset to flush out old writes – Contents of bitmap drive mini merge operation – May 2004
Write Bitmap for Mini Merge • Write bitmap originally released on V 7. 2 -2 for mini copy – Each system has an in-memory bitmap to track write I/O – Each bitmap has one system that is the master – 2 KB memory per Gbyte of storage per bitmap per system • Originally 6 bitmaps, HBMM extended to 12 – There are 6 bitmaps are available (per VU) for mini merge use – There also 6 bitmap slots reserved for mini copy use • HBMM recovery must be by a bitmap master system • Use multiple bitmap masters for availability of the bitmap after a system crash May 2004
MINI COPY Bitmaps • Created with $DISMOUNT/POLICY=MINICOPY • NO redundancy • Uses bitmap first 6 bitmap slots May 2004
HBMM Bitmap Master Policy • The policy defines number of bitmap masters for a VU in the cluster – the location of masters (in counted groups) – the bitmap reset threshold – • Named policies Are known cluster wide – Any named policy can be assigned to an individual VU or to multiple VUs – A named policy can be deleted at any time – • That does not affect VUs that had it applied May 2004
Bitmap Master Policy • A policy must be directly assigned to VU for HBMM to be enabled on that VU • With the following exception If a DEFAULT policy has been defined, then every VU in the cluster, that does not have a named policy, will automatically “pick up” the DEFAULT policy – To prevent a VU from “picking up” the DEFAULT policy use – • SET SHADOW DSA 1: / POLICY = HBMM = NODEFAULT • In summary – May 2004 the DEFAULT policy and a mix of VU specific policies can be used in the cluster at the same time
Policy Definition SET SHADOW /POLICY = HBMM = ( (MASTER_LIST = (NODE 1, NODE 2, NODE 3), COUNT=2), (MASTER_LIST = (NODE 4, NODE 5, NODE 6), COUNT=2), RESET_THRESHOLD=50000 ) / NAME = POLICY_1 SET SHADOW DSA 1: / POLICY = HBMM = POLICY_1 May 2004
Policy Definition continued SET SHADOW DSA 1: /POLICY = HBMM = ( (MASTER_LIST = (NODE 1, NODE 2, NODE 3), COUNT=2), (MASTER_LIST = (NODE 4, NODE 5, NODE 6), COUNT=2), RESET_THRESHOLD=50000 ) May 2004
Other HBMM Controls SET SHADOW DSA 1: / DISABLE = HBMM – Disables HBMM on DSA 1 SET SHADOW DSA 1: / ENABLE = HBMM Enables HBMM on DSA 1 – If there is a policy for the VU to enable – SET SHADOW DSA 1: / POLICY=HBMM=(MASTER=*) – – May 2004 Allows all systems in the cluster to be bitmap masters First six to mount DSA 1: will become masters – Others eligible when a master dismounts VU or crashes
Other HBMM Controls SET SHADOW / POLICY = HBMM = (MASTER_LIST = …. ) / NAME = DEFAULT – Creates default policy for all shadow sets that are mounted in the cluster that do not have a VU specific policy in place SET SHADOW DSA 1: / POLICY = HBMM = NODEFAULT Disables HBMM for DSA 1 – Can be used to override a DEFAULT policy on a specific VU – Can be viewed as a policy governor – SET SHADOW / POLICY = HBMM / NAME=POLICY_1 / DELETE – May 2004 Will delete the policy named “POLICY_1”
HBMM rules – If a policy is associated with a VU, HBMM is automatically enabled upon first mount on a system that has been named as bitmap master in that policy – If a master system ceases to be a master (due to dismount or crash), a new master bitmap will be automatically started on another master system, subject to the policy in force on the VU – Devices capable of MSCP Based (HSC/HSJ/HSD) mini merge are not eligible for HBMM – To enable host based mini merge on a VU, all systems that mount the VU must be HBMM capable May 2004
AMCVP Automatic Mini Copy on Volume Processing • aka Multiuse Bitmaps • Available in V 8. 3 • Extension of HBMM • May 2004
AMCVP Considerations • Multiuse bitmaps are valid for both mini MERGE and mini COPY operations • Multiuse bitmaps are NOT zeroed, therefore mini MERGEs will take longer and longer • Multiuse bitmaps must be deleted manually if the removed member is never coming back to the set May 2004
Policy Definition SET SHADOW /POLICY = HBMM = ( (MASTER_LIST = (NODE 1, NODE 2, NODE 3), COUNT=2, MULTIUSE=2), (MASTER_LIST = (NODE 4, NODE 5, NODE 6), COUNT=2, MULTIUSE=1), RESET_THRESHOLD=50000 ) / NAME = POLICY_1 SET SHADOW DSA 1: / POLICY = HBMM = POLICY_1 May 2004
Write Bitmap Message Modes • Single Message Mode – • Only one logical WBM write request per SCS message Buffered Message Mode Supports up to nine logical WBM write requests per SCS message – Message dispatching occurs – • when the message is full • when an internal WBM timer expires May 2004
Write Bitmap Message Mode Control SYSGEN Parameters for WBM • WBM_MSG_UPPER – • The upper threshold for the number of messages sent during the calibration interval that will initiate buffered message mode WBM_MSG_LOWER – The lower threshold for the number of messages sent during the calibration interval that will initiate single message mode V 8. 3 and V 8. 3 -1 H 1 – CHECK YOUR PARAMETERS! May 2004
Merge and Copy Prioritization May 2004
Current Merge and Copy Controls • Management controls to determine order and choice of system for copy and merge operations are incomplete. • Important volumes may be merged after less important volumes • Systems better suited to perform merge or copy operations on some volumes are not always selected May 2004
Merge and Copy Control • Allow user to assign a priority to every VU • Better predict which system will perform any transient state operations (merge or copy operations) – requires SYSGEN settings • Utilize SHADOW_MAX_COPY dynamic characteristic May 2004
Shadow Priority • New command qualifier – $ SET SHADOW /PRIORITY = n DSAnnnn: • A range of 0 through 10, 000 • Default is 5000 • 1 is the lowest priority • Zero has special meaning • At MOUNT time each VU will be placed in system wide priority linked list by this value – • VUs at the same priority have an undefined ordering Governs merge and copy priority for VUs on this system May 2004
Show Shadow Priority New command $ SHOW SHADOW /BY_PRIORITY • Lists the DSA devices on this system using the priority assigned to each, highest to lowest • Shows transient state % and system performing operation Device DSA 3233: DSA 2325: DSA 42: May 2004 Priority 3233 2325 42 Virtual Unit State Steady State Not Mounted Full Merge Active % Completed on Node 14% on CSGF 1
SYSGEN parameter SHADOW_REC_DLY (Shadow Recovery Delay) • This parameter governs how many seconds, after the VU enters a merge transient state, that this system will wait before it attempts to manage that state • Default of 20 seconds • Making this value different across the cluster will guide which system will manage transient state operations May 2004
Using Priority List Until all SHADOW_MAX_COPY threads are used on a system, the priority list is processed in two phases: Host Based Mini Merge or MSCP Based Mini Merge Then – • Full Copy state or • Full Merge state This implies that all HBMM VUs are processed before any full copy or full merge VUs … regardless of the priority value for any VU May 2004
Per SSM Served Delay Each SSM on an MSCP served path has an additional delay added before attempting to obtain the _COPIER lock to manage a merge operation. If the default value (30 seconds) is not suitable it can be changed • SET SHADOW/<tbs>=n DSAn: – • Will use this value for all SSMs on an MSCP served path SET SHADOW/<tbs>=n /ALL Will change the default value of 30 that HBVS will use for all VUs that are mounted on this system – Will change the value on all VUs currently mounted on this system – May 2004
Managing Transient State Operations New SET SHADOW command qualifiers: /EVALUATE=RESOURCES – Is a system specific command … i. e. not cluster specific • Thus it only affects the VUs that are being merged or copied by this system – Useful if the command is issued before • SHADOW_MAX_COPY is increased or is decreased – To start or stop a merge or a copy operation • The priority of a VU is changed – Priority of 0 now non-zero or the inverse – Priority value has been raised or lowered May 2004
Managing Transient State Operations New SET SHADOW command qualifier: /DEMAND_MERGE DSAn: Changes the state of the VU to a merge required state • The type of merge initiated will depend on merge recovery characteristics that are enabled currently on the VU • To insure that a full merge is initiated on a VU • Disable host based mini merge that it is currently enabled • This is especially useful if the shadow set had been created with INIT/ SHADOW without / ERASE • Can be used if differences are found on the members with ANALYZE /DISK /SHADOW May 2004
The Future of Shadowing V 8. 4 Enhancements May 2004
HBMM Dismount • Further extension of HBMM and AMCVP • Allows all 12 bitmap slots to be used by HBMM or Mini. Copy • Provides bitmap redundancy for removed members May 2004
Policy Definition SET SHADOW /POLICY = HBMM = ( (MASTER_LIST = (NODE 1, NODE 2, NODE 3), COUNT=2, MULTIUSE=2, DISMOUNT=2), (MASTER_LIST = (NODE 4, NODE 5, NODE 6), COUNT=3, MULTIUSE=2, DISMOUNT=2), RESET_THRESHOLD=50000 ) / NAME = POLICY_1 May 2004
Fast MINI operations Performance enhancement for Mini Copy and Mini Merge Look ahead in the bitmap No user intervention required Significantly reduces I/O from SHADOW_SERVER May 2004
Write Bitmap Enhancements • Fix buffered mode transitions • Multicast “set bit” messages • Optimize sequential writes May 2004
SPLIT_READ_LBNS • Attempt to utilize read ahead controller cache • Currently reads are “round robin” among members • Simply divide the virtual unit into smaller chunks based on LBN and send all reads for the same area to the same member • Only members with the lowest and same READ_COST are used May 2004
SPLIT_READ_LBNS $ show shad dsa 33/full Device $1$DGA 33 Index 0 Status 000000 A 0 mbr_src, mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81 DCA 540 Error Count 0 LBN_LIMIT: 0000 Device $1$DGA 41 Master Member Index 1 Status 000000 A 0 mbr_src, mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81 DB 8 B 80 Error Count 0 LBN_LIMIT: 0000 Device $1$DGA 34 Index 2 Status 200000 A 0 mbr_src, mbr_valid, user_supplied_read_bia Ext. Member Status 00 Read Cost 3 Site 0 Member Timeout 120 UCB 81 DD 5 E 40 Error Count 0 LBN_LIMIT: 0000 May 2004
SPLIT_READ_LBNS $ SET SHADOW/ENABLE=SPLIT_READ_LBNS $ SHOW SHADOW/FULL DSA 33 Device $1$DGA 33 Index 0 Status 000000 A 0 mbr_src, mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81 DCA 540 Error Count 0 LBN_LIMIT: 000 F 4240 Device $1$DGA 41 Master Member Index 1 Status 000000 A 0 mbr_src, mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81 DB 8 B 80 Error Count 0 LBN_LIMIT: 001 E 8480 Device $1$DGA 34 Index 2 Status 200000 A 0 mbr_src, mbr_valid, user_supplied_read_bia Ext. Member Status 00 Read Cost 3 Site 0 Member Timeout 120 UCB 81 DD 5 E 40 Error Count 0 LBN_LIMIT: 0000 May 2004
SPLIT_READ_LBNS $ SET SHADOW – _$ /DISABLE=SPLIT_READ_LBN DSA 1: May 2004
One More! May 2004
XMBRS $ show dev DSA 6 Device Name DSA 16: $1$DGA 1: $1$DGA 2: $1$DGA 3: $1$DGA 4: $1$DGA 5: $1$DGA 6: May 2004 Device Status Mounted (CSGF 2) (CSGF 2) Error Volume Count Label 0 XMBRS Shadow. Set. Member 0 Free Trans Mnt Blocks Count Cnt 49954 1 1 (member of DSA 6: ) (member of DSA 6: )
XMBRS • To turn it on, just specify more than 3 members for the shadow set $ MOUNT/SHADOW DSA 16 – _$ /SHADOW=($1$dga 1, $1$dga 2, $1$dga 3, $1$dga 4) – _$ DSA 6 No new qualifiers • No changes to DISMOUNT • May 2004
XMBRS Compatibility • Mixed version, 3 member shadow sets will continue to work • To use 4 or more members, all systems that have the VU MOUNTed, must have the new software • MOUNT/INCLUDE of a once XMBR virtual unit on an older version may not find all of the members May 2004
XMBRS Performance testing in progress • Reads could be faster • Writes *WILL* be slower • • Important to use SET SHADOW commands – May 2004 Noticeable difference with /COPY_SOURCE
Questions? ? May 2004
- Slides: 43