Module 7 VPLEX Troubleshooting Upon completion of this

Module 7: VPLEX Troubleshooting Upon completion of this module, you should be able to: • Define data collection processes • Perform basic troubleshooting on various VPLEX elements • Collect and analyze data to resolve connectivity errors Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 1

Module 7: VPLEX Troubleshooting Lesson 1: Data Collection Upon completion of this lesson, you should be able to: • Use logs and debugging tools to troubleshoot VPLEX • Capture and analyze VPLEX log and configuration files Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 2

VPLEX Logs, Cores, and Configuration Data VPlexcli: /> collect-diagnostics 2012 -09 -09 08: 23: 32 UTC: ****Initializing collect-diagnostics. . . 2012 -09 -09 08: 23: 34 UTC: No cluster-witness server found. 2012 -09 -09 08: 23: 34 UTC: Free space = 43. 32 GB 2012 -09 -09 08: 23: 34 UTC: Total space needed = 5. 43 GB 2012 -09 -09 08: 23: 34 UTC: ****Starting collect-diagnostics, this operation might take a while. . . 2012 -09 -09 08: 23: 36 UTC: ****Starting Remote Management server sms dump log collection, this operation might take a while. . . Initiating sms dump. . . No such file or directory: /var/log/allmessages No such file or directory: /var/log/rsyncd. log No such file or directory: /var/log/syslog Warning: Could not access source directory /opt/emc/VPlex/Event_Msg_Folder No such file or directory: /etc/snmp/Vplex. Snmp. Perf. Config. xml 2012 -09 -09 08: 24: 10 UTC: ****Executing debug. Tower. Dump on directors. . . 2012 -09 -09 08: 24: 16 UTC: ****Completed debug. Tower. Dump on directors. 2012 -09 -09 08: 24: 16 UTC: ****Executing appdump on directors. . . 2012 -09 -09 08: 24: 26 UTC: ****Completed appdump on directors. 2012 -09 -09 08: 24: 26 UTC: ****Executing connectivity director on directors. . . 2012 -09 -09 08: 24: 27 UTC: ****Completed connectivity director on directors. 2012 -09 -09 08: 24: 27 UTC: ****Executing getsysinfo on directors. . . 2012 -09 -09 08: 24: 32 UTC: ****Completed getsysinfo on directors. 2012 -09 -09 08: 24: 33 UTC: ****Executing command export storage-view show-powerpath-interfaces -verbose on clusters. . . 2012 -09 -09 08: 24: 33 UTC: ****Completed command export storage-view show-powerpath-interfaces -verbose on clusters. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 3

collect-diagnostics • Collects VPLEX log files • Places logs in: /diag/collect-diagnostics-out 4 Extended Log: <tla>-diagnostics-extended-<time-stamp>. tar. gz 4 Base Log: <tla>-diagnostics-<time-stamp>. tar. gz • Collects all VPLEX logs including logs from the VPLEX Witness VPlexcli: /> collect-diagnostics 2012 -07 -20 17: 14: 55 UTC: ****Initializing collect-diagnostics. . . 2012 -07 -20 17: 14: 55 UTC: No cluster-witness server found. 2012 -07 -20 17: 14: 55 UTC: Free space = 46079912 K 2012 -07 -20 17: 14: 55 UTC: Total space needed = 4001362 K 2012 -07 -20 17: 14: 55 UTC: ****Starting collect-diagnostics, this operation might take a while. . . Thread-dump written to /diag/collect-diagnostics-tmp/debug-thread-output. txt Initiating sms dump. . . No such file or directory: /var/log/allmessages No such file or directory: /var/log/rsyncd. log No such file or directory: /var/log/syslog No such file or directory: /home/service/. b Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 4

VPLEX Configuration Data VPlexcli: /> cluster configdump -c cluster-1 -f /var/log/VPlex/cli/configdump_9_15_12. xml Initiating cluster configdump. . . VPlexcli: /> Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 5

Diagnostic TAR Location [email protected] 07 -VPLEX-Mgmt: ~> cd /diag/collect-diagnostics-out/ [email protected] 07 -VPLEX-Mgmt: /diag/collect-diagnostics-out> ll total 1064168 -rw-r--r-- 1 service users 128767754 2012 -09 -15 13: 15 FNM 00121800313 -diagnostics-2012 -09 -15 -13. 03. 13. tar. gz -rw-r--r-- 1 service users 959863439 2012 -09 -15 13: 14 FNM 00121800313 -diagnostics-extended-2012 -09 -1513. 03. 13. tar. gz Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 6

collect-diagnostics. log 2012 -02 -23 03: 27: 54 UTC: ****Executing cluster status. . . {'MALVPLEX 02': 'operational-status': ok, 'health-state': ok, 'transition-indications': [], 'transition-progress': [], 'health-indications': []}, 'MALVPLEX 01': 'operational-status': ok, 'health-state': ok, 'transition-indications': [], 'transition-progress': [], 'health-indications': []} }2012 -02 -23 03: 27: 54 UTC: ****Completed cluster status. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 7

collect-diagnostics. log 2012 -02 -23 03: 27: 54 UTC: ****Executing cluster summary. . . Clusters: Name Cluster ID Connected Expelled Operational Status ---------- ------------MALVPLEX 01 1 true false ok MALVPLEX 02 2 true false ok Health State ------ok ok Islands: Island ID Clusters ---------------MALVPLEX 01, MALVPLEX 02 2012 -02 -23 03: 27: 54 UTC: ****Completed cluster summary. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 8

collect-diagnostics. log /engines/engine-1 -1/directors/director-1 -1 -A/hardware/ports: Name Address Role Port Status -------------A 0 -FC 00 0 x 50001442606 af 700 front-end up A 0 -FC 01 0 x 50001442606 af 701 front-end up A 0 -FC 02 0 x 50001442606 af 702 front-end up A 0 -FC 03 0 x 50001442606 af 703 front-end up A 1 -FC 00 0 x 50001442606 af 710 back-end up A 1 -FC 01 0 x 50001442606 af 711 back-end up A 1 -FC 02 0 x 50001442606 af 712 back-end up A 1 -FC 03 0 x 50001442606 af 713 back-end up A 2 -FC 00 0 x 50001442606 af 720 wan-com up A 2 -FC 01 0 x 50001442606 af 721 wan-com up A 2 -FC 02 0 x 50001442606 af 722 wan-com no-link A 2 -FC 03 0 x 50001442606 af 723 wan-com no-link A 3 -FC 00 0 x 50001442606 af 730 local-com up A 3 -FC 01 0 x 50001442606 af 731 local-com up A 3 -FC 02 0 x 00000000 down A 3 -FC 03 0 x 00000000 down Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 9

Health-Check Overview VPlexcli: /> health-check Product Version: 5. 1. 0. 02. 00. 01 Clusters: ----Cluster Name ----cluster-1 cluster-2 Cluster ID ------1 2 Meta Data: -----Cluster Name ----cluster-1 cluster-1 Volume Name ----------------Pod 7 Meta_backup_2012 Aug 29_233000 logging_Pod 09 Logging_vol Pod 09 Meta Pod 7 Meta_backup_2012 Aug 30_233001 Pod 09 Meta_backup_2012 Sep 07_235001 Pod 09 Meta_backup_2012 Sep 08_235000 Oper State ----ok ok Health State -----ok ok Copyright © 2012 EMC Corporation. All Rights Reserved. Connected Expelled ----True -------False Volume Type -------meta-volume logging-volume meta-volume Oper State ----ok ok ok Health State -----ok ok ok Active -----False True False Module 7 - VPLEX Troubleshooting 10

Validate the VPLEX Back-end VPlexcli: /> connectivity validate-be Summary Cluster cluster-1 0 storage-volumes which are dead or unreachable. 0 storage-volumes which do not meet the high availability requirement for storage volume paths*. 0 storage-volumes which are not visible from all directors. 0 storage-volumes which have more than supported (4) active paths from same director. *To meet the high availability requirement for storage volume paths each storage volume must be accessible from each of the directors through 2 or more VPlex backend ports, and 2 or more Array target ports, and there should be 2 or more ITLs. Cluster cluster-2 0 storage-volumes which are dead or unreachable. 0 storage-volumes which do not meet the high availability requirement for storage volume paths*. 0 storage-volumes which are not visible from all directors. 0 storage-volumes which have more than supported (4) active paths from same Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 11

Validate the VPLEX System VPlexcli: /> validate-system-configuration Validate cache replication Checking cluster-1. . . rmg component not found skipping the validation of cache replication. ok Validate logging volume No errors found ok Validate back-end connectivity Cluster cluster-1 0 storage-volumes which are dead or unreachable. 0 storage-volumes which do not meet the high availability requirement for storage volume paths*. 0 storage-volumes which are not visible from all directors. 0 storage-volumes which have more than supported (4) active paths from same director. *To meet the high availability requirement for storage volume paths each storage Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 12

Module 7: VPLEX Troubleshooting Lesson 1: Summary During this lesson the following topics were covered: • How to use logs and debugging tools to troubleshoot VPLEX • How to capture and analyze VPLEX log and configuration files Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 13

Module 7: VPLEX Troubleshooting Lesson 2: VPLEX System Volumes Upon completion of this lesson, you should be able to: • Analyze and correct VPLEX system volume failures • List the components and steps to restore a Meta volume Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 14

Recovery After a Metadata Volume Failure VPlexcli: /clusters/cluster-1/storage-elements/storage-volumes> meta-volume backup VPD 83 T 3: 6006016036502200 ee 5 c 628 f 984 ae 011 VPlexcli: /clusters/cluster-1/storage-elements/storage-volumes> ll /**/system-volumes /clusters/cluster-1/system-volumes: Name Volume Type Operational Health Active Ready Geometry Block Capacity ----------------- Status State ------ Count Size -------------------- ------ -------- -------cluster-1 -log_vol logging-volume ok ok raid-1 2620800 4 K 10 G pod 3_meta-volume ok ok true raid-1 23592704 4 K 90 G pod 3_meta_backup_2011 Mar 10_160622 meta-volume ok ok false true raid-1 23592704 4 K 90 G Slots ----32000 VPlexcli: /clusters/cluster-1/storage-elements/storage-volumes> meta-volume move -t pod 3_meta_backup_2011 Mar 10_160622 VPlexcli: /clusters/cluster-1/storage-elements/storage-volumes> ll /**/system-volumes /clusters/cluster-1/system-volumes: Name Volume Type Operational Health Active Ready Geometry Block Capacity ----------------- Status State ------ Count Size -------------------- ------ -------- -------cluster-1 -log_vol logging-volume ok ok raid-1 2620800 4 K 10 G pod 3_meta-volume ok ok false true raid-1 23592704 4 K 90 G pod 3_meta_backup_2011 Mar 10_160622 meta-volume ok ok true raid-1 23592704 4 K 90 G Slots ----32000 VPlexcli: /clusters/cluster-1/storage-elements/storage-volumes> meta-volume destroy pod 3_meta Meta-volume 'pod 3_meta' will be destroyed. Do you wish to continue? (Yes/No) y VPlexcli: /clusters/cluster-1/storage-elements/storage-volumes> ll /**/system-volumes /clusters/cluster-1/system-volumes: Name Volume Type Operational Health Active Ready ----------------- Status State ---------------------- ------ ----cluster-1 -log_vol logging-volume ok ok pod 3_meta_backup_2011 Mar 10_160622 meta-volume ok ok true Copyright © 2012 EMC Corporation. All Rights Reserved. Geometry -------raid-1 Block Count -------2620800 23592704 Block Size ----4 K 4 K Capacity -------10 G 90 G Slots ----32000 Module 7 - VPLEX Troubleshooting 15

Recovery After a Metadata Volume Failure • Coinciding with a system failure, the metadata volume fails and cannot be recovered 4 Restart the cluster with a backup metavolume 4 Activate a backup copy of the metadata volume you created prior to the failure 4 meta-volume activate <target volume> VPlexcli: /clusters/cluster-1/storage-elements/storage-volumes> meta-volume activate -t pod 3_meta_backup_2011 Mar 10_160622 –f The system metadata will be restored with metadata backup in the target meta-volume. All metadata collected after the backup will be discarded. Continue? (Yes/No) Yes Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 16

Replace an Array: List the Metadata Volume Components Mirror Metadata Volume New Array Old Array VPlexcli: /clusters/cluster-1/system-volumes/c 1_meta/components> ll Name Slot Type Operational --------------------- Number -------- Status --------------------- -----VPD 83 T 3: 6006016010 f 01 e 006 af 7433 c 574 de 1111 1 storage-volume ok VPD 83 T 3: 6006048000018790016263694 d 3031333 1 storage-volume ok Copyright © 2012 EMC Corporation. All Rights Reserved. Health State ------ok ok Capacity -------78 G Module 7 - VPLEX Troubleshooting 17

Replace an Array: Attach a New Metadata Volume Mirror Metadata Volume Old Array Mirror New Array VPlexcli: /> meta-volume attach-mirror –v c 1_meta –d VPD 83 T 3: 6006016010 f 01 e 00 ee 250253574 de 111 VPlexcli: /clusters/cluster-1/system-volumes/c 1_meta/components> rebuild status [1] storage_volumes marked for rebuild Global rebuilds: No active global rebuilds. cluster-1 local rebuilds device rebuild type ---------c 1_meta full rebuilder director ---------s 1_48 d 6_spa Copyright © 2012 EMC Corporation. All Rights Reserved. rebuilt/total ------5. 47 G/78 G percent finished --------7. 01% throughput -----22 M/s ETA ------56. 4 min Module 7 - VPLEX Troubleshooting 18

Replace an Array: Rebuild the New Metadata Volume Mirror VPlexcli: /clusters/cluster-1/system-volumes/c 1_meta/components> ll Name Slot ------------------ Number ------------------ Type Operational Health ------- Status State ------- ------ VPD 83 T 3: 6006016010 f 01 e 006 af 7433 c 574 de 111 1 VPD 83 T 3: 6006016010 f 01 e 00 ee 250253574 de 111 2 VPD 83 T 3: 6006048000018790015253594 d 303133 0 storage-volume ok error ok ok critical-failure ok VPlexcli: /clusters/cluster-1/system-volumes/c 1_meta/components> ll Name Slot ------------------ Number ------------------ Type Operational Health ------- Status State ------- ------ VPD 83 T 3: 6006016010 f 01 e 006 af 7433 c 574 de 111 1 VPD 83 T 3: 6006016010 f 01 e 00 ee 250253574 de 111 2 VPD 83 T 3: 6006048000018790015253594 d 303133 0 storage-volume Copyright © 2012 EMC Corporation. All Rights Reserved. ok ok ok Module 7 - VPLEX Troubleshooting 19

Replace an Array: Remove the Old Metadata Volume Mirror Metadata Volume Mirror New Array Old Array VPlexcli: /clusters/cluster-1/system-volumes/c 1_meta/components> ll Name Slot ------------------ Number ------------------ Type Operational ------- Status ------- VPD 83 T 3: 6006016010 f 01 e 006 af 7433 c 574 de 111 0 VPD 83 T 3: 6006016010 f 01 e 00 ee 250253574 de 111 1 storage-volume Copyright © 2012 EMC Corporation. All Rights Reserved. ok ok Health Capacity State ------ok ok 78 G Module 7 - VPLEX Troubleshooting 20

Replace an Array: Migrate the Logging Volumes Mirror New Logging Volume Mirror VPlexcli: /clusters/cluster-1/system-volumes> ds dd set-log –d dd –l New-log_vol logging-volume destroy log-1_vol Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 21

Replace an Array: Destroy a Logging Volume Vplexcli: /> logging-volume destroy New-log_vol/ logging-volume destroy: Evaluation of <<logging-volume destroy Newlog_vol/>> failed. cause: Failed to destroy logging-volume ‘New-log_vol’. cause: Unable to destroy logging-volume ‘New-log_vol’. cause: Firmware command error cause: One or more controllers has the virtual volume of the device at Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 22

Module 7: VPLEX Troubleshooting Lesson 2: Summary During this lesson the following topics were covered: • How to analyze and correct VPLEX system volume failures • How to list the components and steps to restore a Meta volume Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 23

Module 7: VPLEX Troubleshooting Lesson 3: VPLEX Connectivity Upon completion of this lesson, you should be able to: • List the connectivity components in VPLEX • Define the logs and commands used to troubleshoot VPLEX • connectivity Perform basic troubleshooting operations to determine and correct root cause analysis Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 24

Latency Between Directors: director ping VPN Connection VPlexcli: /> director ping —n director— 1— 1—A —i 128. 221. 252. 35 Round—trip time to 128. 221. 252. 35: O. 671 ms VPlexcli: /> director ping —n director— 1— 1—A —i 128. 221. 253. 36 Round—trip time to 128. 221. 253. 36: O. 114 ms VPlexcli: /> director ping —n director— 1— 1—A —i 128. 221. 252. 67 Round—trip time to 128. 221. 252. 67: 0. 84 ms VPlexcli: /> director ping —n director— 1— 1—B —i 128. 221. 253. 68 Round—trip time to 128. 221. 253. 68: 0. 596 ms Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 25

VPLEX Local IP Layout Management Server Management Port for B-side network: Service Port: Management Port for A-side network: Public LAN port: 128. 221. 253. 33 128. 221. 252. 2 128. 221. 252. 33 Customer Assigned IP Address FC Switch 1 IP Address: 128. 221. 252. 34 MS FC 1 Engine 4, Cluster 1 Director A IP Addresses: 128. 221. 252. 41, 128. 221. 253. 41 Director B IP Addresses: 128. 221. 252. 42, 128. 221. 253. 42 Engine 3, Cluster 1 Director A IP Addresses: 128. 221. 252. 39, 128. 221. 253. 39 Director B IP Addresses: 128. 221. 252. 40, 128. 221. 253. 40 Engine 2, Cluster 1 Director A IP Addresses: 128. 221. 252. 37, 128. 221. 253. 37 Director B IP Addresses: 128. 221. 252. 38, 128. 221. 253. 38 Engine 3 Engine 2 Engine 1, Cluster 1 Director A IP Addresses: 128. 221. 252. 35, 128. 221. 253. 35 Director B IP Addresses: 128. 221. 252. 36, 128. 221. 253. 36 FC Switch 2 IP Address: 128. 221. 253. 34 Engine 1 FC 2 Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 26

Test the IP Connectivity [email protected] 07 -VPLEX-Mgmt: ~> ping -c 3 -I eth 0 128. 221. 252. 67 PING 128. 221. 252. 67 (128. 221. 252. 67) from 128. 221. 252. 33 eth 0: 56(84) bytes of data. 64 bytes from 128. 221. 252. 67: icmp_seq=1 ttl=63 time=0. 563 ms 64 bytes from 128. 221. 252. 67: icmp_seq=2 ttl=63 time=0. 397 ms 64 bytes from 128. 221. 252. 67: icmp_seq=3 ttl=63 time=0. 550 ms --- 128. 221. 252. 67 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2003 ms rtt min/avg/max/mdev = 0. 397/0. 503/0. 563/0. 077 ms If pings do not work, try a different protocol such as ssh C 1: ssh C 2: ssh [email protected]. 221. 252. 65 [email protected]. 221. 253. 65 ssh [email protected]. 221. 252. 33 [email protected]. 221. 253. 33 Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 27

Latency Between Directors: director tracepath • Provides route information from a director to a given target • machine Displays round-trip latency and MTU of each hop to the destination VPlexcli: /> director tracepath -n director-1 -1 -B -i 128. 221. 252. 67 Destination 128. 221. 252. 67 reachable. 3 hops: Source: endpoint 128. 221. 252. 36, latency 0. 044 ms, mtu 1500, reachable Hop 1: endpoint 128. 221. 252. 33, latency 0. 151 ms, mtu 1500, reachable Hop 2: endpoint 128. 221. 252. 33, latency 0. 14 ms, mtu 1428, reachable Hop 3: endpoint 128. 221. 252. 67, latency 1. 13 ms, mtu 1428, reachable Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 28

Identify Inter-Cluster Link Failures: cluster summary VPlexcli: /> cluster summary Clusters: Name Cluster ID Connected cluster— 1 1 true cluster— 2 2 true Islands: Island ID ----1 2 Operational Status Health State ok ok transitioning degraded Clusters ----cluster— 1 cluster— 2 VPlexcli: /> cluster summary Clusters: Name Cluster ID cluster— 1 1 cluster— 2 2 Islands: Island ID ----1 Expelled false Connected true Expelled false Operational Status Health State ok ok Clusters ----cluster— 1, cluster— 2 Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 29

Identify Inter-Cluster Link Failures: WAN COM Links VPlexcli: /> ll /**/ports /engines/engine-1 -1/directors/director-1 -1 -A/hardware/ports: Name Address Role Port Status -------------A 0 -FC 00 0 x 5000144240167700 front-end up A 0 -FC 01 0 x 5000144240167701 front-end up A 0 -FC 02 0 x 5000144240167702 front-end no-link A 0 -FC 03 0 x 5000144240167703 front-end no-link A 1 -FC 00 0 x 5000144240167710 back-end up A 1 -FC 01 0 x 5000144240167711 back-end up A 1 -FC 02 0 x 5000144240167712 back-end no-link A 1 -FC 03 0 x 5000144240167713 back-end no-link A 2 -XG 00 192. 168. 10. 35 wan-com up A 2 -XG 01 192. 168. 11. 35 wan-com no-link A 3 -FC 00 0 x 5000144240167730 local-com up A 3 -FC 01 0 x 5000144240167731 local-com up A 3 -FC 02 0 x 00000000 up A 3 -FC 03 0 x 00000000 up Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 30

Identify Inter-Cluster Link Failures: WAN COM Links VPlexcli: /clusters/cluster-1/consistency-groups/async_consist> ll Attributes: Name Value ------------------------------------------active-clusters [cluster-1] cache-mode asnychronous detach-rule winner cluster-1 after 5 s operational-status [(cluster-1, ( summary: : suspended, details: : [cluster-departure] )), (cluster-2, { summary: : suspended, details: : [cluster-departure] ))] passive-clusters storage-at-clusters virtual-volumes visibility [cluster-2] [cluster-1, cluster-2] [dd_vol] [cluster-1, cluster-2] Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 31

Identify VPN Issues: vpn status VPlexcli: /> vpn status Verifying the VPN status between the management servers. . . IPSEC is UP Remote Management Server at IP Address 10. 127. 58. 82 is reachable Remote Internal Gateway addresses are reachable Verifying the VPN status between the management server and the cluster witness server. . . Cluster Witness Server at IP Address 128. 221. 254. 3 is not reachable VPlexcli: /> vpn status Verifying the VPN status between the management servers. . . IPSEC is UP Remote Management Server at IP Address 10. 127. 58. 82 is reachable Remote Internal Gateway addresses are reachable Verifying the VPN status between the management server and the cluster witness server. . . IPSEC is UP Cluster Witness Server at IP Address 128. 221. 254. 3 is reachable Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 32

Check the VPN Uptime service@ Management-server-2: ~> sudo /usr/sbin/ipsec statusall 000 interface lo/lo : : 1: 500 000 interface lo/lo 127. 0. 0. 1: 500 000 interface eth 0/eth 0 128. 221. 253. 65: 500 000 interface eth 3/eth 3 10. 12. 178. 47: 500 000 interface eth 2/eth 2 128. 221. 252. 65: 500 000 interface eth 1/eth 1 128. 221. 252. 2: 500 000 %myid = (none) 000 debug none 000 Performance: uptime: 2 days, since Aug 02 16: 30: 38 2011 worker threads: 9 idle of 16, job queue load: 0, scheduled events: 6 loaded plugins: aes des sha 1 sha 2 md 5 fips-prf random x 509 pubkey xcbc hmac gmp kernel-netlink stroke updown Listening IP addresses: 128. 221. 253. 65 10. 12. 178. 47 128. 221. 252. 65 128. 221. 252. 2 Connections: net-net: 10. 12. 178. 47[C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: LEVEL 3 MEDIUM 02, [email protected]. com]. . . 10. 12. 178. 46[C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: LEVEL 3 MEDIUM 01, [email protected]. com] net-net: CAs: "C=US, ST=Massachusetts, L=Hopkinton, O=EMC, OU=EMC, CN=LEVEL 3 MEDIUM 01, [email protected]. com". . . %any net-net: public key authentication net-net: 128. 221. 252. 64/27 128. 221. 253. 64/27 === 128. 221. 252. 32/27 128. 221. 253. 32/27 Security Associations: net-net[22]: ESTABLISHED 26 minutes ago, 10. 12. 178. 47[C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: LEVEL 3 MEDIUM 02, [email protected]. com]. . . 10. 12. 178. 46[C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: LEVEL 3 MEDIUM 01, [email protected]. com] net-net[22]: IKE SPIs: 06 db 3 d 4 cd 388 d 131_i* 74 f 967 a 554 f 24965_r, public key reauthentication in 2 hours net-net[22]: IKE proposal: 3 DES/AUTH_HMAC_SHA 2_256_128/PRF_HMAC_SHA 2_256/MODP_2048_BIT net-net{22}: INSTALLED, TUNNEL, ESP SPIs: cd 94995 f_i cd 1 b 75 be_o net-net{22}: AES_CBC-256/AUTH_HMAC_SHA 2_256_128, rekeying in 17 minutes, last use: 0 s_i 0 s_o Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 33

Cache Vaulting Process Flow Vault Inactive Manual vault Cluster wide power failure Power restored Ride-out Power ride-out expired Quiesce IO Dirty cache pages frozen Write vault Vault written Stop director firmware Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 34

Vault Status VPlexcli: /> vault status -c cluster-2 --verbose ======================================== Cluster level vault status summary ======================================== Cluster: /clusters/cluster-2 Cluster is vaulting Total number of bytes remaining to vault in the cluster: 1. 104809984 GB Estimated time remaining for cluster's vault completion: 10 seconds ======================================== Director level vault status summary ======================================== /engines/engine-2 -1/directors/director-2 -1 -B: state: Vaulting - Writing vault data to vault disk Total number of bytes to vault: 566. 403072 MB Total number of bytes vaulted: 8. 220672 MB Total number of bytes remaining to vault: 558. 182400 MB Percent vaulted: 1% Average vault rate: 54. 050816 MB/second Estimated time remaining to vault complete: 10 seconds /engines/engine-2 -1/directors/director-2 -1 -A: state: Vaulting - Writing vault data to vault disk Total number of bytes to vault: 554. 848256 MB Total number of bytes vaulted: 8. 220672 MB Total number of bytes remaining to vault: 546. 627584 MB Percent vaulted: 1% Average vault rate: 51. 875840 MB/second Estimated time remaining to vault complete: 10 second Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 35

Forced Vault Recovery VPlexcli: /> vault override. Unvault. Quorum --evaluate-override-before-execution -c cluster-1 Cluster's unvault recovery quorum status: Only 3 out of 4 configured directors on this cluster are running, and none has reported a valid vault. All configured directors must be present to verify if any director has successfully vaulted dirty data the last time the cluster was servicing I/O. Missing directors in the cluster: director-1 -1 -A VPlexcli: /> vault override. Unvault. Quorum -c cluster-1 Warning: Execution of this command can result in possible data loss based on the current vault status of the cluster. Cluster's unvault recovery quorum status: Only 3 out of 4 configured directors on this cluster are running, and none has reported a valid vault. 3 out of 4 directors that were servicing I/O the last time the cluster had vaulted are present, which is sufficient to proceed with vault recovery. Do you wish to override unvault quorum? (Yes/No) Yes Execution of the override unvault quorum has been issued! Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 36

VPLEX Witness Failure Domain #3 Failure Domain #1 IP Management Network Failure Domain #2 Inter-cluster Network A Inter-cluster Network B Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 37

VPLEX Witness Troubleshooting: VPLEX Witness Connectivity The output for configuration automation has been captured in /var/log/VP!ex/cli/capture/VPlexconfiguration—session. txt configuration Evaluation of <<configuration cw-vpn-configure: -i 10. 127. 58. 76>> failed. cause: Command execution failed. cause: Unable to get the certificate subject info file from 10. 127. 58. 76 — Scp Connection Failure 3–Way VPN will Fail because of connectivity [email protected]: /> ping 10. 127. 58. 76 PING 10. 127. 58. 76 (10. 127. 58. 76) 56(84) bytes 64 bytes from 10. 127. 58. 76: icmp_seq=l tt 1=64 64 bytes from 10. 127. 58. 76: icmp_seq=2 tt 1=64 64 bytes from 10. 127. 58. 76: icmp_seq=3 tt 1=64 of data. time=4. 92 ms time=0. 403 ms time=0. 415 ms Management Server should be able to ping the VPLEX Witness Round trip latency should be < 1 Second Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 38

Module 7: VPLEX Troubleshooting Lesson 3: Summary During this lesson the following topics were covered: • How to list the connectivity components in VPLEX • How to define the logs and commands used to troubleshoot VPLEX connectivity • How to perform basic troubleshooting operations to determine and correct root cause analysis Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 39

Module 7: VPLEX Troubleshooting Lesson 4: Recovering Distributed Devices Upon completion of this lesson, you should be able to: • List the methods used to recover from WAN failure • Define the choices of resume and rollback • Show to recover from a WAN failure Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 40

Methods to Resume after WAN Failure • consistency-group resolve-conflicting-detach • consistency-group resume-after-data-loss-failure • consistency-group resume-after-rollback • consistency-group resume-at-loser Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 41

Link Failure Effects on Consistency Groups Detach Rule Cluster-1 Wins Cluster-2 Wins Active Cluster Wins No Automatic Winner C 1 is Active (I/O) C 2 is Passive (No I/O) C 1 is Passive (No I/O) C 2 is Active (I/O) C 1 is Active (I/O) C 2 is Active (I/O) I/O is allowed on C 1 Requires resume-after-rollback at C 1 I/O suspends at C 2 Requires resume-after-rollback at C 2 No data loss / No data rollback Requires a Data Rollback Requires resume-after-rollback at C 1 I/O suspends at C 1 Requires resume-after-rollback at C 2 I/O is allowed at C 2 Requires resume-after-rollback at C 2 Requires a Data Rollback No data loss / no data rollback Requires a Data Rollback I/O is allowed at C 1 I/O suspends at C 1 Requires resume-after-rollback at C 1 I/O suspends at C 2 I/O is allowed at C 2 Requires resume-after-rollback at C 2 No data loss / no data rollback Requires a Data Rollback I/O suspends at C 1 I/O suspends at C 2 No data loss / no data rollback No data loss / No data rollback Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 42

Consistency Group States Inter-Cluster Link is Down During a Failure Group has detached (I/O is going to the winning cluster). Group does not have a detach rule and I/O is suspended at both clusters. Group was active at the losing cluster and is suspended requiring the consistency-group resume-after-rollback command at the winning cluster. Inter-Cluster Link is Up After a Failure Group that was detached during failure continues to allow I/O at the winning cluster. Group that was previously suspended is now running and healthy at both clusters. Group has a conflicting detach because the administrator allowed I/O to go to both clusters while the inter-cluster link was down. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 43

Consistency-group resolve-conflicting-detach Winner Application Cluster Loser SAN SAN Restore Direction Distributed Device Access to storage during fail Copyright © 2012 EMC Corporation. All Rights Reserved. No access to storage during fail Module 7 - VPLEX Troubleshooting 44

Resolve-conflicting-detach Scenario VPlexcli: /clusters/cluster-1/consistency-groups/cg 1> ls Attributes: Name Value ------------------------------active-clusters [cluster-1, cluster-2] cache-mode asynchronous detach-rule no-automatic-winner operational-status [(cluster-1, {summary: : ok, details: : [requires-resolve-conflicting-detach]}), (cluster-2, {summary: : ok, details: : [requires-resolve-conflicting-detach]})] passive-clusters [] recoverpoint-enabled false storage-at-clusters [cluster-1, cluster-2] virtual-volumes [dd 1_vol, dd 2_vol] Visibility [cluster-1, cluster-2] Contexts: advanced recoverpoint Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 45

Resolve-conflicting-detach Resolution VPlexcli: /> VPlexcli: /clusters/cluster-1/consistency-groups/cg 1> resolve-conflicting-detach -c cluster-1 This will cause I/O to suspend at clusters in conflict with cluster-1, allowing you to stop applications at those clusters. Continue? (Yes/No) Yes VPlexcli: /clusters/cluster-1/consistency-groups/cg 1> ls attributes: Name Value --------------------------------------active-clusters [cluster-1, cluster-2] cache-mode asynchronous detach-rule no-automatic-winner operational-status [(cluster-1, {summary: : ok, details: : []}), (cluster-2, {summary: : suspended, details: : [requires-resume-atloser]})] passive-clusters [] recoverpoint-enabled false storage-at-clusters [cluster-1, cluster-2] virtual-volumes [dd 1_vol, dd 2_vol] visibility [cluster-1, cluster-2] Contexts: advanced recoverpoint Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 46

Consistency-group resume-after-data-loss-failure Winner Application Cluster Loser SAN SAN Restore Direction Distributed Device Access to storage during fail Copyright © 2012 EMC Corporation. All Rights Reserved. May have access to storage during fail Module 7 - VPLEX Troubleshooting 47

Resume-after-data-loss-failure Resume=True Scenario VPlexcli: /clusters/cluster-1/consistency-groups/CG 1> ls Attributes: Name Value --------------------------------------active-clusters [cluster-1, cluster-2] cache-mode synchronous detach-rule winner cluster-2 after 5 s operational-status [suspended, requires-resume-after-data-loss-failure] passive-clusters [] recoverpoint-enabled false storage-at-clusters [cluster-1, cluster-2] virtual-volumes DR 1_RAM_c 2 win_lr 1_36_vol, DR 1_RAM_c 2 win_lr 1_16_vol, DR 1_RAM_c 2 win_lr 0_6_vol, DR 1_RAM_c 2 win_lr. C_46_vol visibility [cluster-1, cluster-2] Contexts: advanced recoverpoint Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 48

Resume-after-data-loss-failure Resume=True Resolution VPlexcli: /clusters/cluster-1/consistency-groups/CG 1> resume-after-data-loss-failure -f -c cluster-1 VPlexcli: /clusters/cluster-1/consistency-groups/CG 1> ls Attributes: Name Value --------------------------------------active-clusters [cluster-2] cache-mode synchronous detach-rule winner cluster-2 after 5 s operational-status [ok] passive-clusters [cluster-1] recoverpoint-enabled false storage-at-clusters [cluster-1, cluster-2] virtual-volumes DR 1_RAM_c 2 win_lr 1_36_vol, DR 1_RAM_c 2 win_lr 1_16_vol, DR 1_RAM_c 2 win_lr 0_6_vol, DR 1_RAM_c 2 win_lr. C_46_vol visibility [cluster-1, cluster-2] Contexts: advanced recoverpoint Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 49

Resume-after-data-loss-failure Resume=False Scenario VPlexcli: /clusters/cluster-1/consistency-groups/CG 1>ls Attributes: Name Value --------------------------------------active-clusters [cluster-1, cluster-2] cache-mode synchronous detach-rule winner cluster-2 after 5 s operational-status [suspended, requires-resume-after-data-loss-failure] passive-clusters [] recoverpoint-enabled false storage-at-clusters [cluster-1, cluster-2] virtual-volumes DR 1_RAM_c 2 win_lr 1_36_vol, DR 1_RAM_c 2 win_lr 1_16_vol, DR 1_RAM_c 2 win_lr 0_6_vol, DR 1_RAM_c 2 win_lr. C_46_vol visibility [cluster-1, cluster-2] Contexts: advanced recoverpoint Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 50

Resume-after-data-loss-failure Resume=False Resolution VPlexcli: /clusters/cluster-1/consistency-groups/CG 1> resume-after-data-loss-failure –f –c cluster-1 VPlexcli: /clusters/cluster-1/consistency-groups/CG 1> ls Attributes: Name Value --------------------------------------active-clusters [cluster-2] cache-mode synchronous detach-rule winnercluster-2 after 5 s operational-status [ok] passive-clusters [cluster-1] recoverpoint-enabled false storage-at-clusters [cluster-1, cluster-2] virtual-volumes. D R 1_RAM_c 2 win_lr 1_36_vol, DR 1_RAM_c 2 win_lr 1_16_vol, DR 1_RAM_c 2 win_lr 0_6_vol, DR 1_RAM_c 2 win_lr. C_46_vol visibility [cluster-1, cluster-2] Contexts: advanced recoverpoint Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 51

Consistency-group resume-after-rollback Winner Application Cluster Loser SAN SAN Distributed Device Access to storage during fail IO stopped during fail Rollback to consistent data for restore Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 52

Resume-after-rollback Scenario VPlexcli: /clusters/cluster-1/consistency-groups/cg 1> choose-winner –c cluster-1 WARNING: This can cause data divergence and lead to data loss. Ensure the other cluster is not serving I/O for this consistency group before continuing. Continue? (Yes/No) Yes VPlexcli: /clusters/cluster-1/consistency-groups/cg 1> ls Attributes: Name Value --------------------------------------active-clusters [cluster-1, cluster-2] cache-mode asynchronous detach-rule no-automatic-winner operational-status [cluster-departure, rebuilding-across-clusters, restore-link-or-choose-winner]}), (cluster-2, {summary: : suspended, details: : [cluster-departure, rebuilding-across-clusters, restore-link-or-choose-winner] passive-clusters [] recoverpoint-enabled false storage-at-clusters [cluster-1, cluster-2] virtual-volumes [dd 1_vol, dd 2_vol] visibility [cluster-1, cluster-2] Contexts: advanced recoverpoint Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 53

Resume-after-rollback Resolution VPlexcli: /clusters/cluster-1/consistency-groups/cg 1> resume-after-rollback This will change the view of data at cluster-1, so you should ensure applications are stopped at that cluster. Continue? (Yes/No) Yes VPlexcli: /clusters/cluster-1/consistency-groups/cg 1> ls Attributes: Name Value --------------------------------------active-clusters [cluster-1, cluster-2] cache-mode asynchronous detach-rule no-automatic-winner operational-status [(cluster-1, {summary: : ok, details: : []}), (cluster-2, {summary: : suspended, details: : [clusterdeparture]})] passive-clusters [] recoverpoint-enabled false storage-at-clusters [cluster-1, cluster-2] virtual-volumes [dd 1_vol, dd 2_vol] visibility [cluster-1, cluster-2] Contexts: advanced recoverpoint Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 54

Consistency-group resume-at-loser Winner Application Cluster Loser SAN SAN Restore Direction Distributed Device Access to storage during fail Copyright © 2012 EMC Corporation. All Rights Reserved. IO stopped during fail Waits for user to resume Module 7 - VPLEX Troubleshooting 55

Resume-at-loser Scenario VPlexcli: /clusters/cluster-1/consistency-groups/cg 1> ls Attributes: Name Value --------------------------------------active-clusters [cluster-1, cluster-2] cache-mode asynchronous detach-rule no-automatic-winner operational-status [(cluster-1, {summary: : ok, details: : []}), (cluster-2, {summary: : suspended, details: : [requires-resume-at-loser]})] passive-clusters [] recoverpoint-enabled false storage-at-clusters [cluster-1, cluster-2] virtual-volumes [dd 1_vol, dd 2_vol] visibility [cluster-1, cluster-2] Contexts: advanced recoverpoint Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 56

Resume-at-loser Resolution VPlexcli: /clusters/cluster-1/consistency-groups/cg 1> resume-at-loser –c cluster-2 This may change the view of data presented to applications at cluster-2. You should first stop applications at that cluster. Continue? (Yes/No) Yes VPlexcli: /clusters/cluster-1/consistency-groups/cg 1> ls Attributes: Name Value --------------------------------------active-clusters [cluster-1, cluster-2] cache-mode asynchronous detach-rule no-automatic-winner operational-status [(cluster-1, {summary: : ok, details: : []}), (cluster-2, {summary: : ok, details: : []})] passive-clusters [] recoverpoint-enabled false storage-at-clusters [cluster-1, cluster-2] virtual-volumes [dd 1_vol, dd 2_vol] visibility [cluster-1, cluster-2] Contexts: advanced recoverpoint Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 57

Module 7: VPLEX Troubleshooting Lesson 4: Summary During this lesson the following topics were covered: • How to list the methods used to recover from WAN failure • How to define the choices of resume and rollback • How to recover from a WAN failure Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 58

VPLEX Troubleshooting This lab covers how to troubleshoot a VPLEX environment. • Perform a system data collection • Determine root cause analysis Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 59

Module 7: Check Your Knowledge - Questions • Which file has readable logs for the field support technician can be found? 4 diagnostics-extended 4 configdump. xls 4 Perpetual-monitors • If the cluster fails and cannot access the meta volume, what is the first step to recover the system? 4 Activate a backup copy 4 Copy the running metadata to disk 4 Promote a backup copy 4 Restore the metadata from the backup • What method is used to replace the array that holds the Metadata volume? 4 Mirror the meta device 4 Migrate the device 4 Configure a Mobility job for the volume 4 Promote a backup device Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 60

Module 7: Check Your Knowledge - Answers • Which file has readable logs for the field support technician can be found? 4 4 diagnostics-extended configdump. xls Perpetual-monitors • If the cluster fails and cannot access the meta volume, what is the first step to recover the system? 4 4 Activate a backup copy Copy the running metadata to disk Promote a backup copy Restore the metadata from the backup • What method is used to replace the array that holds the Metadata volume? 4 4 Mirror the meta device Migrate the device Configure a Mobility job for the volume Promote a backup device Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 61

Module 7: Check Your Knowledge - Questions • What does the director ping command test? 4 4 Latency between the sites Latency between the directors WAN latency VPN performance • What command is used to restore an asynchronous consistency groups that fails and both sites were active and have dirty cache? 4 4 resolve-conflicting-detach resume-after-data-loss-failure resume-after-rollback resume-at-loser Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 62

Module 7: Check Your Knowledge - Answers • What does the director ping command test? 4 4 Latency between the sites Latency between the directors WAN latency VPN performance • What command is used to restore an asynchronous consistency groups that fails and both sites were active and have dirty cache? 4 4 resolve-conflicting-detach resume-after-data-loss-failure resume-after-rollback resume-at-loser Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 63

Module 7: Summary • Show to perform data collection processes • Show to perform basic troubleshooting on various VPLEX • elements Show to collect and analyze data to resolve connectivity errors Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 64

Course Summary • Common VPLEX terms, configuration options, hardware and • • • software architecture Installation of a VPLEX Metro configuration Performing a Non-disruptive upgrade How data flows within a VPLEX system at a high level Managing VPLEX using the GUI and CLI Provisioning virtual volumes to hosts Encapsulating an existing SAN storage volumes into VPLEX Performing VPLEX mobility Designing VPLEX into a new or existing data center Performing routine VPLEX monitoring tasks Copyright © 2012 EMC Corporation. All Rights Reserved. Module 7 - VPLEX Troubleshooting 65