Disaster Recovery Technical Infrastructure at George Mason University

Disaster Recovery Technical Infrastructure at George Mason University January 13, 2011 Ron Secrest Mark Craft Barry Freese Where Innovation Is Tradition

Copyright Ron Secrest, Barry Freese and Mark Craft, 2011. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author. Where Innovation Is Tradition

Historical DR Strategies • “Sister Site” – Lack of excess capacity – Configuration issues • Rapid Replacement – Hardware availability? – Unable to test the process • DR Service Center Concept – Costly – Configuration management issues Where Innovation Is Tradition

The Basics: Risk Mitigation Techniques • 24 X 7 Vendor Support for Production Systems • RAID Disk Subsystems • System Backups (on-site & off-site) • Redundant Servers • UPS and Emergency Generator • Smoke Detection & Fire Suppression System Where Innovation Is Tradition

DR System Objectives • Off-Site University DR Facility • Phased deployment • Cost effective, scalable, ACTIVE-PASSIVE environment • Support mission critical systems, applications and services • Recovery Time Objective (RTO): 4 hours • Recovery Point Objective (RPO): 5 minutes Where Innovation Is Tradition

Features of the DR System • Dedicated gigabit network connectivity • Replication of mission critical databases, files, and systems • Redundancy • Remote monitoring • Remote systems management • Essentially a “lights-out” operation • On-site backups: Data Domain system • Active Services: DNS, DHCP, LDAP, AD, Kerberos Where Innovation Is Tradition

APC Rack Infrastructure Where Innovation Is Tradition

Replication Strategies In-line SAN Replication • Pros – No vendor lock-in – Similar hardware not required; JBOD could be deployed remotely. • Cons – Not integrated with hardware (software solution on 3 rd party hardware). Failover is problematic. – More technology to maintain – Additional vendor for support issues (e. g. problems, firmware/OS driver code levels). Where Innovation Is Tradition

In-line SAN Replication Where Innovation Is Tradition

Replication Strategies Native SAN Replication • Pros – Integrated with storage hardware – Less technology to manage • Cons – Vendor lock-in – Often requires similar hardware local and remotely Where Innovation Is Tradition

Native SAN Replication Where Innovation Is Tradition

Current Data Replication Methodology • Dedicated, private, (2) 1 Gb/sec network links between the Fairfax campus data center and the DR facility at the Prince William campus • 3 PAR SAN Remote Copy – Asynchronous – Block Level – Data changes are replicated every 5 minutes • • • Oracle Database Replication as a backup (Oracle standby) Backup replication: Data Domain system rsync (Linux) (Samba file server, Web server content) Active Directory & LDAP synchronization Kerberos Where Innovation Is Tradition

DR Technologies • VMware Site Recovery Manager (SRM) • SAN-to-SAN Asynchronous Replication • Native Replication Applications (e. g. LDAP, Active Directory, Kerberos, DNS) • DNS switchover Where Innovation Is Tradition

VMware Site Recovery Manager Where Innovation Is Tradition

VMware Site Recovery Manager • Pros – Automated failover – Non-interruptive testing – Summary report of testing results • Cons – Additional VMware licensing required – Falling back requires a new recovery plan to be created. – Possible IP/DNS changes needed for Virtual Machines on the recovery side. Where Innovation Is Tradition

Testing Methods • • Phase Testing is cumulative Fine tuning after technology changes Active Services (e. g. Active Directory): Extended Testing • VMware SRM failover (anytime including during business hours) Where Innovation Is Tradition

DR Failover Process • Shutdown local databases (if possible). • Disable data replication from the local Storage Area Network (SAN) to the remote DR SAN. • Initiate VMware Site Recovery via Site Recovery Manager (SRM). • After the SRM failover is finished initiate network routing changes. • Present DR SAN storage to the remote DR servers (SRM automatically provides this function for VMware servers). • Promote DR Active Directory domain controller. • Check status of DR LDAP, DR Kerberos and DR Active Directory. • Run scripts to switch DNS for servers or update DNS manually if necessary. This includes the emergency static web page. NOTE: DNS changes for all virtual hosts for the www. gmu. edu server will be necessary to provide a robust web environment if deemed necessary. Where Innovation Is Tradition

DR Site Documentation Kindle DR PDF Document Repository • Allows local access to documents at a remote site even if the servers are headless (USB port on Solaris server). • No additional infrastructure necessary to view the PDF documents • No maintenance necessary • Auto charges / auto mounts allowing remote access for updates without operational support. • Secure: only way to access documents is through the secure server • Efficient: first person on site can unplug and view procedures immediately. Where Innovation Is Tradition

Lessons Learned • DR funding should be included in the budget for new IT projects • Phased deployment is the right approach but be prepared for refreshing the technical infrastructure • Active-Passive is less complicated and more cost effective than an Active-Active architecture but fall-back time is significant Where Innovation Is Tradition

Summary of Benefits of Mason’s DR Program • • • Helps ensure business continuity Highly scalable architecture Allows for an incremental rollout Proven technology Ease of acquisition Provides a framework for meeting state and internal audit requirements • Due Diligence: A “real” Disaster Recovery solution Where Innovation Is Tradition

Questions? • Ron Secrest: Basic system architecture, acquisition, DR program management and testing • Barry Freese: System and storage architecture/management, replication and LDAP • Mark Craft: VMware, Site Recovery Manager and Active Directory Where Innovation Is Tradition

Mason Contact Information Ron Secrest Director, Enterprise Servers & Messaging 703 -993 -3354 rsecrest@gmu. edu Mark Craft Senior Technology Advisor 703 -993 -3454 mcraft 2@gmu. edu Barry Freese Senior Computer Systems Engineer/System Architect 703 -993 -9498 bfreese@gmu. edu Where Innovation Is Tradition

Thanks for attending! Where Innovation Is Tradition