Principles of Incident Response and Disaster Recovery Chapter

Principles of Incident Response and Disaster Recovery Chapter 8 Disaster Recovery: Operation and Maintenance Principles of Incident Response and Disaster Recovery

Objectives • Understand the key challenges an organization faces when engaged in disaster recovery operations • Know what actions organizations take to prepare for the activation of the DR plan • Recognize what critical elements compose the response phase of the DR plan • Know what occurs in the recovery phase of the DR plan Principles of Incident Response and Disaster Recovery 2

Objectives (continued) • Understand how an organization uses the resumption phase of the DR plan • Know how an organization resumes normal operations using the restoration phase of the DR plan Principles of Incident Response and Disaster Recovery 3

Introduction • An organization should operate on the premise that it is only a matter of time until a disaster strikes • Proper response to a disaster requires meticulous preparation and ongoing diligence • In the event of a total loss, an organization must be prepared to promptly reestablish operations at a new permanent location Principles of Incident Response and Disaster Recovery 4

Facing Key Challenges • Disasters are not confined to the IT department or limited to the assets of the organization • Disasters may also affect the community and employees personally, as well as vendors and suppliers • In a major or widespread disaster, there may be challenges associated with local emergency services, service providers, and other-non business issues Principles of Incident Response and Disaster Recovery 5

Facing Key Challenges (continued) • Areas possibly affected in a major disaster: – – – Basic emergency and transportation services Food and survival supplies Water supplies and sanitation Electrical power Products and services delivered by vendors and suppliers – Telecommunications services (land cellular) – Transportation services (freeways, highways, and local streets) Principles of Incident Response and Disaster Recovery 6

Facing Key Challenges (continued) • Major disaster can result in: – Declaration of state of emergency – Imposition of martial law – Restrictions on movement or quarantines • DR plan typically involves 5 phases: – – – Preparation Response Recovery Resumption Restoration Principles of Incident Response and Disaster Recovery 7

Preparation: Training the DR Team and the Users • In DR planning, there is no prevention phase • Take steps during preparation to minimize losses • Preparation: making an organization ready for possible contingencies that escalate to disaster • Preparation phase is continuous, but other phases are activated by triggers such as: – – – Management notification Employee notification Emergency management notification Local emergency services Media outlets Principles of Incident Response and Disaster Recovery 8

Disaster Recovery Planning as Preparation • 3 primary objectives of the DR plan: – Eliminate or reduce potential for injuries or loss of life, damage to facilities, and loss of assets and records to minimize disruption and financial loss and reduce or limit liability exposure – Stabilize the effects of the disaster to allow recovery efforts to begin – Implement DR procedures Principles of Incident Response and Disaster Recovery 9

Disaster Recovery Planning as Preparation (continued) • Recovery efforts must be prioritized as follows: – – – Employees Customers Facilities Assets Records • CP team creates scenario development and impact analysis, and categorizes the level of threat for each potential disaster Principles of Incident Response and Disaster Recovery 10

Disaster Recovery Planning as Preparation (continued) • Key features of the DR plan: – Clear delegation of roles and responsibilities – Execution of the alert roster and notification of key personnel – Use of employee check-in systems – Clear establishment and communication of business resumption priorities – Complete and timely documentation of the disaster – Preparations for alternative implementations Principles of Incident Response and Disaster Recovery 11

Disaster Recovery Planning as Preparation (continued) • All employees should have 2 types of emergency information in possession at all times: – Personal emergency information (who to notify) – Instructions on what to do in the event of an emergency (snapshot of the DR plan) • Emergency info should include contact number or hotline for the organization, emergency services numbers, evacuation and assembly locations, disaster recovery coordinator, etc. • Crisis management: focused steps that deal with safety of people who are involved in the disaster Principles of Incident Response and Disaster Recovery 12

DR Training and Awareness • DR training focuses on the roles each individual is expected to execute during an actual disaster • For most employees, training is limited to awareness • General job function training is key to being prepared for disaster recovery actions • Cross-training should also be considered, both vertically and horizontally, to deal with personnel shortages • Training should include operating in degraded mode Principles of Incident Response and Disaster Recovery 13

DR Training and Awareness (continued) • Disaster management team (command control group) training is primarily about communication • Communications team training involves preparing information notices, news releases, and internal memorandums and directives • Hardware recovery team training may include training to rebuild damaged systems by scavenging from other damaged systems Principles of Incident Response and Disaster Recovery 14

DR Training and Awareness (continued) Principles of Incident Response and Disaster Recovery 15

DR Training and Awareness (continued) • Systems recovery team training is mostly the same as their normal operations training • Network recovery team training may include wireless network installation as a quick recovery mechanism, walkie-talkie deployment, and other connectivity mechanisms • Storage recovery team training may include rebuilding damaged storage systems and recovering data from offsite Principles of Incident Response and Disaster Recovery 16

DR Training and Awareness (continued) Principles of Incident Response and Disaster Recovery 17

DR Training and Awareness (continued) • Applications recovery team training primarily consists of skills used in normal operations • Data management team training focuses on rapid data restoration and recovery from backup • Vendor contact team training focuses on methods of obtaining resources as quickly as possible • Damage assessment and salvage team training primarily consists of hardware repair skills that enable team members to determine if items are repairable or not Principles of Incident Response and Disaster Recovery 18

DR Training and Awareness (continued) Principles of Incident Response and Disaster Recovery 19

DR Training and Awareness (continued) • Business interface team training includes communication skills and mechanisms for assisting with routine needs • Logistics team training includes training in purchasing and procurement and providing rest and comfort for other workers Principles of Incident Response and Disaster Recovery 20

DR Plan Testing and Rehearsal • Testing of the plan and the training and rehearsal of the plan can overlap • Testing can involve several levels of assessment: – – Employee self-assessments Peer evaluations Formally appointed internal assessors External certification or accreditation groups • Classroom training should come first before actual rehearsals Principles of Incident Response and Disaster Recovery 21

DR Plan Testing and Rehearsal (continued) • Testing strategies include: – DR plan desk check: individual review of plan – DR plan structured walk-through: group exercise – DR plan simulation: each individual works independently – DR plan parallel testing: act as if the disaster had occurred but do not interfere with normal operations – DR plan full interruption: act as if disaster had occurred, and perform all steps including data recovery – DR plan war gaming: few tools available for this in the private sector Principles of Incident Response and Disaster Recovery 22

Rehearsal and Testing of the Alert Roster • Alert roster must be tested more often than other plan components due to employee turnover • Quarterly testing is recommended • Alert message contains just enough information to allow employees to determine which part of the DR plan to implement • Auxiliary phone alert and reporting system: automated system for activating the alert roster • You are never completely ready for a disaster • Key skills to retain from rehearsals are flexibility, decisive decision making, and professionalism Principles of Incident Response and Disaster Recovery 23

Disaster Response Phase • Response phase: the phase associated with implementing the reaction to a disaster • Response phase focuses on controlling or stabilizing the situation for the purposes of: – Protecting human life and well-being – Limiting or containing damage to facilities and equipment – Managing communications with employees and other stakeholders Principles of Incident Response and Disaster Recovery 24

Recovery Phase • Recovery phase: – Initiates the recovery of the most time-critical business functions – Focuses on getting up and running as quickly as possible, even in degraded mode; less critical operations must wait for the resumption phase • Primary goals of the recovery phase: – Recover critical business functions – Coordinate recovery efforts – Acquire resources to replace damaged or destroyed equipment or materials – Evaluate whether to implement the business continuity plan Principles of Incident Response and Disaster Recovery 25

Resumption Phase • Resumption phase: focuses on non-critical functions • BIA should guide in the prioritization of critical and secondary functions • Goals of the resumption phase: – – Initiate implementation of secondary functions Finalize implementation of primary functions Identify additional needed resources Continue planning for restoration Principles of Incident Response and Disaster Recovery 26

Restoration Phase • Restoration phase: the final phase of disaster recovery • Primary goals of restoration phase: – Repair all damage to primary site or select or build a replacement facility – Replace damaged or destroyed contents of primary site including supplies, equipment, and material – Coordinate relocation from temporary offices to primary site or suitable new replacement facility – Restore normal operations at primary site, beginning with critical functions, then secondary operations – Stand down the DR team and conduct the afteraction review Principles of Incident Response and Disaster Recovery 27

Repair or Replacement • Two possibilities in restoration phase: – Reestablish operations at primary site – Establish operations at a new permanent site • Reestablish operations at primary site: – Must be able to rebuild damaged facilities – May need to relocate administrative functions to provide space to the operational functions while rebuilding is underway • New permanent site options: – New location – Complete rebuild on site of destroyed facilities Principles of Incident Response and Disaster Recovery 28

Restoration of the Primary Site • After physical facilities are rebuilt, the contents must be replaced, including: – Office furniture, PCs, photocopies, filing systems, office supplies, etc. • Must assess what will be covered by insurance and service contracts Principles of Incident Response and Disaster Recovery 29

Relocation from Temporary Offices • Transition back to the primary site must be carefully coordinated to minimize additional disruptions to business functions • If data management functions must move, may want to use a movement coordinator to plan the relocation of personnel, equipment, materials, and data back to the primary site Principles of Incident Response and Disaster Recovery 30

Resumption at the Primary Site • Must reestablish all normal operations, including tertiary operations that may have been suspended due to relocation, such as: – – – Managing employee benefit packages Employee training and awareness programs Organizational planning retreats and meetings Routine progress meetings and reports Long-term planning activities Research and development activities Principles of Incident Response and Disaster Recovery 31

Standing Down and the After-Action Review • Standing down: the deactivation of the disaster recovery team, releasing individuals back to normal duties • After-action review provides a method for management to obtain input and feedback from each group represented in the team • AAR log serves as legal and planning record and tool for future training • Official report should include AAR and reports from individual teams Principles of Incident Response and Disaster Recovery 32

Summary • An organization should operate on the premise that it is only a matter of time until a disaster strikes • 5 phases in the DR plan: preparation, response, recovery, resumption, restoration • Goals of DR and business resumption planning: eliminate or reduce potential for injuries or loss of life, stabilize the effects of the disaster, implement the DR plan based on type and impact of disaster • Recovery phase attempts to recover the most critical business functions immediately Principles of Incident Response and Disaster Recovery 33

Summary (continued) • Resumption focuses on the remaining unrestored functions • Restoration seeks to: – Repair all damage to primary site or arrange a replacement facility – Replace all damaged or destroyed contents – Coordinate relocation from temporary back to primary site – Restore normal operations at primary site – Stand down the DR teams and conduct the AAR Principles of Incident Response and Disaster Recovery 34