A Business Continuity Plan for Government George Bomar

A Business Continuity Plan for Government George Bomar Dianne Casey Texas Department of Licensing and Regulation

A practiced logistical plan for how an organization will recover and restore partially or completely interrupted critical functions within a predetermined time after a disaster or extended disruption.

The Focus on People “For the main event, CIO Steve Yates wanted to test more than the company's technology procedures; he wanted to incorporate the most unpredictable element in any contingency planning exercise: the people. ” USAA Insurance Company

Legacy of Y 2 k - Computer failures in banking, power, health, telecommunications and financial institutions September 11, 2001– “Worst case” scenario concept shifted

Selected Stats 80% of companies worldwide are not prepared for a pandemic or a natural disaster U. S. DOL estimates over 40% of businesses never reopen following a disaster Of the remaining 60%, 25% close within 2 years.

Selected Stats Fires permanently close 44% of businesses affected 90% of companies that lose data are forced to shut down within 2 years 1993 World Trade Center bombing 150 of 350 affected businesses failed

More Arkansas Poultry Flocks Checked For Bird Flu (UPDATED SATURDAY, JUNE 14, 2008 5: 55 PM CDT IN NEWS) By The Associated Press “Within a few days all commercial chicken houses in the area had been tested and the 15, 000 birds affected were killed and buried. The next step was for the commission to go door-to-door, checking for other cases. ”

The Food and Drug Administration is expanding its warning to consumers nationwide that a salmonellosis outbreak has been linked to consumption of certain raw red plum, red Roma, and red round tomatoes, and products containing these raw, red tomatoes. June 5, 2008 The Emergency Email and Wireless Network

What does BCP “look like” Formal printed manual Full access by employees Stored in multiple locations Secondary work center Copies of critical materials

Relationship to Disaster Recovery Plan DR - focused on information technology applications domain Overlap with BCP Crisis mgmt structure Secondary work center

Data requirements between primary and secondary work centers: Telecommunications architecture; Data replication methodology; Application and software availability; Any physical data requirements at secondary site.

Recommended BCP approach Plan for the BIG disasters Smaller ones always contain partial elements of larger disasters BCP should be broader than disaster recovery alone or in case of emergency (“ICE”) procedures

BCP Purpose To enable leaders to maintain essential business processes and practices and equip the organization with means of becoming less vulnerable to incidents

The TDLR Plan Identifies management team members Designates remote site(s) Enumerates four (4) major scenarios Itemizes recovery steps to be taken within five (5) primary business functions

Events That might trigger an interruption Loss of key personnel Weather-related Infrastructure-related Internal system breakdowns

Events That might trigger an interruption Failure of an external business partner Health crisis impacting the work force A cyber attack An act of terrorism

Rating the Triggers Probabilities of occurrence 1 - Least likely to happen 4 - Most likely to happen

Impacts DURATION Will the effects be short-term, or longer? EXTENT How much of work force is impacted?

Devising a Template A questionnaire was circulated to capture: Recovery procedure Recovery time objective Recovery location Dependencies Other considerations Summary of recovery steps

The Process Solicit written input from key personnel via templates Interview managers Prepare draft for each business function Obtain review comments and incorporate into revised draft

How About Prevention? Mitigate the impact of a disaster: Practice good housekeeping Adhere to security procedures Observe information security procedures Maintain up-to-date operating guidelines

An Emergency Management Team Convenes to decide: Implement the BCP? Activation prompted by Team Lead

Alternate Location(s) Primary Site Alternate Site BCP provides directions to the sites

Scenario I The population of possible causes was condensed into four (4) major scenarios: Loss of key executive personnel for a protracted period due to accident or other unforeseen event;

Scenario II Loss of building access because of weather (or other natural disaster)-related event;

Scenario III Contractor default, or other supplier of a critical service to the agency, abruptly goes out of business without warning; and,

Scenario IV Health crisis (or act of terrorism) leads to an exorbitant rate of employee absenteeism (and temporary replacements are unavailable).

Functions Impacted The plan identifies five (5) main business functions adversely affected by the crisis: Licensing of individuals and businesses Education and examination activities Measures to ensure compliance Administrative support Technological support

Initial Approach For each of the five (5) business functions, Identify impact, Recovery procedures, and Dependencies Redundancy

Adopted Approach For each of the four (4) scenarios: Identify how each business function would be adversely impacted

Example I If key personnel were lost (Scenario I) Notify the agency’s directors Convene emergency meeting of the Commission Formulate short-term succession plan Notify Governor’s office and key legislators Designate primary agency contacts Implement plans to notify the public, equip customer service, respond to complaints

Example II If building was inaccessible (Scenario II) Licensing Education and Examinations Compliance Administrative Support Technological Support

Example III If major contractor failed (Scenario III) Identify affected functions Marketplace alternatives? Make temporary process changes Procure new/other contractor

Example IV If a health crisis decimated the work force (Scenario IV) Identify skills of available staff Can skills be realigned? Determine what functions (e. g. inspections) can be postponed or suspended Consider tapping into regulated industries for temporary expertise

A Summary of Recovery Steps Plan must specify: Key actions to be taken, By whom, In what order, For each business function.

Important Addenda Identify in an Appendix BCP Team Lead and Members with current contact information Name and address Phone number(s) E-mail address(es)

Include: a Phone Tree listing - who will contact whom; Identify how information will be disseminated to employees; List first group(s) to report to alternate site.

Periodically, re-assess your BCP and update as needed!

Testing Purpose: Achieve organizational acceptance Determine that the BCP solution is appropriate for recovery requirements Identify and correct design flaws Identify and correct implementation errors

After 9/11, those companies with tested BCP manuals had business resumption within days.

Selected Stats 45% of companies with a BCP do not test it annually 80% of companies have not developed an IT crisis management function 40% of companies that have a crisis management plan do not have a dedicated crisis management team

Mistakes and Pitfalls Failing to gain senior level management support Not identifying all critical systems (including laptop data) Failing to bring the entire business into planning and testing Not identifying and planning for all gaps in recovery objectives Insufficient funding for testing

USAA Story 20, 000+ employees - needed Haz. Mat training, an evacuation plan and a recovery plan Live exercises were confined to technology assets - recovering data from backup data Otherwise, passive exercises – tabletop and paper simulations, role-play, guessing how people would react

USAA Story Post 9/11, built alternative center 200 miles away from San Antonio, on different power grid and water supply Steve Yates designed large scale continuity exercises At the first one, USAA discovered: The setup process for computers and phones took nearly two hours leaving employees standing in the hot Texas sun.

USAA Story USAA ‘take-away’ from testing: Those who walked through the simulation were in the best position to find flaws and offer suggestions. Those who practice emergency situations are less likely to panic and are more likely to remember the plan.

Plan Maintenance Cycle Revisit annually or biannually Confirm information; roll out to all staff Perform staff training Test and verify technical solutions for recovery Test organization recovery procedures

Questions ? ?

Presenters: George Bomar – 512 -936 -4313 GBomar@license. state. tx. us Dianne Casey – 512 -463 -7182 Dianne@license. state. tx. us Texas Department of Licensing and Regulation