Lets Get Real Disaster Recovery and Business Continuity

Let’s Get Real: Disaster Recovery and Business Continuity in Public Safety Is Yours Just a Paper Plan or a Real Way to Prepare and Respond to Incidents and Disasters?

Presentation Overview • Key DR/BC Concepts and Issues – Report card and dashboard – Scenarios – Requirements: What has to operational by when for work to be done by how many at what locations serving what customers who are where? – Facilities – People – Systems – Integration – Coordination – Daily readiness and simulated escalations – Testing and independent verification and validation – Implementation and triage – Recovery, discovery, and improvements • • • Player Scorecard: Who Is In the Game and Why? DR/ BC Framework Action Steps to a Real Plan – – – • First steps Critical functions Funding and leveraging scarce resources Think out of the box Integration with the big picture DR/BC plan and activities of your jurisdiction Conclusions

Key DR/BC Concepts and Issues

The Report Card and Dashboard • All aspects of the plan, test, and implementation should be scored simply (Red, Yellow, and Green) • Key indicators of planning and readiness need a dashboard to enable assessment and action – Score or status – Trend – Key issue

Public Safety Scenarios • Public safety entities have a more difficult challenge • Your IT DR/BC plan is intertwined with risk scenarios • You may be affected by the risks of a given scenario and your IT plan must address those risks appropriately to maintain operations • You also have a role in response to the scenario so the events will affect your operational requirements

Scenarios Overview • Threat driven geographic circles of impact • Kinds of threats and events • Responsibility – What will you do, what is shared, what do others have to do for themselves • Tolerance for risk and uncertainty • Lesson learned: if you have a well known and documented local risk: – Have a real plan or get ready for a career change…

Source: IBM

Scenarios • Identify Possible and Likely Natural Disasters and Environmental Conditions By Kind and Duration of Effects – Tornado – Hurricane – Tsunami – Flood – Snowstorm – Drought – Earthquake

Scenarios • Identify Possible and Likely Natural Disasters and Environmental Conditions By Kind and Duration of Effects – Electrical storms – Fire – Subsidence and landslides – Freezing Conditions

Scenarios • Identify Possible and Likely Natural Disasters and Environmental Conditions By Kind and Duration of Effects – Contamination, Toxic releases and environmental hazards – Epidemic – Pandemic – Animal or crop disease outbreak

Scenarios • Organized and/or Deliberate Disruption – Act of terrorism • WMD – Acute and short lived (bomb) – Acute and long lived (dirty bomb) – Chronic » Long term (contaminants and biohazards) » Permanent (radioactivity, etc. ) • WLD (suicide bombers, car bombs, utility sabotage) • Bioterrorism or genetically modified or inorganic organisms – Direct contact – Infectious » Contact » Airborne

Scenarios • Organized and/or Deliberate Disruption – Act of Sabotage – Product or food tampering – Act of war – Theft – Arson – Labor Disputes / Industrial Action

Scenarios • Loss of Utilities and Services – Electrical power failure – Loss of gas supply – Loss of water supply – Petroleum and oil shortage • Raw materials • Refined materials – Communications services breakdown – Loss of drainage / waste removal and trash pickup

Scenarios • Equipment or System Failure – Internal power failure – HVAC failure – Equipment failure (excluding IT hardware)

Scenarios • Serious Information Security Incidents – Cyber crime – Malware – Zombie attacks – Denial of service – Loss or alteration of records or data – Disclosure of sensitive information

Scenarios • IT system failure (local or hosted) – Hardware – Software • Commercial application • Locally developed application – Data – Communications

Scenarios • Other Emergency Situations – Workplace violence – Public transportation disruption – Neighborhood hazard – Health and safety issues

Scenarios • Multiple and compound hazards and events – Purposeful – Coincidental – Causally connected – Interrelated

IT Requirements • What systems need to function • How fast – Maximum and optimum time frame for each system or function to be restored • How well – Sometimes minimal functionality is sufficient

IT Requirements • Where will it be used and by whom and will the communications infrastructure support it? – Employees – Users or beneficiaries • By what priority will systems be restored • The priority will be modified by what contingencies – E. g. a long term total evacuation changes the operational needs for criminal justice systems and personnel

Facilities • • Hot, warm, cold Mirrored, recoverable, reload-able Properly located EOC Non-EOC Operational IT facilities For user interaction with IT systems

Facilities • New kinds of mutual aid and sister city/county/state arrangements – Work with friends, colleagues, associations, and vendors – To match you with a comparable entities that are located outside the various geographic threat circles – Who can mirror your IT operations (hardware, software, operating systems, and culture)

People • The right numbers, skills, location, redundancy, etc. – Skills and abilities inventory • • Employees Contractors Vendors Mutual aid and “the cavalry”

People • Force in depth—who is the backup to the backup? • Consider the actual health and physical abilities and disabilities of a person when assigning tasks for a disaster scenario – The disaster is not the time to find out the electrician in the hazmat suit has a heart condition • What family and personal duties may interfere with performing official duties (e. g. save your own kids or save a stranger)?

Systems • • Daily operational Interdependent systems Emergency only Identity security and access management for physical and logical security – Follow FIPS 201 for federal/state/local interoperability

Integration • With whom should you work closely? • Identify integration issues between: – Internal systems and public safety entities – Other governmental systems – Related actors – Non-governmental systems and processes • Example: 911 and 311 or its equivalent – Normally separate but related – Emergencies blur the line – Co-location, cross training, and system integration

Coordination • • • Within organization Within unit of government Across units of government Across levels of government Across public and private boundaries

Daily Readiness and Simulated Escalations • • A disaster a day (“What, that’s not normal? ”) Realistic scenarios Captured lessons Learning and actually responding to lessons learned within risk framework • A quality and security framework for daily operations has substantial overlap with DR/BC

Security Capabilities Models Like similar capability models from the Carnegie Mellon SEI, SCMM models brings benefits: – Helps close security holes – Serves as a foundation for growth – Guides security leadership – Is evolutionary, not chaotic – Supports point solutions Security Leadership Strategy Security Sponsorship Causes Security Strategy Security Program Structure Security Program Resources and Skillsets Management Security Policies, Standard and Guidelines Security Management Security Administration Security Monitoring Knowledge User Management User Awareness Information Asset Security Application Security Technologies Database / Information Security Host Security Internal Network Security Network Perimeter Security Technology Protection and Continuity Support Physical and Environment Controls Contingency Planning Controls KPMG SCMM Model Effects

Capability Maturity Continuously improving process Optimizing (5) Like the SCI Managed CMM models, Predictable process (4) the KPMG Standard, Security Defined consistent (3) process Capability Model has Repeatable Disciplined (2) process five levels of maturity: Initial Informal process (1)

Testing and Independent Verification and Validation • Does the planned response or action step actually work? • Who verifies that it does? • What do you do if it fails the test?

Implementation and Triage • Someone better be in charge • Dispute resolution processes • Who will be your Sensibility and Sanity Checker (off site, not affected by the disaster, and actually getting enough sleep to make sound decisions)? • Baton Rouge example with Mayor Holden

Recovery, Discovery, and Improvements • What will the new normal be and when will it happen • Learn from history, both recent and long past • Document while the event occurs if at all possible (make it someone’s job) or soon after before memories fade

Player Scorecard Who Is In the Game and Why

Overlapping and Inter. Related Responsibilities Disaster Preparedness and Recovery and Business Continuity Physical Security Public Safety Quality Assurance Methodologies Cyber Security

The Usual Suspects in Public Safety • Police • Fire • Other sworn officers (transit, game, building or branch based, etc. ) • National Guard • Public Health • Public Works • Transportation • Environmental Protection

The Usual Suspects in Emergency Management • Federal, state and local emergency management entities • National Guard • NOAA, NWS, NSSL, other National Laboratories, • Corps of Engineers

IT Entities • CIO, CTO, and Enterprise IT Shops • Distributed IT Departments and leadership • Government IT contractors – DR/BC specific entities – Applications developers and software – Hardware – Service providers (ASP, MSP, call centers, etc. • Communications providers

Policy Makers • Executive, legislative, and judicial – Those who hold the seat and those who actually make the decisions… – Go below the top level to ensure clarity, alignment, and redundancy • EOC designees • Emergency authorizers

Non-Governmental Organizations • Media – Broadcast and satellite • Emergency Broadcast System Members – Print – New media • The Web – Government site mangers – Commercial site managers – Citizens and bloggers – Self-organizing communities (e. g. Craig’s List)

Non-Governmental Organizations • • Charities Businesses and business associations Community organizations Vital private services (hospitals, nursing homes, etc. )

A DR/BC Framework

Business Operations and Technology • Create a matrix, not a linear or organizational view • Strategy • Organization • Processes • Applications and data • Technology • Facilities

Source: IBM

Action Steps to a Real Plan

First Steps

First Steps • Leadership: clarity, alignment, and commitment • Authority or consensus? • Stakeholders roles and responsibilities • Be clear about risk tolerance • Applications and IT assets inventory – If needed, dust off and update your Y 2 K work • Good data on plan status, readiness, test results, response, and compliance

First Steps • Make a friend in accounting—actuarially accurate threat scenarios are more likely to be funded as risk and cost can be properly balanced • Review existing plan or make a plan • Borrow or buy a template • Review peer plans and conduct site visits • Communicate until it hurts

Critical Functions

Nail Down Your Critical Functions • Law and order essentials (people, mobility, tools, survival basics, etc. ) • Communications • Personnel management (policies, scheduling, notification trees and systems, counseling, etc. ) • Data and the connections to data and people • Transactional systems

Nail Down Your Critical Functions • Rescue and response • Pipeline to the health care system • Building/location/hazmat information for fire and first responders • Justice processing and incarceration • Dispatch

Nail Down Your Critical Functions • Records • Mobility – Devices and local storage if communications are intermittent or fail (e. g. mobile maps and databases) • Know what you can actually cover (and what you are just waiving your hands at and hoping it either works or is never needed)

Funding and Leverage

Funding and Leverage • Work within your risk/threat/cost/benefit matrix and follow your own rules • How serious are you about being prepared?

Funding and Leverage • Stop building single purpose infrastructures and reuse what you have – “Ask not, what an infrastructure can do for you, but what it can do for your taxpayers” • Use shared services • Follow standards or help create them if lacking

Funding and Leverage • Determine what pre-existing, unmet needs can be addressed by a new investment • Determine whether existing public safety or enterprise systems will do the job and if you can use them • Invest wisely – Vendors over inventors – COTS over customization – Web services over hard coding

Think Out of the Box

Think Third World • • Hand crank your computers Bike generators Solar and wind power Portable water purifiers Emergency shelter Runners and mountain bikes Hand tools

Think New World • Internet Protocol (IP) everything – Bridge between radio, wireless data/WI-FI and use each as IP conduits as needed • Gigs of portable flash memory • Satellite data and telephony

Think New World • • Instant Message Text and mobile email Cell On Wheels/Boat/Balloon Negotiate/legislate priority and bumping rights in telecommunications provisioning

Integrate With the Big DR/BC Picture

The Big Picture • Consult EM before, during, and after • Once essential public safety systems have a DR/BC IT and overall plan it can be incorporated into the overall EM plan for the jurisdiction • Tie it all together in formal and informal agreements • Create a focal point such as your EOC

EOC Basics • Not located in a hazard area (floodway) • 500 square feet minimum floor space • Communications section adjacent to EOC • Three methods of communications with state EMA and local responders • UPS and generator systems located above flood level • Sleeping space for identified staff • Kitchen space/food or meal contract • New construction to International Building Code Source: Alabama EMD

Conclusion: Essential Public Safety Systems and Organizations Must Be Disaster Resistant, Flexible, Diversified, and Redundant (Or We Are All In Big Trouble) Contact Information Richard J. H. Varn Center for Digital Government rjmvarn@msn. com

Model Plan Outline • What follows is a private sector based, but broadly applicable tool that sells for $199 • To buy a copy of the business continuity plan generator see http: //www. eoncommerce. com/rusecure/bcp. asp

Model Plan Outline • • Business Continuity - Preparing the Plan Initiating the BCP Project Initiation Activities BC 010101 Review of Existing BCP (if available)

Model Plan Outline • BC 010102 Benefits of Developing a BCP • BC 010103 BCP Policy Statement • BC 010104 Preliminary BCP Project Budget • BC 010105 Procedure for Approving BCP Content

Model Plan Outline • BC 010106 Communication on BCP Project to All Employees • Project Organization • BC 010201 Terms of Reference for BCP Project Manager • BC 010202 Appoint BCP Project Manager and Deputy • BC 010203 Select and Notify BCP Project Team

Model Plan Outline • BC 010204 Initial BCP Project Meeting • BC 010205 Project Objectives and Deliverables • BC 010206 Project Milestones • BC 010207 Project Reporting Requirements and Frequency • BC 010208 Required Documents and Information

Model Plan Outline • Assessing Business Risk and Impact of Potential Emergencies • Emergency Incident Assessment • BC 020101 Environmental Disasters • BC 020102 Organized and / or Deliberate Disruption

Model Plan Outline • BC 020103 Loss of Utilities and Services • BC 020104 Equipment or System Failure • BC 020105 Serious Information Security Incidents • BC 020106 Other Emergency Situations • Business Risk Assessment

Model Plan Outline • BC 020201 Key Business Processes • BC 020202 Establish Time-Bands for Business Service Interruption Measurement • BC 020203 Financial and Operational Impact • IT and Communications

Model Plan Outline • BC 020301 Specifications of IT and Communication Systems and Business Dependencies • BC 020302 Key IT, Communications and Information Processing Systems • BC 020303 Key IT Personnel and Emergency Contact Information • BC 020304 Key IT and Communications Suppliers and Maintenance Engineers • BC 020305 Existing IT Recovery Procedures

Model Plan Outline • Existing Emergency Procedures • BC 020401 Summary of Existing Procedures for Handling Emergency Situations • BC 020402 Key Personnel Responsible for Handling Existing Emergency Procedures • BC 020403 External Emergency Services and Contact Numbers

Model Plan Outline • BC 020500 Premises Issues • BC 020501 Responsibility and Authority for Building Repairs • BC 020502 Back-up Power Arrangements • Preparing for a Possible Emergency

Model Plan Outline • Back-up and Recovery Strategies • BC 030101 Alternative Business Process Handling Strategy • BC 030102 IT Systems Back-Up and Recovery Strategy • BC 030103 Premises and Essential Equipment Back-up and Recovery Strategy

Model Plan Outline • BC 030104 Customer Service Back-up and Recovery Strategy • BC 030105 Administration and Operations Back-up and Recovery Strategy • BC 030106 Information and Documentation Back-up and Recovery Strategy • BC 030107 Insurance Coverage • Key BCP Personnel and Supplies

Model Plan Outline • BC 030201 Functional Organization Chart • BC 030202 BCP Project Co-coordinator and Deputy for Each Functional Area • BC 030203 Key Personnel and Emergency Contact Information • BC 030204 Key Suppliers and Vendors and Emergency Contact Information • BC 030205 Manpower Recovery Strategy

Model Plan Outline • BC 030206 Establishing the Disaster Recovery Team • BC 030207 Establishing the Business Recovery Team • Key Documents and Procedures • BC 030301 Documents and Records Vital to the Business Process • BC 030302 Off-site Storage

Model Plan Outline • BC 030303 Emergency Stationery and Office Supplies • BC 030304 Media Handling Procedures • BC 030305 Emergency Authorization Procedures • BC 030306 Prepare Budget for Back-up and Recovery Phase

Model Plan Outline • Disaster Recovery Phase • Planning for Handling the Emergency • BC 040101 Identification of Potential Disaster Status • BC 040102 Involvement of Emergency Services • BC 040103 Assessing Potential Business Impact of the Emergency

Model Plan Outline • BC 040104 Project Management Activities • Notification and Reporting During Recovery Phase • BC 040201 Mobilizing the Recovery Team • BC 040202 Notification to Management and Key Employees

Model Plan Outline • BC 040203 Handling Personnel Families Notification • BC 040204 Handling Media during the Disaster Recovery Phase • BC 040205 Maintaining Event Log during Disaster Recovery Phase • BC 040206 Disaster Recovery Phase Report • Business Recovery Phase

Model Plan Outline • Managing the Business Recovery Phase • BC 050101 Mobilizing the Business Recovery Team • BC 050102 Assessing Extent of Damage and Business Impact • BC 050103 Preparing Specific Recovery Plan

Model Plan Outline • BC 050104 Monitoring Progress • BC 050105 Keeping Everyone Informed • BC 050106 Handing Business Operations Back to Regular Management • BC 050107 Preparing Business Recovery Phase Report • Business Recovery Activities

Model Plan Outline • BC 050201 Power and Other Utilities • BC 050202 Premises, Fixtures and Furniture (Facilities Recovery Management) • BC 050203 Communication Systems • BC 050204 IT Systems (Hardware and Software)

Model Plan Outline • • BC 050205 Production Equipment BC 050206 Other Equipment BC 050207 Warehouse and Stock BC 050208 Trading, Sales and Customer Service

Model Plan Outline • BC 050209 Human Resources • BC 050210 Information and Documentation • BC 050211 Office Supplies • BC 050212 Operations and Administration (Support Services)

Model Plan Outline • • • Testing the Business Recovery Process Planning the Tests Develop Objectives and Scope of Tests Setting the Test Environmental Disasters

Model Plan Outline • • Organized and / or deliberate disruption Loss of Utilities and Services Equipment or System Failure Serious Information Security Incidents Other Emergency Situations Prepare Test Data Identify Who is to Conduct the Tests

Model Plan Outline • Identify Who is to Control and Monitor the Tests • Prepare Feedback Questionnaires • Prepare Budget for Testing Phase • Training Core Testing Team for each Business Unit

Model Plan Outline • Conducting the Tests • Test each part of the Business Recovery Process • Test Accuracy of Employee and Vendor Emergency Contact Numbers • Assess Test Results • Training Staff in the Business Recovery Process

Model Plan Outline • • Managing the Training Process Develop Objectives and Scope of Training Needs Assessment Training Materials Development Schedule Prepare Training Schedule Communication to Staff Prepare Budget for Training Phase Assessing the Training

Model Plan Outline • • Feedback Questionnaires Assess Feedback Keeping the Plan Up-to-date Maintaining the BCP

Model Plan Outline • Change Controls for Updating the Plan • Responsibilities for Maintenance of Each Part of the Plan • Test All Changes to Plan • Advise Person Responsible for BCP Training