Lessons Learned From Superstorm Sandy Raj Goel CISSP

  • Slides: 20
Download presentation
Lessons Learned From Superstorm Sandy Raj Goel, CISSP Chief Technology Officer Brainlink International, Inc.

Lessons Learned From Superstorm Sandy Raj Goel, CISSP Chief Technology Officer Brainlink International, Inc. raj@brainlink. com / 917 -685 -7731

Murphy has plans… …and he executes better than you. © Bob Gorrell, www. Gorrell.

Murphy has plans… …and he executes better than you. © Bob Gorrell, www. Gorrell. Art. com In the last decade, New York City has experienced: • Multi-state power blackouts • 9/11 World Trade Center Collapse • Con-Ed steam pipe explosions • Tornadoes & Cyclones • Earthquakes • Sandy

NYC After Dark

NYC After Dark

Absolute power corrupts absolutely… …but we still need gas. • For the 1 st

Absolute power corrupts absolutely… …but we still need gas. • For the 1 st time in decades, NYC implemented gas rationing. • Even is people had power in their homes, and their neighborhoods were functioning, lack of gasoline kept people at home.

Nuclear reactors pack a lot of power… …but water is still king. • Rising

Nuclear reactors pack a lot of power… …but water is still king. • Rising tides & storm surges caused 5 nuclear reactors to go offline. –(Fukushima Daiichi problems were multiplied by ocean water flooding the backup generators). • Reactors going offline or being forced offline caused further strain on the electrical grid

Disaster Recovery & Business Continuity Lessons

Disaster Recovery & Business Continuity Lessons

1) In a flood zone, don’t put transformers or generators in the basement •

1) In a flood zone, don’t put transformers or generators in the basement • If you live in a flood zone, putting transformers or generators in the basement isn’t the smartest idea. • Nurses and staff saved countless lives by carrying patients out of the hospital manually. NYU’s BCP & DR plan was inadequate.

2) Utility outage maps are crucial • Just like pizza, even bad ones are

2) Utility outage maps are crucial • Just like pizza, even bad ones are better than nothing. • Even after LIPA & Con. Ed stopped updating their maps, knowing which areas were out, and which ones were functional allowed us to deal with employees better.

3) Redundant generators are awesome …as long as you have redundant fuel as well.

3) Redundant generators are awesome …as long as you have redundant fuel as well. • Peer 1’s data center had generators on 2 nd floor. • Peer 1 had their own generator on 17 th floor as backup. • Basement flooded – building generators offline. Peer 1 kept running…until diesel almost ran out.

4) Cash is King • Normal Hertz rate: $300/wk • Sandy rates: $2000/wk •

4) Cash is King • Normal Hertz rate: $300/wk • Sandy rates: $2000/wk • Normal hotel rate: $300/night • Sandy rates: $800/night • Helicopter hired by photographer Iwan Baan required cash up front to charter the chopper. • Brainlink had spare servers, drives, switches & firewalls set aside for clients BEFORE the storm

5) Geographical redundancy matters • A large, multinational firm with thousands of employees globally

5) Geographical redundancy matters • A large, multinational firm with thousands of employees globally hosted their exchange servers from NYC HQ. NYC lost power for a week. • No one had emails…globally. (CIO/COO had rejected previous recommendations for redundant data centers and offsite backups).

6) Leadership matters 1) Shutting down the traffic tunnels and subway lines was the

6) Leadership matters 1) Shutting down the traffic tunnels and subway lines was the best decision NYC’s government made. 2) Keeping cars and unnecessary vehicles off the street was a smart decision. This also made subsequent recovery faster. 3) Christie (NJ Governor) calling mayors stupid for not evacuating when ordered to – SMART! Saved thousands of lives and billions in losses.

7) People are your BEST assets • Are your employee contact lists up to

7) People are your BEST assets • Are your employee contact lists up to date? • Do you have out-of-state next-ofkin info? • Cellphones? IM/Skype IDs? Home phones? Spouse & children names, ages, contact info? • Prescription & OTC medications on hand?

How Brainlink dealt with Sandy Before the storm 1. We tested all client backups

How Brainlink dealt with Sandy Before the storm 1. We tested all client backups in the DR center 2. Ensured we have contact info for clients, client staff, family members 3. We published the DISASTER PREPAREDNESS TIPS page • http: //www. brainlink. com/2012/10/tropical-storm-sandy-disasterpreparedness-tips/

How Brainlink dealt with Sandy During the storm 1. I published a daily blog

How Brainlink dealt with Sandy During the storm 1. I published a daily blog updating clients (and others) with resources for recovery. • • http: //www. brainlink. com/2012/10/sandy-recovery-resources/ Free or low-cost office space, places to sleep or get hot food, hot showers, etc. 2. Called, texted, skype’d clients, employees, family members for 48 hours.

How Brainlink dealt with Sandy After the storm 1. We visited every client 2.

How Brainlink dealt with Sandy After the storm 1. We visited every client 2. Replaced many UPSes and power strips 3. Reviewed DR & BCP Plans 4. Clients purchased redundant / backup circuits for singlehomed clients 5. More clients adopted virtualization

Summary 1. 2. 3. 4. 5. 6. 7. Large, unprecedented events will happen more

Summary 1. 2. 3. 4. 5. 6. 7. Large, unprecedented events will happen more frequently Review building codes and best practices Power (and fuel) is KEY. Budget for spare resources. Geographical redundancy is imperative How your city or state plans for disasters MATTERS! People are more important than technology

Humor • Patron: “Barkeep, make me a Sandy!” • Barkeeper: “What’s that? ” •

Humor • Patron: “Barkeep, make me a Sandy!” • Barkeeper: “What’s that? ” • Patron: “You know…a watered down Manhattan : -) “ • They should have named the storm A-Rod. • Why? • Because then, it wouldn’t have hit anything.

Contact Information Raj Goel, CISSP Chief Technology Officer Brainlink International, Inc. C: 917 -685

Contact Information Raj Goel, CISSP Chief Technology Officer Brainlink International, Inc. C: 917 -685 -7731 raj@brainlink. com www. brainlink. com

About Brainlink Founded in 1994, Brainlink provides Computer Consulting for Small Businesses in New

About Brainlink Founded in 1994, Brainlink provides Computer Consulting for Small Businesses in New York City. Across the USA, Raj Goel personally provides • - COMMON SENSE BASED IT Security and Privacy Breach law compliance audits • - Information Security Audits • - HIPAA & HITECH audits for Healthcare If you like what you're hearing, hire us! www. Brainlink. com / www. Raj. Goel. com

Superstorm Sandy Lessons Learned and the Changing LandscapeSuperstorm Sandy Lessons Learned and the Changing Landscape
CISSP Overview CISSP Overview 1 Introduction 2 CISSPCISSP Overview CISSP Overview 1 Introduction 2 CISSP
Sandy Millar Avanade Australia sandy millaravanade com SandySandy Millar Avanade Australia sandy millaravanade com Sandy
Borough of Little Ferry Superstorm Sandy Public PresentationBorough of Little Ferry Superstorm Sandy Public Presentation
HURRICANE SANDY THE TENTH HURRICANE AND FIRST SUPERSTORMHURRICANE SANDY THE TENTH HURRICANE AND FIRST SUPERSTORM
Superstorm Sandy Geophysical and Human Activities Impacts TheoSuperstorm Sandy Geophysical and Human Activities Impacts Theo
Response and Recovery from Superstorm Sandy FWQA FebruaryResponse and Recovery from Superstorm Sandy FWQA February
Effect of Superstorm Sandy on Forest Health InEffect of Superstorm Sandy on Forest Health In
Superstorm Sandy Public Health Practice and Response NJSuperstorm Sandy Public Health Practice and Response NJ
CISSP Thomas Moore CISSP Thomas Moore Ph DCISSP Thomas Moore CISSP Thomas Moore Ph D
LDC Lessons Learned The Coaching Perspective Lessons LearnedLDC Lessons Learned The Coaching Perspective Lessons Learned
Lessons learned about lessons learned about hunger andLessons learned about lessons learned about hunger and
Lessons Learned from Science Society Survival Lessons LearnedLessons Learned from Science Society Survival Lessons Learned
Lessons Learned Process A Strawman Lessons Learned PurposeLessons Learned Process A Strawman Lessons Learned Purpose
Lessons Learned October 2017 Cameron Dewey Lessons LearnedLessons Learned October 2017 Cameron Dewey Lessons Learned
Lessons learned writing exploits LESSONS LEARNED WRITING EXPLOITSLessons learned writing exploits LESSONS LEARNED WRITING EXPLOITS
DBA LESSONS LEARNED Ten Lessons I Have LearnedDBA LESSONS LEARNED Ten Lessons I Have Learned
Lessons Learned LESSONS LEARNED PCs are only 10Lessons Learned LESSONS LEARNED PCs are only 10
Center Reorganization SRs Lessons Learned Lessons Learned onCenter Reorganization SRs Lessons Learned Lessons Learned on