When Technology Falters The Care Group Network Outage
- Slides: 30
When Technology Falters: The Care. Group Network Outage John D. Halamka MD CIO, Care. Group CIO, Harvard Medical School
Agenda In depth overview of the Network Outage Key Lessons The Sequel – SQL Slammer Questions and Answers
Care. Group Network as Built
Timeline November 13, 2002 1: 45 pm – Napster-like internal attack – Change begins, redundant links cut – Callisma and Cisco on site November 14, 2002 – Spanning tree issues – WAN issues – CAP declared at 4: 00 pm
Core Switch Utilization
Timeline November 15, 2002 – PACS Rebuild – Research/Cardiology rebuild – Reboot of core and distribution layer November 16, 2002 – VLAN mismatch – Redundant Core built as contingency
Core Switch Utilization
Root Cause Analysis Care. Group Network grew organically by Merger and Acquisition into a massive bridged switched network which was not within Spanning Tree spec Equipment was not life cycle managed Router/switch configuration was not in accordance with best practices i. e. multicast dense mode
Spanning Tree Problems When TAC was first able to access and assess the network, we found the Layer 2 structure of the network to be unstable and out of specification with 802. 1 d standards. The management vlan (vlan 1) had in some locations 10 Layer 2 hops from root. The conservative default values for the Spanning Tree Protocol (STP) impose a maximum network diameter of seven. This means that two distinct bridges in the network should not be more than seven hops away from one to the other.
Key Lessons Partner with your network vendor – Encourage external audits of your network – Engage advanced engineering services – Avoid senior management blind spots
Key Lessons Avoid flat topology bridged switched networks. Best Practice Care. Group Network One VLAN per Subnet per VLANs span many physical switches Limited or no bridging Extensive use of bridging Layer 2 switching limited to Layer 2 switching access layer extended across core
Key Lessons Re-evaluate the enterprise architecture of your network – Routed core – Switched distribution and access layers – Robust Firewall
Key Lessons Life Cycle Manage your network – Eliminate Legacy Protocols – Recognize the value of new feature sets – Hardware must keep up with the demands of a changing organization – video over IP, IP telephony, bioinformatics, image management
Key Lessons Implement appropriate monitoring and diagnostic tools to maintain the health and hygiene of your network – Concord – NATKit – Cisco. Works – Open. View
Key Lessons Have a robust downtime plan – Out of band diagnostics – Dial up modems and computers in key clinical areas – Overview of Care. Group Disaster Recovery plan
Service Objectives
Protection Features
Protection features
Protection Techniques Cost versus Benefit
Protection Techniques by Vulnerability
Key Lessons Implement Strict Change Control – Standards, configurations, devices, protocols, links, processes, procedures, or services – Prior review and approval of all network infrastructure changes – Multi-discipline membership – Changes classed as substantial, moderate, or minimal impact
Key Lessons Implement Strict Change Control (cont) – Substantial changes require Cisco AES review – Changes scheduled 2 am – 5 am weekends – Changes require baseline, testing, and recovery plans – As-Built documentation to include overall, physical and logical diagrams – NCCB recommends expense allocation
The Sequel – SQL Slammer Released at 12: 30 am on January 25 Infected East Coast at 12: 40 am Microsoft SQLServer 2000 was patched, however Microsoft did not issue any patches or security warnings on Microsoft Data Engine 2000 (MSDE), which is included with numerous desktop products
Spread of the Worm
Exact effect on Care. Group MSDE and non-IS maintained databases infected Network saturated by worm activity Shut off links to Research areas Blocked all traffic from the public internet Network traffic levels returned to normal
Cleanup Restart of servers and desktops that were disrupted by the outage Once all areas research areas had cleaned desktops, we restored port 1433 connectivity
Further Lessons learned VPN as a security risk Implement a scanning program to analyze research desktop and server vulnerabilities Ensure you have modern network equipment that afford you the tools to control intra. VLAN traffic
Conclusions Lifecycle manage your network just as you would your desktop Ensure senior management understands the value of the network as a strategic asset Build great downtime procedures including out of band connectivity just in case the technology falters
- Network outage reporting system
- Primary secondary tertiary health care
- Look angle definition
- Harita bright point
- At&t fiber outage
- Nitin nayar
- Osi network management model
- Asiconnect
- Equinix power outage
- Wmeco outage map
- Salesforce outage dns
- Hình ảnh bộ gõ cơ thể búng tay
- Lp html
- Bổ thể
- Tỉ lệ cơ thể trẻ em
- Chó sói
- Tư thế worms-breton
- Bài hát chúa yêu trần thế alleluia
- Các môn thể thao bắt đầu bằng tiếng bóng
- Thế nào là hệ số cao nhất
- Các châu lục và đại dương trên thế giới
- Công của trọng lực
- Trời xanh đây là của chúng ta thể thơ
- Cách giải mật thư tọa độ
- 101012 bằng
- độ dài liên kết
- Các châu lục và đại dương trên thế giới
- Thể thơ truyền thống
- Quá trình desamine hóa có thể tạo ra
- Một số thể thơ truyền thống
- Bàn tay mà dây bẩn