Paradigm Shift in Outbreak Response Educause Security Professionals

Paradigm Shift in Outbreak Response Educause Security Professionals Conference Washington, D. C. April 4, 2005 Lance D. Jordan GSEC Beth E. Binde GSEC/GCIA/GCFA Copyright April 2005, Lance Jordan and Beth E. Binde. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.

In other words……. • “Now that I got SYSLOG running…. . what do I do with all this data? ”

Agenda • • About Rutgers University Challenges Summer of 2003 Birth of Automated Alert & Dispatch • Process details • Future Direction

About Rutgers University 500+ buildings 40 K+ hosts 60 K+ users 345 Mbps (Internet) Camden (Exit 5) 5 K+ students Newark (Exit 15 W) 10 K+ students New Brunswick/ Piscataway(Exit 9) 5 campuses 35 K+ students Public University Founded 1766 $1 Bil+ Budget 270 Degree programs 130 Research Institutes AAU Member

About Rutgers University Rutgers vs. Princeton November 6, 1869 Rutgers won 6 -4

About Rutgers University

About OIT @ RU 300+ FTEs 450 -500 Students 200 + FTEs 100 s Students

Challenges • Our computing environment…. . Probably looks a lot like yours…… • Stable & competent central IT staff – Departmental staff varying skill sets – Not centrally managed • • Large untrained & transient user population Transient work force (student workers) Numerous vendor products & multiple operating systems Open & highly distributed computing systems Competing priorities & projects Collaborative process Expectation of privacy Ease of access to information

A long, long time ago…. . Summer of 2003 Remember Mr. Peabody and the WABAC Back Machine? Jay Ward

Summer of 2003…. • Internet worm was predicted following the announcement of the RPC buffer overflow vulnerabilities by Microsoft • Exploit code appeared on July 2 • Department of Homeland Security issued warnings • Increased network scanning was noted

Summer of 2003…. • August 2003 – one of the worst months in the history of computer security – W 32/Blaster (RPC –DCOM) – W 32/Sobig. F (File shares & Email attachments) – W 32/Nachi/Welchia • And throw in a power blackout for extra credit….

Not-So-Good Old Days… • Incidents reported by email sent to abuse@rutgers. edu • 90% of reports from sources external to the university • Data entry was manual • Notifications sent in customized email, one message at a time Copyright © 2004 The Art Institute of Chicago

Not-So-Good Old Days… We needed a new game plan…… Automated Analysis and Dispatch (AAD) is born….

Overview of AAD Process • Collect data (firewall & IDS logs) • Analyze for abnormal network activity • Email alerts to departmental computing staff based on current threat signatures • External attacking IP data goes to DShield

And Now The Details…. Overview of Log Processing • • • Gather logs Transfer logs Process logs Dispatch notices Reports Clean up

Gather Logs • Strategically located sensors (network firewalls and IDS) direct logs to central server via syslog • Sensors are located on the network edge and on internal subnets.

Central Syslog Architecture

Transfer Logs • Logs are rotated and archived at midnight on syslog server • Logs are transferred to central computing host for processing • Procmail kicks off processes

Process Logs • For each log type, – Identify signature matches (listed in table) – Remove “exceptions” (list of IP addresses that trip false positives) – Check for the number of times each IP appears in logs (increment) – If appearance count is greater than the ceiling, write to work file – Identify most recent time stamp and write to output file

Dispatch Notices • Notices are sent to the RP (Responsible Person) of record • Multiple events are consolidated into one email message • Message may contain multiple alerts for one host, or multiple hosts with the same or multiple alerts

Dispatch Notice (Example) Subject: Event(s) of Interest: Your Department Name #2004 -11 -08 -1010 --2 From: Rutgers University CIRT <abuse@rutgers. edu> Date: Mon, 04 Apr 2005 10: 11: 03 -0500 (EST) To: joe@rutgers. edu, mary@rutgers. edu, sam@rutgers. edu, harry@rutgers. edu Please be advised that the Computing Incident Response Team (CIRT) has analyzed logs from firewalls and intrusion detection systems around the University and detected the network activity noted below. Please let us know your findings so that the reporting can be improved. The following network event(s) are reported from hosts on your subnet: Date/Time ; Dept ; IP ; #Repeats ; Event Type ; OS (best guess) 2004 -11 -07 22: 29: 50 ; evpacacooknjaesmisdialups ; MY. NET. 59. 136 445/tcp sweeps detected by firewall ; os not available ; 2 ; T 00445 port 2004 -11 -07 20: 27: 36 ; evpacacooknjaesmisdialups ; MY. NET. 59. 154 445/tcp sweeps detected by firewall ; os not available ; 1 ; T 00445 port

Reports • Hosts with 3 or more sequential events are flagged for staff follow-up • Disconnection from network is possible if problems persist

Sample Disconnect Report IP Address Count Source #1. MY. NET. 26. 201 5 E 10019 Possible Rogue. IRC 06 #2. MY. NET. 30. 205 4 E 10016 Possible Rogue. IRC 03 #3. MY. NET. 99. 226 4 D 00001 port scans on 445 #4. MY. NET. 30. 148 3 E 10016 Possible Rogue. IRC 03 #5. MY. NET. 31. 195 3 E 10016 Possible Rogue. IRC 03 #6. MY. NET. 193. 206 3 E 10019 Possible Rogue. IRC 06 #7. MY. NET. 70. 164 3 D 00001 port scans on 445 #8. MY. NET. 230. 221 3 E 10017 Possible Rogue. IRC 04 #9. MY. NET. 230. 221 3 E 10016 Possible Rogue. IRC 03 #10. MY. NET. 29. 209 3 D 00001 port scans on 445 #11. MY. NET. 30. 140 3 E 10016 Possible Rogue. IRC 03 #12. MY. NET. 90. 47 3 D 00001 port scans on 445 #13. MY. NET. 70. 164 3 T 00445 port 445/tcp sweeps detected by firewall #14. MY. NET. 90. 47 3 T 00445 port 445/tcp sweeps detected by firewall

Reports • Script creates (restricted) web pages created with log all information for each host in the output report • Summary data (Snort. Snarf & fwanalog) • Frequency analyses help pinpoint possible new attack patterns

Report Examples Blocked Packet Report Organization Report Packet Source Host Report Source Port Report

Sample Daily Dshield Report

Clean up • Remove work files • Reset files that contain run parameters • Archive log files on central server • Archive output • Check disk quota and remove old webpages as needed

Current Issues/Future Directions • Replace current process with commercial off the shelf software or “improved homebrew ? ” • “Necessity is the mother of invention, ” Current system developed in the heat of battle • Hodgepodge of scripts taken from other applications and “stirred with a stick” • Time vs. cost vs. flexibility • “Homebrew” allows customized, immediate dispatch of incidents with multiple IP addresses

Thank you for your attention Office of Information Technology Enterprise Systems and Services Information Protection and Security 56 Bevier Road, ASB Annex I VOICE: 732 -445 -8011 EMAIL: rusecure@rutgers. edu WEBSITE: http: //rusecure. rutgers. edu