University of Nebraska 2012 Breach Chris Cashmere University

University of Nebraska 2012 Breach Chris Cashmere, University of Nebraska Central Administration Dan Buser, University of Nebraska at Lincoln 1

Overview In May 2012 the University of Nebraska experienced one of the largest breaches in higher education It took months to figure out exactly what happened It took years to finally come to somewhat of a conclusion It cost the University hundreds of thousands of dollars It was investigated by the FBI and DOE (Department of Education) The attacker was caught and prosecuted by the US Attorney 2

Today’s Agenda We will cover just a fraction of what happened We will jump around just covering the highlights Some technical details Public response and communication Impact and costs Legal stuff Lessons learned and takeaways 3

Who Exactly? The breach involved two entities The University of Nebraska Central Administration The Student Information System (670, 000 records) A person obtained unauthorized administrator access to the student information system database for approximately 24 hours The University of Nebraska at Lincoln campus The Degree Audit System (20, 215 students) A person (same as above) obtained degree audit system records via a web application attack 4

UNL Attacks What he got from the outside was some, but he wanted more Used a VPN service in Sweden to try to cover his tracks • Feb 20 th thru May 20 th • Scanned a total of 72 systems • Dumped a database from the Degree Audit system containing student information (but no PII) He needed insider access 5

Darwin 6

UNL VPN Setup UNL took control of its own VPN from Central just months before the breach All University Facility, Staff, and Students have access to VPN At the time, all general VPN users were put into one big IP pool Firewall rules were open at Central to allow that pool through, instead of just Admin users For unrelated reasons we had full PCAP capture on the UNL VPN…which we will cover later… 7

Ne. SIS Background Nebraska Student Information System (Ne. SIS) Overview - 2012 Major components are the People. Soft application and the Oracle database Central Administration manages two complete instances, one for the University System and one for the Nebraska State College System Ne. SIS had been operational for two years at the time of the breach 8

University of Nebraska/Nebraska State College System (Ne. SIS) Chadron Wayne CSC WSC UNO Omaha Ne. SIS consists of 4 Universities 3 State Colleges 1 Technical College* 2 separate environments Universities State Colleges Kearney UNK Curtis NCTA UNCA Lincoln UNMC UNL PSCPeru Over 59, 000 active users in the two systems

The Ne. SIS Database PII which included SSN, name, address, DOB, etc. for… ◦ ◦ Current Students Parents Alumni and non-graduates (separated), many years worth Faculty and Staff ◦ I was never a student at UN yet I was in the student system? ? ? ◦ ACT/SAT – Potential Students ◦ If you took the ACT/SAT in Nebraska or checked a Nebraska school for your scores ◦ This data is where most of the compromised SSNs originated ◦ Included bank account information for more than 20, 000 students ◦ No Credit Cards 670, 000 Records/Users* 590, 000 Users after de-duplication* (*at the time of the breach, May 2012) 10

Technical Details 11

How Did This Happen? Insider Access • The attacker was a University of Nebraska at Lincoln student and was allowed VPN access to internal campus resources. An insider? Firewall Config • The internal firewall that protects the SIS Oracle database was open to connections from the VPN IP space which students were assigned Default Password The Hack • The People. Soft default "PEOPLE" user password which permits access to passwords tables during user authentication was not changed • The attacker was, using the PEOPLE account, was able to derive the plain text passwords for the database SYSADM accounts from the encrypted form •

Cyber Kill Chain Access Recon Exploit Foothold Exfiltration CAUGHT HERE 13

VPN/Firewall was configured (unintentionally) to allow VPN ip addresses access to the SIS system Based on the VPN IP connecting to the databases it was easy to track that back to a user that was assigned the IP He logged into the VPN with his regular student ID, so very easy to tell who it was Once we knew the user we could find all the external IP addresses the VPN account logged in from. Three different locations, these were turned over to law enforcement within 24 hours Lincoln NE, his apartment Omaha, turned out to be his Mom and Dad’s house Lincoln NE, his place of employment in (data mining company) Access 14

Scanning/Recon Basically tried various scans and exploits three weeks, May 6 th to May 23 Scanning with Linux Free tools like, we found out about some of these from FBI/DOE NMAP (port scanner) Nessus (vulnerability scanner) Hydra (network login hacker) Scuba (database vulnerability scanner) Nikto (web application scanner) Tried an Oracle TNS listener attacked for a while …but he flew under the radar All the connections he made were allowed Recon 15

PEOPLE/peop 1 e People. Soft used to be delivered with a default user and password This was the case when we deployed the PS system in 2010 People. Soft is now delivered without a default PEOPLE password, you have to enter one in when you install People. Soft, this was done in the past 18 -24 months New installs of 8. 53 or later you are good. You get a prompt to set a password for the PEOPLE account. Upgraded? You will still have the default PS password People gets you access to a few tables, 2 tables are important Exploit 16

PSOPRDEFN and PSACCESSPRFL June 12 th – Received a DVD from FBI/Police with file named “user_table_from_DB. csv” • The ‘data’ was from one of the seized computers. • File contained 179, 377 rows of IDs and encrypted and hashed passwords. Possible to crack these at the time of the breach? Exploit? 17

People Soft Login This was at the time of the breach, still may be the case… Most People. Soft desktop tools work this way, all have the default PEOPLE/peop 1 e user and password. User enters valid ID and any password credentials App first connects to database using the PEOPLE database account The user 'people' issues a couple of select statements to validate the credentials provided. Checks if the user exists. The 'people' user also gets the data owner usually SYSADM and the encrypted password for SYSADM. Desktop app disconnects from database Desktop app decrypts the SYSADM password Desktop app connects using SYSADM/password to verify the users password Desktop app disconnects from database The user is logged in Exploit? 18

Vice President of Finance May 23, 2012 10: 00 p. m. Central Administration staff received an error message in People. Soft. An account creation had failed, which was a common occurrence. An impromptu investigation determined it was result of a VP 2 account being created Locked suspicious accounts immediately and continued to monitor the application. VP 1 is a delivered PS account We locked and stripped privileges/security Using SYSADM he unlocked VP 1 and escalated it’s privileges and logged in Cloned security persons (Denise Goin) account into a VP 2 account VP 2 may go unnoticed? The creation process of the VP 2 account creation failed and alerted our system administrators. Failed during a sync of accounts Foothold 19

Exfiltration? (which we interrupted) 20

The Notebook Here is where we first got a sense of what he was up to… Search warrant executed on May 25 th. Law enforcement seized computers and a NOTEBOOK from his home and work. Law enforcement asked us to look to see if we could help identify what was in his notebook. A lot if it would not make sense to an ‘outsider’ Notes for classes (CSCE 421 Foundations of Constraint Processing) Hostnames and IP for internal SIS systems SYSADM passwords for all the databases PS 3 DES Encrypt(char*; int; char*; int) Exfiltration 21

The Notebook Notes about Gambling sites Notes on Banks in Canada Notes on how to monetize information gathered How and where to sell PII Net. Teller Moneybrokers Mt. Gox (bitcoin) Aurum. Xchange (bitcoin) Web. Money Dwolla Ipredator (VPN) Intercasino <- Net. Teller <-Ukash Interpoker. com He claimed he going to change his grades PS_stdnt_grd was on the same page as the SYSADM passwords Exfiltration 22

PCAPS Full PCAPS for all his activities. 7 GB of PCAP files. Every packet he sent or received was there. Luck was on our side here We did lots of searches in the PCAPS and extracted analyzed every select statement he performed and the results We were able to determine, with a quite a bit of certainty, only about 150 records with SSNs were downloaded or viewed We were able to see everything he did and every bit of data he received. He basically “fumbled around” for a few weeks 99% of the select statements – ‘no rows returned’ Split tunnel VPN, traffic for our networks went through the VPN and was captured while general internet traffic did not However in the PCAPS we saw all his DNS requests Visit some forums and then try other stuff and fail 23

Public Response 24

Communication 25

Communication Established a single point of contact, CISO First Press Release issued less than 48 hours of discovery ◦ Issued other PRs when significant information needed to be shared Created dedicated email box, asked for questions and answered every question we could Built a dedicated website for the security incident to serve as central information site. (http: //nebraska. edu/security) Established our own call center to address specific questions about a week after the first press release (June 1 st) Built web application to allow call center and eventually public search to see if their data was part of breach Mostly did email notifications, very few “letters” sent 26

Call Center June 1 st 2012, Friday +8 days 27

News 28

Public Response • Received over 250 questions and comments through website submission • Received 656 calls to call center June 1 st to 7 th • Web traffic peaked the first four days and steadily decreased Peak: 5, 526 Lasted for two years, every news story and every court proceeding refocused attention an brought news stories and phone calls 29

Impact December 22 nd 2012, Happy Holidays! 30

Impact December 22 nd 2012, Happy Holidays! 31

12 Charges December 22, 2012 • 2 violations of Section 18 of US Code 1030 (a)(5)(A), felonies with up to 10 years imprisonment 18 U. S. Code § 1030 - Fraud and related activity in connection with computers (a)Whoever— (5) -- (A) knowingly causes the transmission of a program, information, code, or command, and as a result of such conduct, intentionally causes damage without authorization, to a protected computer; • 10 violations of Section 18 of US Code 1030(a)(2)(C), misdemeanors with no more than 1 year imprisonment 18 U. S. Code § 1030 - Fraud and related activity in connection with computers (a)Whoever— (2) intentionally accesses a computer without authorization or exceeds authorized access, and thereby obtains— (C) information from any protected computer 32

Subpoena (email PDF) 18 months after the incident 12 months after being charged 33

Plea Pled guilty to 1 felony violation of Section 18 of US Code 1030 (a)(5)(A) 1 felony and 10 misdemeanors were dismissed Six months of work release from a halfway house in Council Bluffs IA Six month house arrest (ankle bracelet) Then 2 years supervised release (probation) Pay restitution Sentenced July 24 2014 The Stratman family collected the restitution money and presented it at sentencing hoping the judge would take that into consideration, he did not. The judge was pretty clear to Stratman the money paid by his family must be paid back to them. 34

Restitution Total damages claimed by the University was $302, 938. 00 Total allowed by the court was $107, 722. 58 Restitution awarded and paid was $107, 722. 58 Most of the restitution was our consultant costs. We could actually prove we spent that money EXCLUSIVLY for the breach. We could not provide enough details to prove most of the damages and costs. The judge threw out those and only allowed the 107 K in damages and restitution. If we’d had actually shut down the SIS system, damages and charges would have also increased Judge: “You are free to try and recover the rest of the damages in civil court. ” 35

Restitution Any institutions here, who experienced a breach, recover some or all of their costs? 36

Our Costs Total $302, 938. 00 This is the number we submitted to the court. • • • UN-Central Administration Staff (3, 124. 5 hours) $152, 000 UN-Lincoln Staff (508 hours) $24, 000 (estimate) Call Center $1, 500 Letter Prep and Printing $325 Postage $330 Storage $700 (7 -1 TB USB Drives) UNCA Consultant $29, 000 UNL Consultant $80, 000 En. Case for UNL $17, 000 150 records / $303 K = $2, 000 per record 590 k records / $303 K = $0. 51 cents per record 37 IBM 2012

Lessons, Takeaways, Observations, Reflections, and Oh By the Way… 38

Evolution of the Breach 1 st Pass: 670, 000 Users potentially impacted 2 nd Take: 590, 000 Users after de-duplication Potential Impact Days Weeks Final Number: Less than 150 users PII was accessed by attacker Confirmed Impact Months

Communication Dilemma Quick Slow Transparent Hiding Not Accurate Happy Unhappy

Lessons/Takeaways Change the PEOPLE password This is the best way to protect your database Apply your patches ASAP The TNS listener attack he tried Oracle patched in their April 2012 CPU October 2014 People. Soft update increases complexity of hashes Review your VPN setup Run an discovery scan (nmap)…before someone else does Review data retention policies Breach impact could have been lessened if unneeded records had been removed from the system 41

Lessons/Takeaways Keep track of your hours during a breach investigation Key players keep notes on time spent Summaries and estimates won’t cut it in court Keep track of all expenses Remember to note the details Summaries and estimates won’t cut it in court The wheels of justice turn very slowly Over two years for it to be resolved Plan on educating the prosecutor Our DBA and People. Soft security persons spent hours getting him up to speed Your prosecutor selection is based on luck, we were unlucky 42

Lessons/Takeaways The Prosecutor was and will be obsessed with “damages” Prosecutors need and want a dollar amount Damages is an escalator for the offence If it were not for the fact he deleted some audit records we may not have been able to prove he actually ‘damaged’ anything Defense will focus on the damages and disputed them Details and documentation is very important You must be able to prove damages They must hold up to scrutiny You may have to testify and be cross examined about time spent Know your campus resources, and build relationships We had a relationship with PKI and faculty running the information Assurance Program at PKI 43

Lessons/Takeaways If you catch the hacker you can drag out a breach for years, in our case two years …but you might recuope some or all of your costs All people will remember is the BIG numbers (http: //www. privacyrights. org/) If Mr. Stratman had come clean when the Police showed up at his apartment on May 25 th 2012, he would have likely avoided most of what happened to him, cost and felonies. Probably 1 misdemeanor and diversion, no criminal record 44

Questions? Is there anything you can’t tell us? What have you done since the breach? Did the University have Cyber Insurance? Why did it take you so long to share this information? Anything changed at your organization because of the breach? 45