15 744 Computer Networking L23 Network Intrusion Detection

  • Slides: 73
Download presentation
15 -744: Computer Networking L-23 Network Intrusion Detection Systems (NIDS)

15 -744: Computer Networking L-23 Network Intrusion Detection Systems (NIDS)

NIDS • Background • Bro: A NIDS • NIDS Traffic Normalization • Honeycomb: NIDS

NIDS • Background • Bro: A NIDS • NIDS Traffic Normalization • Honeycomb: NIDS signature generation 2

Recall: Network “ 101” vs. Reality Traditional view: “Dumb” network Reality: Lots of in-network

Recall: Network “ 101” vs. Reality Traditional view: “Dumb” network Reality: Lots of in-network processing Appliances or Middleboxes: IDS, Firewall, Proxies, Load balancers…. 3

Recall: Middleboxes Galore! Data from a large enterprise Type of appliance Survey across 57

Recall: Middleboxes Galore! Data from a large enterprise Type of appliance Survey across 57 network operators Number Firewalls 166 NIDS 127 Media gateways 110 Load balancers 67 Proxies 66 VPN gateways 45 WAN Optimizers 44 Voice gateways 11 Total Middleboxes 636 Total routers ~900 APLOMB (SIGCOMM’ 13) 4

Firewalls • Network-level firewalls: • Limit access to the network • Installed at perimeter

Firewalls • Network-level firewalls: • Limit access to the network • Installed at perimeter of the network • Allows traffic specified in the policy • Drops everything else Internal Network Firewall Internet 5

Typical Firewall Configuration • Internal hosts can access DMZ and Internet • External hosts

Typical Firewall Configuration • Internal hosts can access DMZ and Internet • External hosts can access DMZ only, not Intranet • DMZ hosts can access Internet only DMZ X X • Advantages? • If a service gets compromised in DMZ it cannot affect internal hosts Intranet 6

Intrusion Detection Systems (IDS) • Firewalls allow traffic only to legitimate hosts and services

Intrusion Detection Systems (IDS) • Firewalls allow traffic only to legitimate hosts and services • Traffic to the legitimate hosts/services may contain attacks • Solution? • Intrusion Detection Systems • Monitor data and behavior • Report when identify attacks 7

Firewall vs. NIDS • Firewall • Active filtering • Fail-close Internet NIDS • Network

Firewall vs. NIDS • Firewall • Active filtering • Fail-close Internet NIDS • Network IDS • Passive monitoring • Fail-open FW Internet • Advantages and disadvantages? 8

IDS justification • Comments on IDS • “IDS easy to circumvent” • “Installing IDS

IDS justification • Comments on IDS • “IDS easy to circumvent” • “Installing IDS is useless” • “Impossible to design bulletproof IDS” • Responses • IDS is one component of a security system: Prevention, Detection & Recovery, Redundancy • Increases difficulty of successful attack: “Raising the bar” • If system defends against 95% of attackers, we can concentrate on the remaining 5% 9

Evolution of the IDS • Notion that audit/log data might help to discover abuse/misuse

Evolution of the IDS • Notion that audit/log data might help to discover abuse/misuse dates to early 1970 s • Framework for analysis established by Jim Anderson in 1980 in “Computer Security Threat Monitoring and Surveillance” http: //seclab. cs. ucdavis. edu/projects/history/CD/ande 80. pdf • Computers can generate lots of log/audit data • Early analysis techniques were based on analysis of this audit data, and used a mix of statistical and AI based techniques 10

Evolution of the IDS • By the late 1980 s, networked computers were increasingly

Evolution of the IDS • By the late 1980 s, networked computers were increasingly common and many intrusions involved external access via a network • The Network Security Monitor developed at the University of California Davis was the first IDS to work directly with network data as opposed to log data • Essentially a packet sniffer feeding data to an analysis engine 11

Evolution of the IDS • Today, the vast majority of IDS systems are based

Evolution of the IDS • Today, the vast majority of IDS systems are based on the analysis of network traffic. • There are numerous commercial offerings as well as a number of research systems and systems offered by the open source community. • Increasingly, the administration of IDS systems is integrated into system and network management frameworks. 12

Types of IDS Signaturebased Anomalybased Host-based Network-based 13

Types of IDS Signaturebased Anomalybased Host-based Network-based 13

Signature-based IDS • Characteristics: • Uses known pattern matching to signify attack • Can

Signature-based IDS • Characteristics: • Uses known pattern matching to signify attack • Can identify intrusions from packet header/data • May use Boolean operators in rule set • ‘this_string’ • ‘this_variable’ AND ‘that_number’ • ‘this_string’ AND ‘that_variable’ NOT ‘that_tcp_flag’ 14

Signature-based IDS • Advantages • Widely available • Easy to implement • Easy to

Signature-based IDS • Advantages • Widely available • Easy to implement • Easy to update • Example? 15

Signature-based IDS • Disadvantages • Cannot detect attacks for which it has no signature

Signature-based IDS • Disadvantages • Cannot detect attacks for which it has no signature • Must be updated for each new attack and attack variant • Lag time from new exploit to update can be dangerous • ‘New’ attack variant can be created by changing a single string • May be resource intensive 16

Anomaly-based IDS • Characteristics • Uses statistical models or a machine learning engine to

Anomaly-based IDS • Characteristics • Uses statistical models or a machine learning engine to characterize normal usage behaviors • Recognizes departures from normal as potential intrusions 17

Anomaly-based IDS • Advantages • Can detect attempts to exploit new and unforeseen vulnerabilities

Anomaly-based IDS • Advantages • Can detect attempts to exploit new and unforeseen vulnerabilities • Can recognize unusual traffic based on a number of characteristics: • Payload • Source address • Time • Can recognize authorized usage that falls outside the normal pattern 18

Anomaly-based IDS • Disadvantages • Generally slower, more resource intensive compared to signature-based IDS

Anomaly-based IDS • Disadvantages • Generally slower, more resource intensive compared to signature-based IDS • Greater complexity, difficult to configure • Higher percentages of false alerts • Link between abnormal and intrusive may be weak 19

Anomaly-based IDS • Typical deployment environments • Currently few typical deployments • Anomaly-based IDS

Anomaly-based IDS • Typical deployment environments • Currently few typical deployments • Anomaly-based IDS considered immature, too error-prone for widespread use • Example: • DDo. S area 20

Host-based IDS • Characteristics: • Runs on a single host • Can analyze audit-trails,

Host-based IDS • Characteristics: • Runs on a single host • Can analyze audit-trails, logs, integrity of files and directories, etc. • Tripwire, Fcheck • Real. Secure® Server Sensor • Research systems such as Emerald • May report to central administrative console 21

Host-based IDS • Advantages: • Relatively easy to deploy and to manage • Only

Host-based IDS • Advantages: • Relatively easy to deploy and to manage • Only one machine is involved • May require only one administrator • Creates single source of log and alert information • Generally not resource intensive - in most cases • Often will not require CPU, memory, etc. beyond what is needed for OS and applications 22

Host-based IDS • Disadvantages: • Works well for a single machine; extremely labor-intensive to

Host-based IDS • Disadvantages: • Works well for a single machine; extremely labor-intensive to monitor multiple machines each running a host-based IDS • If the host is compromised, the IDS may cease to function and thus no more alerts will be generated 23

Host-based IDS • Typical deployment environments: • A single mission-critical machine • User’s desktop

Host-based IDS • Typical deployment environments: • A single mission-critical machine • User’s desktop machine 24

Network-based IDS • Characteristics: • Network monitor • Passively captures traffic and inspects it

Network-based IDS • Characteristics: • Network monitor • Passively captures traffic and inspects it • Can also function in a client-server model • Sensors are located on multiple machines across the network • All sensors feed data to console • Console machine handles logging and alerting 25

Network-based IDS • Advantages • Positioned properly, can test effectiveness of firewalls, router access

Network-based IDS • Advantages • Positioned properly, can test effectiveness of firewalls, router access lists, etc. • Can monitor multiple machines from one physical and logical location • Console can generate an alert if a monitored machine/network has ceased to send information • Operator can see patterns in traffic • Amount • Type 26

Network-based IDS • Disadvantages: • Since it is capturing all network packets, can produce

Network-based IDS • Disadvantages: • Since it is capturing all network packets, can produce large log/alert files • Can be difficult to cull through vast amount of information • Console machine generally must be quite powerful, similar to a workgroup server • If console machine goes down then multiple machines may be left unmonitored • Communication from sensors to console may increase overall network traffic levels 27

NIDS • Background • Bro: A NIDS • NIDS Traffic Normalization • Honeycomb: NIDS

NIDS • Background • Bro: A NIDS • NIDS Traffic Normalization • Honeycomb: NIDS signature generation 28

Bro: Detecting intruders in real-time • Bro is a standalone NIDS developed by Vern

Bro: Detecting intruders in real-time • Bro is a standalone NIDS developed by Vern Paxson • Designed to keep LBL an open environment (to resist the need to install a firewall) • Goals • • • High-speed monitoring, no packet drops Real-time notification Separate mechanism from policy Extensibility Simple to use, guard against mistakes Tolerate attacks on NIDS • More powerful than Snort, but less popular. Why? • Misuse detection (signature-based) or anomaly detection (specification-based or statistical-based)? Paxon, 1998 29

Bro system architecture Policy script Alerts/notifications Policy Script Interpreter Event control Complexity of operations

Bro system architecture Policy script Alerts/notifications Policy Script Interpreter Event control Complexity of operations increases Event stream Event Engine tcpdump filters Volume of data decreases Filtered packet stream libpcap Packet stream Network 30

libpcap Layer • Only passes relevant packets to Event Engine • Uses BSD Packet

libpcap Layer • Only passes relevant packets to Event Engine • Uses BSD Packet Filter (BPF) to efficiently filter packets • Filter rules • tcp port finger or tcp port ftp or tcp port telnet or port 111 or tcp[13] & 7 != 0 31

Reminder: TCP header format 32

Reminder: TCP header format 32

Event engine • State for each connection, based on <Src. IP, Src. Port, Dst.

Event engine • State for each connection, based on <Src. IP, Src. Port, Dst. IP, Dst. Port> • If state not present, allocate fresh state • TCP processing • Update state based on SYN/FIN/RST flags • Process acknowledgment • SYN generates a timer event, if nothing happens after 5 min, generate connection_attempt event • UDP processing • Initial packets generate udp_request and udp_reply events 33

Policy script interpreter • Clear separation of event generation and response to achieve clear

Policy script interpreter • Clear separation of event generation and response to achieve clear separation between mechanism and policy • Advantage: extensibility (adding a new protocol analyzer and new event handler usually separate from other components) • Event are stored in a FIFO queue and processed sequentially • Policy script interpreter executes event handlers 34

Bro policy scripts • Goal: A clear and error-free language • Written in a

Bro policy scripts • Goal: A clear and error-free language • Written in a specialized language: • • • Network types (IP addresses, connections, protocol, etc. ) Typed constants, variables Network operators (comparison, ranges, etc. ) Control statements (IF/THEN, etc. ) Regular expressions • It can • Generate alerts • Call exterior programs 35

Bro policy scripts: variables and operators if ([H, S] in allowed_services) … it’s okay

Bro policy scripts: variables and operators if ([H, S] in allowed_services) … it’s okay … 36

Bro policy scripts: Statements, functions, and events 37

Bro policy scripts: Statements, functions, and events 37

Offline Analysis • Checkpointing in order to • Reclaim memory of dormant connections •

Offline Analysis • Checkpointing in order to • Reclaim memory of dormant connections • Offline-analysis • Logs maintained for a long time for • Forensics on past break-ins • Complex analysis that would be too expensive to be real -time 38

Attacks on Bro • Overload attack • Send packets that match filters • Send

Attacks on Bro • Overload attack • Send packets that match filters • Send packet streams that generate events • Try to generate events that lead to recording to disk • Defense strategy 1: Assume policy script is secret. • Good assumption? • Defense strategy 2: Lower the load (e. g. , stop capturing HTTP traffic) • Effective if attacker does not know how Bro lowers load 39

Attacks on Bro monitor • Crash attack • Find packet sequence that crashes monitor

Attacks on Bro monitor • Crash attack • Find packet sequence that crashes monitor • Exhaust memory, disk resources • Defense strategy 1: Careful code analysis • Defense strategy 2: OS-level watchdog timer with subsequent packet capturing (sacrificing real-time detection) 40

Attacks on Bro monitor • Subterfuge attack • Mislead Bro: find traffic patterns that

Attacks on Bro monitor • Subterfuge attack • Mislead Bro: find traffic patterns that Bro and the end system interpret differently • Example: Carefully setting TTL field • Defense strategy: Traffic normalization 41

Application-specific processing • Case studies: • • Finger FTP Portmapper Telnet • FTP Example:

Application-specific processing • Case studies: • • Finger FTP Portmapper Telnet • FTP Example: • Match username against sensitive IDs • Check access to sensitive files 42

NIDS • Background • Bro: A NIDS • NIDS Traffic Normalization • Honeycomb: NIDS

NIDS • Background • Bro: A NIDS • NIDS Traffic Normalization • Honeycomb: NIDS signature generation 43

NIDS: Evasion and Normalization • Problems • NIDS only has partial knowledge of what

NIDS: Evasion and Normalization • Problems • NIDS only has partial knowledge of what traffic the host sees (e. g. , TTL expires, MTU) • Ambiguities in TCP/IP (e. g. , Overlapping IP & TCP fragments) • Different OS implement standard differently • Approach: traffic normalization Handely et al. , 2001 44

Small TTL attack NIDS sees: A T T I A C K NIDS Host

Small TTL attack NIDS sees: A T T I A C K NIDS Host Internet End-host sees: Attacker’s data stream A T T I A C K A T T A C K same TCP seq #, “I” has short TTL 45

Fragmentation overlap attack NIDS sees: A T T A I C K NIDS Internet

Fragmentation overlap attack NIDS sees: A T T A I C K NIDS Internet Host End-host sees: Attacker’s data stream A T T A I C K same TCP seq # or same IP frag offset 46

Approach: traffic normalizer • Introduce “bump in the wire”: traffic normalizer to evade protocol

Approach: traffic normalizer • Introduce “bump in the wire”: traffic normalizer to evade protocol ambiguities NIDS Internet Normalizer Host 47

Alternative approaches • Host-based IDS • Loses the advantages of monitoring the entire site

Alternative approaches • Host-based IDS • Loses the advantages of monitoring the entire site cheaply • Major deployment and management efforts • Detailed Intranet map • Required knowledge of every OS and app • Bifurcating analysis • If the NIDS does not know which of the two interpretations the end system may apply to an input packet, split the analysis context • State explosion? • Aside: New opportunities given SDN/NFV? 48

Normalization tradeoffs • Normalization vs. protection • Off-load firewall/NIDS functionality • End-to-end semantics •

Normalization tradeoffs • Normalization vs. protection • Off-load firewall/NIDS functionality • End-to-end semantics • Drop overlapping IP/TCP fragments • Increase TTL in packets with low TTL • What applications does this break? • Keeping state • “Fail closed” possible, given sufficient state: • Terminate dormant connections first • Do. S through state exhaustion is a challenge • Inbound vs. outbound traffic • Assumes one side of the connection is trusted 49

Real-world considerations • Cold start • Normalizer will reboot periodically • Normalizer lacks knowledge

Real-world considerations • Cold start • Normalizer will reboot periodically • Normalizer lacks knowledge of ongoing connections • Need to carefully design normalizer not to disturb established connections 50

Real-world considerations • Attacks against normalizer • State exhaustion attacks • Note: occasionally dropping

Real-world considerations • Attacks against normalizer • State exhaustion attacks • Note: occasionally dropping a packet is ok, IP is unreliable • SYN flooding, ACK flooding (allocate state to deal with cold start), unacknowledged data flooding • Defense strategy: • Monitor system resources to determine whether normalizer is under attack • Instantiate state only if traffic seen from an internal host 51

Identifying normalizations • Methodology: Walk through each header field 1. Identify fields that may

Identifying normalizations • Methodology: Walk through each header field 1. Identify fields that may cause packet to be dropped/interpreted differently by router or end-host 2. Find ways to normalize it 3. Analyze impact on protocols 52

Identifying normalizations Example: TTL (Time-to-live) • Solution 1: NIDS could measure the number of

Identifying normalizations Example: TTL (Time-to-live) • Solution 1: NIDS could measure the number of hops to every end-host • Con: Large number of states, change of routing, … • Solution 2: NIDS may see ICMP time-exceeded-intransit packets • Con: ICMP messages may be rate limited or filtered 53

Identifying normalizations Example: TTL (Time-to-live) • Solution 3: • Configure the normalizer with the

Identifying normalizations Example: TTL (Time-to-live) • Solution 3: • Configure the normalizer with the longest path across the internal site. • Rewrite the TTL field of incoming packets to the above value • Cons: • What if a routing loop passes through the normalizer? • Breaking semantics of traceroute 54

Identifying normalizations • IP identification field is used for uniquely identifying the group of

Identifying normalizations • IP identification field is used for uniquely identifying the group of fragments of a single IP datagram. 55

Identifying normalizations: IP Identification • IP ID field used for stealth port scanning •

Identifying normalizations: IP Identification • IP ID field used for stealth port scanning • Permute IP ID deterministically prevents internal hosts from being misused as patsies • Reliable RST protects internal hosts from being victims 56

NIDS • Background • Bro: A NIDS • NIDS Traffic Normalization • Honeycomb: NIDS

NIDS • Background • Bro: A NIDS • NIDS Traffic Normalization • Honeycomb: NIDS signature generation 60

Honeycomb: Motivation • NIDS work based on signatures • How to generate signatures to

Honeycomb: Motivation • NIDS work based on signatures • How to generate signatures to begin with? • Common practice is manual and expertisebased • Can we do better? Kreibich and Corwcroft, 2004 61

Honeycomb: Background • Good NIDS signatures should be • Narrow enough; otherwise, high false

Honeycomb: Background • Good NIDS signatures should be • Narrow enough; otherwise, high false positives • Flexible enough; otherwise, high false negatives • Honeypots: decoy computer resources to detect or counteract computer resources • Examples: dummy database items, dummy web servers, … • Key Idea behind Honeycomb: • Traffic sent to a honeypot should be malicious • We can extract its pattern and use it as a NIDS signature 62

Honeycomb: Architecture • Good NIDS signatures: 63

Honeycomb: Architecture • Good NIDS signatures: 63

Honeycomb: Per-packet workflow 64

Honeycomb: Per-packet workflow 64

Honeycomb - Signature Creation • If there is any existing connection state for the

Honeycomb - Signature Creation • If there is any existing connection state for the new packet, that state is updated, otherwise new state is created. • If the packet is outbound, processing stops here. • Honeycomb performs protocol analysis at the network and transport layer. 65

Honeycomb – Connection Tracking • Signature creation is based on comparing new data to

Honeycomb – Connection Tracking • Signature creation is based on comparing new data to old data therefore connections and packets must be maintained for a period of time • Handshake and established connections are kept separate as not to fill up the hashtables. 66

Honeycomb – Connection Tracking • Goal: Find attack signatures by • finding deviation from

Honeycomb – Connection Tracking • Goal: Find attack signatures by • finding deviation from protocols • finding inbound traffic patterns 67

Honeycomb: Pattern matching in flow content • Use a longest common substring (LCS) algorithm

Honeycomb: Pattern matching in flow content • Use a longest common substring (LCS) algorithm to spot similarities in packet payloads 68

Honeycomb: Pattern matching in flow content • Horizontal pattern detection: happens every nth message

Honeycomb: Pattern matching in flow content • Horizontal pattern detection: happens every nth message and applies LCS algorithm 69

Honeycomb: Pattern matching in flow content • Vertical detection: concatenates messages then applies LCS

Honeycomb: Pattern matching in flow content • Vertical detection: concatenates messages then applies LCS algorithm • Advantage? 70

Honeycomb – Signature generation • Signatures are indefinite and can be built upon if

Honeycomb – Signature generation • Signatures are indefinite and can be built upon if they are improved • Signature aggregation • Example: An HTTP GET substring in attack flow • Signatures are output in Bro and Snort-like signatures 71

Honeycomb - Results Net. Bios web MSSQL (Slammer worm) Distribution of TCP and UDP

Honeycomb - Results Net. Bios web MSSQL (Slammer worm) Distribution of TCP and UDP traffic destination ports in packets directed at the honeypot. 72

Honeycomb - Results Signature Honeycomb created for the Slammer Worm 73

Honeycomb - Results Signature Honeycomb created for the Slammer Worm 73

Honeycomb - Results Signature Honeycomb created for the Code. Red II Worm 74

Honeycomb - Results Signature Honeycomb created for the Code. Red II Worm 74

Summary • Network intrusion detection systems is a complement to a firewall • Goal:

Summary • Network intrusion detection systems is a complement to a firewall • Goal: Finding malicious traffic both low false positives and false negatives • Decoupling policy from analysis is important • Protocols are ambiguous • Unclear how end-hosts implement the ambiguous parts • Can be used to evade NIDS • Possible solution: Protocol normalization (a. k. a. protocol scrubbing) • Opportunities for automatic signature generation 75

Next lecture • Privacy • Required reading: • Balancing Accountability and Privacy in the

Next lecture • Privacy • Required reading: • Balancing Accountability and Privacy in the Network • Skim: • Tor: The Second-Generation Onion Router • Optional: • Infranet: Circumventing Web Censorship and Surveillance 76