Intrusion Detection Outline p Intrusion detection and computer

  • Slides: 55
Download presentation
Intrusion Detection

Intrusion Detection

Outline p Intrusion detection and computer security p Current p Data intrusion detection approaches

Outline p Intrusion detection and computer security p Current p Data intrusion detection approaches Mining Approaches for Intrusion Detection p Summary

Intrusion Detection and Computer Security p Computer security goals: n p Confidentiality, integrity, and

Intrusion Detection and Computer Security p Computer security goals: n p Confidentiality, integrity, and availability Intrusion is a set of actions aimed to compromise these security goals p Intrusion prevention (authentication, encryption, etc. ) alone is not sufficient p Intrusion detection is needed

Intrusion Examples p Intrusions: Any set of actions that threaten the integrity, availability, or

Intrusion Examples p Intrusions: Any set of actions that threaten the integrity, availability, or confidentiality of a network resource p Examples n Denial of service (Do. S): attempts to starve a host of resources needed to function correctly n Scan: reconnaissance on the network or a particular host n Worms and viruses: replicating on other hosts n Compromises: obtain privileged access to a host by known vulnerabilities

Intrusion Detection p Intrusion detection: The process of monitoring and analyzing the events occurring

Intrusion Detection p Intrusion detection: The process of monitoring and analyzing the events occurring in a computer and/or network system in order to detect signs of security problems p Primary assumption: User and program activities can be monitored and modeled p Steps n Monitoring and analyzing traffic n Identifying abnormal activities n Assessing severity and raising alarm

Monitoring and Analyzing Traffic p TCPdump and Windump n p Provide insight into the

Monitoring and Analyzing Traffic p TCPdump and Windump n p Provide insight into the traffic activity on a network p ftp: //ftp. ee. lbl. gov/tcpdump. tar. Z p http: //netgroupserv. polito. it/windump Ethereal n GUI to interpret all layers of the packet

Goals of Intrusion Detection System (IDS) p p Detect wide variety of intrusions n

Goals of Intrusion Detection System (IDS) p p Detect wide variety of intrusions n Previously known and unknown attacks n Suggests need to learn/adapt to new attacks or changes in behavior Detect intrusions in timely fashion n May need to be real-time, especially when system responds to intrusion p n Problem: analyzing commands may impact response time of system May suffice to report intrusion occurred a few minutes or hours ago

Goals of Intrusion Detect. System (IDS) (2) p Present analysis in simple, easy-to-understand format

Goals of Intrusion Detect. System (IDS) (2) p Present analysis in simple, easy-to-understand format p Be accurate n Minimize false positives, false negatives p False positive: An event, incorrectly identified by the IDS as being an intrusion when none has occurred p False negative: An event that the IDS fails to identify as an intrusion when one has in fact occurred n Minimize time spent verifying attacks, looking for them

IDS Architecture p Sensors (agent) n p Analyzers (detector) n n p to collect

IDS Architecture p Sensors (agent) n p Analyzers (detector) n n p to collect data and forward info to the analyzer p network packets p log files p system call traces To receive input from one or more sensors or from other analyzers To determine if an intrusion has occurred User interface n To enable a user to view output from the system or control the behavior of the system

IDS Architecture

IDS Architecture

Signature-Based Intrusion Detection p Human analysts investigate suspicious traffic p Extract signatures n Features

Signature-Based Intrusion Detection p Human analysts investigate suspicious traffic p Extract signatures n Features of known intrusions p Use pre-defined signatures to discover malicious packets p Examples n La. Brea Tarpit by Tom Liston n Snort and Snort rules Marty Roesch

Snort by Marty Roesch p p An open source free network intrusion detection system

Snort by Marty Roesch p p An open source free network intrusion detection system n Signature-based, use a combination of rules and preprocessors n On many platforms, including UNIX and Windows n www. snort. org Preprocessors n IP defragmentation, port-scan detection, web traffic normalization, TCP stream reassembly, … n Can analyze streams, not only a single packet at a time

Problems in Signature-Based Intrusion Detection Systems p Many false positives: prone to generating alerts

Problems in Signature-Based Intrusion Detection Systems p Many false positives: prone to generating alerts when there is no problem in fact n Signatures are not specific enough n A packet is not examined in context with those that precede it or those that follow p Cannot detect unknown intrusions n Rely on signatures extracted by human experts

Misuse vs. Anomaly Detection p Misuse detection: use patterns of well-known attacks to identify

Misuse vs. Anomaly Detection p Misuse detection: use patterns of well-known attacks to identify intrusions p n Classification based on known intrusions n E. g. , three consecutive login failures: password guessing. Anomaly detection: use deviation from normal usage patterns to identify intrusions n Any significant deviations from the expected behavior are reported as possible attacks

Misuse vs. Anomaly Detection Misuse Detection Anomaly Detection Definition matching the sequence of “signature

Misuse vs. Anomaly Detection Misuse Detection Anomaly Detection Definition matching the sequence of “signature actions” of known intrusion scenarios using statistical measure on system features Shortcoming p Example STAT [HLMS 90] Has to handp Rely upon in coded known selecting the system pattern. features. p Unable to detect p Has to study any future intrusion sequential interrelation between transactions IDES [LTG+92]

Host-based vs. Network-based p According to data sources p Host-based detection: the data is

Host-based vs. Network-based p According to data sources p Host-based detection: the data is collected from an individual host p n Directly monitor the host data files and OS processes n Can determine exactly which host resources are the targets of a particular attack Network-based detection: the data is traffic across the network n A set of traffic sensors within the network n Can easily harder against attacks and hide from the attackers

OUTLINE p Intrusion detection and computer security p Current intrusion detection approaches p Data

OUTLINE p Intrusion detection and computer security p Current intrusion detection approaches p Data Mining Approaches for Intrusion Detection p Summary

Current Intrusion Detection Approaches—Misuse Detection p Misuse detection : n Record the specific patterns

Current Intrusion Detection Approaches—Misuse Detection p Misuse detection : n Record the specific patterns of intrusions n Monitor current audit trails (event sequences) and pattern matching n Report the matched events as intrusions n Representation models: expert rules, Colored Petri Net, and state transition diagrams, etc.

Misuse Detection Example p Expert systems: use a set of rules to describe attacks

Misuse Detection Example p Expert systems: use a set of rules to describe attacks n p Signature analysis: capture features of attacks in audit trail n p Haystack, Net. Ranger, Real. Secure, Mu. Sig State-transition analysis: use state-transition diagrams n p IDES, Computer. Watch, NIDX, P-BEST, ISOA STAT, USTAT and Net. STAT Other approaches n n Colored petri nets, e. g. , IDIOT Case-based reasoning, e. g. , AUTOGUARD

Current Intrusion Detection Approaches—Anomaly Detection p Anomaly detection: n Establishing the normal behavior profiles

Current Intrusion Detection Approaches—Anomaly Detection p Anomaly detection: n Establishing the normal behavior profiles n Observing and comparing current activities with the (normal) profiles n Reporting significant deviations as intrusions n Statistical measures as behavior profiles: ordinal and categorical (binary and linear)

Anomaly Detection Example p Statistical methods: multivariate, temporal analysis n p IDES, NIDES, EMERALD

Anomaly Detection Example p Statistical methods: multivariate, temporal analysis n p IDES, NIDES, EMERALD Expert systems n Computer. Watch, Wisdom & Sense

Problems of Current Intrusion Detection Approaches p Main problems: manual and ad-hoc n Misuse

Problems of Current Intrusion Detection Approaches p Main problems: manual and ad-hoc n Misuse detection: p p n Known intrusion patterns have to be hand-coded Unable to detect any new intrusions (that have no matched patterns recorded in the system) Anomaly detection: p p Selecting the right set of system features to be measured is ad hoc and based on experience Unable to capture sequential interrelation between events

OUTLINE Intrusion detection and computer security p Current intrusion detection approaches p Data Mining

OUTLINE Intrusion detection and computer security p Current intrusion detection approaches p Data Mining Approaches for Intrusion Detection p Summary p

Why Can Data Mining Help? p Data mining: applying specific algorithms to extract patterns

Why Can Data Mining Help? p Data mining: applying specific algorithms to extract patterns from data p Normal and intrusive activities leave evidence in audit data p From the data-centric point view, intrusion detection is a data analysis process

Why Can Data Mining Help? Successful applications in related domains, e. g. , fraud

Why Can Data Mining Help? Successful applications in related domains, e. g. , fraud detection, fault/alarm management p Learn from traffic data p n n p Supervised learning: learn precise models from past intrusions Unsupervised learning: identify suspicious activities Maintain or update models on dynamic data

Frequent Patterns p Patterns that occur frequently in a database p Mining Frequent patterns

Frequent Patterns p Patterns that occur frequently in a database p Mining Frequent patterns – finding regularities p Process of Mining Frequent patterns for intrusion detection n Phase I: mine a repository of normal frequent itemsets for attack-free data n Phase II: find frequent itemsets in the last n connections and compare the patterns to the normal profile

Frequent Pattern Mining in MINDS p MINDS: a IDS using data mining techniques n

Frequent Pattern Mining in MINDS p MINDS: a IDS using data mining techniques n p University of Minnesota Summarizing attacks using association rules n {Src IP=206. 163. 27. 95, Dest Port=139, Bytes [150, 200)} {ATTACK}

Patterns About Alerts Ning et al. CCS’ 02 p Find correlated alerts – the

Patterns About Alerts Ning et al. CCS’ 02 p Find correlated alerts – the frequent patterns of alerts p n n p Attack scenarios – the logical connections between alerts A hyper-alerts correlation graph approach Use the correlation of intrusion alerts to identify high level attacks

Associate rules p Used for link analysis p E. g. : n If the

Associate rules p Used for link analysis p E. g. : n If the number of failed login attempts (num_failed_login_attempts) and the network service on the destination (service) are features, an example of rule is: n num_failed_login_attempts = 6, service = FTP => attack = Do. S [1, 0. 28 ]

Sequential Pattern Analysis p Models sequence patterns p (Temporal) order is important in many

Sequential Pattern Analysis p Models sequence patterns p (Temporal) order is important in many situations p n Time-series databases and sequence databases n Frequent patterns (frequent) sequential patterns Sequential patterns for intrusion detection n Capture the signatures for attacks in a series of packets

Classification: A Two-Step Process p Model construction: describe a set of predetermined classes n

Classification: A Two-Step Process p Model construction: describe a set of predetermined classes n Training dataset: tuples for model construction p Each tuple/sample belongs to a predefined class Classification rules, decision trees, or math formulae Model application: classify unseen objects n Estimate accuracy of the model using an independent test set n Acceptable accuracy apply the model to classify data tuples with unknown class labels

Classification Methods Basic Algorithm ID 3 p Neural networks p Bayesian classification p n

Classification Methods Basic Algorithm ID 3 p Neural networks p Bayesian classification p n n p Naïve Bayesian classification Bayesian belief network Support vector machines

Classification for Intrusion Detection p Misuse detection n p Classification based on known intrusions

Classification for Intrusion Detection p Misuse detection n p Classification based on known intrusions Example: Sinclair et al. “An application of machine learning to network intrusion detection” n Use decision trees and ID 3 on host session data n Use genetic algorithms to generate rules p If <pattern> then <alert>

HIDE p p “A hierarchical network intrusion detection system using statistical processing and neural

HIDE p p “A hierarchical network intrusion detection system using statistical processing and neural network classification” by Zheng et al. Five major components n n n Probes collect traffic data Event preprocessor preprocesses traffic data and feeds the statistical model Statistical processor maintains a model for normal activities and generates vectors for new events Neural network classifies the vectors of new events Post processor generates reports

Intrusion Detection by NN and SVM p S. Mukkamala et al. , IEEE IJCNN

Intrusion Detection by NN and SVM p S. Mukkamala et al. , IEEE IJCNN May 2002 p Discover useful patterns or features that describe user behavior on a system p Use the set of relevant features to build classifiers p SVMs have great potential to be used in place of NNs due to its scalability and faster training and running time p NNs are especially suited for multi-category classification

Clustering p Group data into clusters p What is a good clustering n High

Clustering p Group data into clusters p What is a good clustering n High intra-class similarity and low inter-class similarity p n p Depending on the similarity measure The ability to discover some or all of the hidden patterns Clustering Approaches n K-means n Hierarchical Clustering n Density-based methods n Grid-based methods n Model-based

Clustering for Intrusion Detection p Anomaly detection n Any significant deviations from the expected

Clustering for Intrusion Detection p Anomaly detection n Any significant deviations from the expected behavior are reported as possible attacks p Build clusters as models for normal activities p “A scalable clustering for intrusion signature recognition” by Ye and Li n Use description of clusters as signatures of intrusions

Alert Correlation p F. Cuppens and A. Miege, in IEEE S&P’ 02 p Use

Alert Correlation p F. Cuppens and A. Miege, in IEEE S&P’ 02 p Use clustering and merging functions to recognize alerts that correspond to the same occurrence of an attack n Create a new alert that merge data contained in these various alerts p Generate global and synthetic alerts to reduce the number of alerts further

Mining Data Streams p Continuous arrival data in multiple, rapid, timevarying, possibly unpredictable and

Mining Data Streams p Continuous arrival data in multiple, rapid, timevarying, possibly unpredictable and unbounded streams p Many applications n Financial applications, network monitoring, security, telecommunications data management, web application, manufacturing, sensor networks, etc.

Mining Data Streams for Intrusion Detection p Maintaining profiles of normal activities n p

Mining Data Streams for Intrusion Detection p Maintaining profiles of normal activities n p The profiles of normal activities may drift Identifying novel attacks n Identifying clusters and outliers in traffic data streams

A Systematic Framework—J. Stolfo et al. p Build good models: n p Build better

A Systematic Framework—J. Stolfo et al. p Build good models: n p Build better models: n p select appropriate features of audit data to build intrusion detection models architect a hierarchical detector system that combines multiple detection models Build updated models: n dynamically update and deploy new detection system as needed

A Systematic Framework p Support for the feature selection and model construction: n Apply

A Systematic Framework p Support for the feature selection and model construction: n Apply data mining algorithms to find consistent interand intra- audit record (event) patterns n Use the features and time windows in the discovered patterns to build detection models n A support environment to semi-automate this process

A Systematic Framework p Combining multiple detection models: n n n p Each (base)

A Systematic Framework p Combining multiple detection models: n n n p Each (base) detector model monitors one aspect of the system They can employ different techniques and be independent of each other The learned (meta) detector combines evidence from a number of base detectors An intelligent agent-based architecture: n n learning agents: continuously compute (learn) the detection models detection agents: use the (updated) models to detect intrusions

A Systematic Framework

A Systematic Framework

Building Classifiers for Intrusion Detection—J. Stolfo et al. p Experiments in constructing classification models

Building Classifiers for Intrusion Detection—J. Stolfo et al. p Experiments in constructing classification models for anomaly detection p Two experiments: p n sendmail system call data n network tcpdump data Use meta classifier to combine multiple classification models

Classification Models on sendmail p The data: sequence of system calls made by sendmail.

Classification Models on sendmail p The data: sequence of system calls made by sendmail. p Classification models (rules): describe the “normal” patterns of the system call sequences. p The rule set is the normal profile of sendmail p Detection: calculate the deviation from the profile n large number/high scores of “violations” to the rules in a new trace suggests an exploit

Classification Models on sendmail p The sendmail data: n n n Each trace has

Classification Models on sendmail p The sendmail data: n n n Each trace has two columns: the process ids and the system call numbers Normal traces: sendmail and sendmail daemon Abnormal traces: sunsendmailcap, syslogremote, syslog-remote, decode, sm 5 x and sm 56 a attacks

Classification Models on sendmail p Lessons learned: n n Normal behavior can be established

Classification Models on sendmail p Lessons learned: n n Normal behavior can be established and used to detect anomalous usage Need to collect near “complete” normal data in order to build the “normal” model But how do we know when to stop collecting? Need tools to guide the audit data gathering process

Classification Models on tcpdump p The tcpdump data (part of a public data visualization

Classification Models on tcpdump p The tcpdump data (part of a public data visualization contest): n Packets of incoming, out-going, and internal broadcast traffic n One trace of normal network traffic n Three traces of network intrusions

Data Preprocessing p p p Extract the “connection” level features: n Record connection attempts

Data Preprocessing p p p Extract the “connection” level features: n Record connection attempts n Watch how connection is terminated Each record has: n start time and duration n participating hosts and ports (applications) n statistics (e. g. , # of bytes) n flag: normal or a connection/termination error n protocol: TCP or UDP Divide connections into 3 types: incoming, outgoing, and inter-lan

Building Classifier for Each Type of Connections p Use the destination service (port) as

Building Classifier for Each Type of Connections p Use the destination service (port) as the class label p Training data: 80% of the normal connections p Testing data: 20% of the normal connections and connections in the 3 intrusion traces p Apply RIPPER to learn rules

Lessons Learned p Data preprocessing requires extensive domain knowledge p Adding temporal features improves

Lessons Learned p Data preprocessing requires extensive domain knowledge p Adding temporal features improves classification accuracy p Need tools to guide (temporal) feature selection

Meta Classifier that Combines Evidence from Multiple Detection Models p Build base classifiers that

Meta Classifier that Combines Evidence from Multiple Detection Models p Build base classifiers that each model one aspect of the system p The meta learning task: n each record has a collection of evidence from base classifiers, and a class label “normal”or “abnormal” on the state of the system p Apply a learning algorithm to produce the meta classifier