Defence RD Canada Ottawa Technical Memorandum DRDC Ottawa

Defence R&D Canada - Ottawa Technical Memorandum DRDC Ottawa TM 2003 -139 November 2003 A Finite State Machine Model of TCP Connections in the Transport Layer J. Treurniet and J. H. Lefebvre Defence R&D Canada - Ottawa Mike Hsiao, 20080822

Outline l l Summary and Introduction Theory – – – l l Implementation of the Algorithm Results and Discussion – – l 2 Transmission Control Protocol TCP Finite State Machine Model Detection of Anomalies Network Management Security Research False Positives Conclusion and Future Work

Summary Background – RFC & implementation l The protocols employed on computer networks are intended to be well-defined via Request for Comments (RFC) documents. – 3 In reality, there is enough freedom in the RFC specifications of the protocols that they can vary widely from one implementation to the next. Additional information: http: //lxr. linux. no/

Summary Diagram of a “textbook” TCP connection Background – TCP l TCP is a reliable protocol, and its behavior follows a pattern that is somewhat predictable. – The “flags” convey the commands that carry a connection through these stages. SYN Three-way handshake SYN-ACK … Data exchange FIN Closing 4 FIN-ACK

Summary Background – FSM l The authors model a TCP connection as an FSM, – – 5 defining the states of a FSM to reflect the stages of a connection and using the flags as the events that bring about transitions among states. The TCP FSM proposed by the authors

Summary Background – Anomaly detection l To use the model for strict anomaly detection, a failure state is introduced to indicate the occurrence of a disallowed event or an attempted illegal transition. – When the time-ordered flags of a TCP connection is input to the FSM, l l – 6 if the connection enters a failure state or otherwise does not complete, State X the connection is flagged as anomalous. Event Y Failure State

Introduction Background l The TCP FSM can be used to detect some network management issues and network security events, and it can also be used as a research tool to study the behavior of TCP on the Internet. l � ”Strict anomaly detection� ” defines a set of permitted events and detects activity which represents exceptions to those events. – 7 TCP, FSM, anomalous

Introduction A snapshot of TCP anomalies l Impart information to – Network management l l – Network security l l 8 Unresponsive hosts Behaviors that lead to resource consumption Scanning activity The appeal of the model is in its simplicity and it utility in a wild range of applications

Theory Transmission Control Protocol 9 Note: Only certain flag combination are valid.

Theory Transmission Control Protocol In an ungraceful termination, the connection is torn down by transmission of a reset (R) from either the client or the server. Graceful Termination 10 On receiving an acceptable reset, if the receiving TCP is in 1) a non-synchronized state, => returns to LISTEN; 2) a synchronized states, => aborts the connection => informs its user.

Theory In Practice l 11 in most Berkeley-derived systems In practice, stages of a connection are only allowed to exist for a set amount of time, to free up resources that aren't being used. twice the max amount of time in RFC; some implement is one minute.

Theory TCP client and server 12 l The traditional state transition diagram for TCP is a graphical representation of two separate but related finite state machines, one for the client and one for the server. l The authors have simplified the traditional TCP state transition diagram to remove the differentiation of client and server.

Theory Server Client Attempt to close socket Receive close request Simultaneous close 13

They proposed a 7 -state TCP FSM. 14

Theory 6 events in the proposed TCP FSM (e. g. , SYN+FIN) 15 Such packets are either “crafted” or appear as a result of corruption of the packet.

Theory almost 10% of all SYN packets are retransmitted 16

Theory Detection of Anomalies 17 l If the final state of the sequence is not Closed (C 2), then the connection is flagged as anomalous. l For streams that terminate in an allowed state other than Closed, we denote the event as “timeout� ” to signify that the stream has ended but the connection has not been closed.

Theory Example of Anomalies l Denial of Service (Do. S) – – l TCP half-open scan – l 18 many H 2 | timeout events for one host X | timeout or C 1 | timeout H 1 | RAPU, H 1 | timeout, H 2 | RAPU or H 2 | timeout Scans for OS fingerprinting (e. g. , Que. OS)

Implementation of the Algorithm Traffic Traces l l l 19 Unfiltered traffic is collected at the external interface of 3 class B networks using tcpdump. Only the first 68 bytes of each packet were collected to obtain the full TCP header. At peak times, an hourly file contained on the order of 2 million packets.

Implementation of the Algorithm Implementation l l The TCP FSM algorithm was written in MATLAB. In practice, each packet was read sequentially from the raw traffic file. – – l 20 Each socket pair was assigned a stream holding the complete trace for that socket. How to deal with the truncation of the traffic data? (i. e. , streams that start or end in the middle of a connection) August 21, 2000 data for 24 hours starting at midnight.

Results and Discussion Analysis Results (1/3) 21 They do not proceed the normal synchronization process.

Results and Discussion Analysis Results (2/3) 22 They do not complete the synchronization process.

Results and Discussion Analysis Results (3/3) 23

Results and Discussion 1) Approximately 37% of the Internet TCP traffic does not obey the protocol. 2) If we remove “SYN scan” and “bad flag scan”, ↓ 24

Results and Discussion Network Management l Unresponsive and non-existant hosts – l Delays (i. e. , packets do not arrive in an appropriate time) – l H 1 | FAPU Backscatter effects – 25 X | timeout and C 1 | timeout Malfuncioning TCP – l L | {SA, APU, FAPU, RAPU}, H 2 | RAPU and X | timeout Abandoned connections – l H 1 | RAPU, H 1 | Timeout, H 2 | RAPU and H 2 | timeout L | SA or L | SA

Results and Discussion Security (1/2) l TCP connect and half-open scans – – – l Stealth scans and inverse mapping (firewall) – – – 26 Open port: {S-SA-APU-RAPU} or {S-SA-APU-FAPU-APU} Half open: H 2 | RAPU Closed port: H 1 | RAPU, H 1 | timeout SYN-FIN (e. g. , Firewalls will be configured to block incoming packets with only the SYN flag set) L | APU + RST or ICMP destination unreachable L | SA, L | FAPU, bad flag, L | RAPU

Results and Discussion Security (2/2) l Aggregating scanning events – L | {SA, APU, FAPU, RAPU}, H 1 | {RAPU, timeout} and H 2 | {FAPU, RAPU} or bad flag 7 3 l Role of the Firewall – – 27 H 2 | timeout H 2 | RAPU 2 1 1 m 10 m 1 h 1 5 h 12 h 1 1 d 2 d 5 d >5 d

Results and Discussion False Positives l By definition, all anomalies found using strict anomaly detection are anomalous and therefore there is no false positive rate to consider. – – 28 It is difficult to determine the intent of the user who sends a SYN and receives a RST in response (failure H 1 | RAPU). It has different meanings and possesses different importance to different applications.

Conclusions and Future Work (1/2) l l l 29 Examining packet follows on a network that do not conform to the protocol Successfully detect TCP-based scanning activity, delays, outages, misconfigurations and other unexpected TCP behaviour Correlation engines could be used to correlate the events found through application of this model. A successful full connect scan cannot be detected with this method because the flag pattern obeys the protocol specification. Include an option for TCP traffic that is not officially allowed but is nevertheless present on the Internet. IDS

Conclusions and Future Work (2/2) l l l 30 Using the proportionalities of anomalies found by implementing the TCP FSM, one can introduce failures using a Markov chain approach. As well as a mechanism for traffic generation, Markov models may provide the capability of assigning a probability that a connection is valid. This may be the basis of a statistical anomaly detection engine.

Comments l l The usage of a protocol can be considered as a distinctive signature to a program. Our work also include service level, and an application is more complicated than TCP. – – l l 31 More events (messages) States are unclear. Too much false positive for anomaly detection model. Perfect attacks may exist.

Further Readings l TCP – – l Apps – – – 32 Floyd, S. (2002). Inappropriate TCP Resets Considered Harmful. (work in progress). End-to-end Interest Maillist. Frequency of RST terminated connections. USC Information Sciences Institute. Sasha/Beetle (2000). A Strict Anomaly Detection Model for IDS. Phrack, 10(56). Das, K. (2002). Protocol Anomaly Detection for Networkbased Intrusion Detection. SANS Institute. Lemonnier, E. (2001). Protocol Anomaly Detection in Network-based IDSs.