Python Networking by Chris Seddon Copyright 2000 13

Advanced Python 1. 2. 3. 4. 5. Networking Sockets Socket. Server Secure Sockets Other

1

Networking stream sockets datagram sockets web browsers web servers 1 Copyright © 2000 -13

Hostnames and services /etc/hosts host name Internet number /etc/services application port number /etc/inetd. conf

/etc/hosts internet number 127. 1 192. 19. 140. 200 192. 19. 140. 202 192. 19. 140. 203 192. 19. 140. 207 Copyright © 2000 -13 CRS Enterprises Ltd official name aliases localhost loopback blue red orange indigo 8

/etc/services service echo netstat ftp telnet hostnames who Copyright © 2000 -13 CRS Enterprises

Sockets for IPC 192. 56. 40. 2 192. 56. 40. 7 sockets P 1

Socket Type Application socket types SOCK_DGRAM SOCK_STREAM datagram virtual circuit Diagnostic socket types SOCK_RAW raw socket SOCK_RDM reliably-delivered message SOCK_SEQPACKET sequenced packets Copyright © 2000 -13 CRS Enterprises Ltd 11

Communications Domain AF_BLUETOOTH AF_INET 6 AF_NETLINK AF_PACKET AF_TIPC AF_UNIX Copyright © 2000 -13 CRS Enterprises Ltd Bluetooth protocol IPv 4 protocols (TCP, UDP) IPv 6 protocols (TCP, UDP) Netlink Interprocess Communication Link-level packets Transparent Inter-Process Communication protocol UNIX domain protocols 12

Unix Domain SOCK_STREAM byte stream bi-directional like the telephone system SOCK_DGRAM message stream bi-directional like the mail system Socket names are inodes 108 character path name Copyright © 2000 -13 CRS Enterprises Ltd 13

Internet Domain SOCK_STREAM TCP/IP byte stream bi-directional built-in error checking SOCK_DGRAM UDP/IP message stream bi-directional no error checking Socket names are 48 bit values 32 bit Internet address 16 bit port number Copyright © 2000 -13 CRS Enterprises Ltd 14

Internet Layered Model ftp TCP ftpd UDP TCP UDP IP IP ethernet protocol Ethernet

Internet Packets 48 bit 32 bit 16 bit 34. f. 6 e. 2. 1 d. a 2 193. 45. 6. 1 23 e. 2 d. 63. a. 12. b 2 195. 45. 2. 7 1945 Ethernet Address Internet Address Copyright © 2000 -13 CRS Enterprises Ltd data CRC Port Address 16

Internet Number Scheme N H H H Class A (0 -127) N N H

Socket Addresses UNIX Domain AF_UNIX “/tmp/mysocket” import socket my. Socket = socket(socket. AF_UNIX, socket. SOCK_STREAM) my. Socket. bind("/tmp/mysocket") Internet Domain AF_INET 7001 197. 46. 74. 6 import socket my. Socket = socket(socket. AF_INET, socket. SOCK_STREAM) my. Socket. bind( (196. 46. 74. 6, 7001) ) Copyright © 2000 -13 CRS Enterprises Ltd 18

Stream Sockets Server socket() Connection oriented bind() Client listen() socket() accept() read() write() close() shutdown() Copyright © 2000 -13 CRS Enterprises Ltd connection establishment connect() read() write() close() shutdown() 19

Types of Stream Socket Raw Sockets created by socket() must be converted to comms or listening socket Comms Sockets created by accept() / connect() used to transfer data cannot establish connections Listening Sockets converted by listen() used to establish connections cannot transfer data Copyright © 2000 -13 CRS Enterprises Ltd 20

Socket Layout in the Kernel Server Client LISTEN SOCKET accept() listen() connect() COMMS SOCKET

Datagram Sockets Server socket() bind() Connectionless Client socket() recvfrom() bind() block until a request is received sendto() process request sendto() Copyright © 2000 -13 CRS Enterprises Ltd recvfrom() 22

2

Sockets 2 Copyright © 2000 -13 CRS Enterprises Ltd 26

Stream Sockets - System Calls socket create kernel data structure bind attach to inode listen convert to listening socket accept create comms socket connect request to connect to comms socket read/write transfer data send/recv transfer priority data close/shutdown terminate connection select monitor socket for activity unlink remove named socket from kernel Copyright © 2000 -13 CRS Enterprises Ltd 27

Datagram Sockets - System Calls socket create kernel data structure bind attach to inode connect remember last address sendto/recfrom transfer data send/recv transfer data (used with connect()) close/shutdown terminate connection unlink remove named socket from kernel Copyright © 2000 -13 CRS Enterprises Ltd 28

socket() from socket import socket sd = socket(domain, type) AF_UNIX AF_INET AF_XNS SOCK_STREAM SOCK_DGRAM

bind() sd. bind(address) my. Socket. bind("/tmp/mysocket") AF_UNIX my. Socket. bind( (196. 46. 74. 6,

listen() and accept() max. no. of pending connection requests blocking call sd. listen(5) comms. Sd, (remote. Host, remote. Port) = sd. accept() new socket descriptor set by the call: address of socket requesting the connection Copyright © 2000 -13 CRS Enterprises Ltd 31

connect() sd. connect(address) sd. connect("/tmp/mysocket") AF_UNIX sd. connect( ("www. abc. com", 7001) ) AF_INET

Sending Data bytes. Sent = sd. send(message, flags) Stream Sockets sd. sendall(message, flags) must be connected send - buffering may mean only part of a message sent may need to repeat call until everything sent sendall - calls send repeatedly until everything transmitted Datagrams bytes. Sent = sd. sendto(message, flags, address) must not be connected recipient's address specified in call buffering may mean only part of a message sent may need to repeat call until everything sent Copyright © 2000 -13 CRS Enterprises Ltd 33

sendall() is easier to use than send None returned on success exception thrown on failure but unlike send(). . . can't determine how much data has been successfully transmitted while(len(message) > 0): bytes. Sent = sd. send(message) message = message[bytes. Sent: ] Copyright © 2000 -13 CRS Enterprises Ltd 34

Receiving Data Stream Sockets message = sd. recv(buffer. Size, flags) must be connected buffer. Size specifies the maximum size of message send - buffering may mean only part of a message sent may need to repeat call until everything sent sendall - calls send repeatedly until everything transmitted Datagrams (message, sender. Address) = sd. recfrom(buffer. Size, flags) must not be connected recipient's address specified in call Copyright © 2000 -13 CRS Enterprises Ltd 35

Network Functions import socket ip_address = socket. gethostbyname(hostname) hostname = socket. gethostbyaddr(ip_address) fq_hostname = socket. getfqdn(ip_address) port_number = socket. getservbyname(service, protocol) dotted_address = socket. inet_ntoa(32 bit_address) 32 bit_address = socket. inet_pton(AF_INET, dotted. Address) host_address = ntohl(network_address) network_address = ntohs(host_address) hostname = socket. gethostname() (hostname, port) = socket. getsockname() (peer_hostname, peer_port) = socket. getpeername() Copyright © 2000 -13 CRS Enterprises Ltd 36

Address Conversion Functions 192 8 61 inet_ntoa 4 inet_pton "192. 8. 61. 4" Copyright

Working with Lines of Text Stream Sockets send data as a byte stream with no regard to demarcation between lines of text Network buffering. . . incomplete reads received data might end half way along a line Programmer's responsibility. . . to organize data into discrete lines makefile() read only file. Like. Object = sd. makefile('r', 0) list. Of. Lines = file. Like. Object. readlines() overcomes these problems builds a file like object backing socket must be in blocking mode Copyright © 2000 -13 CRS Enterprises Ltd 38

Non Blocking Sockets. . . client. Descriptor, client. Address = sd. accept() client. Descriptor. setblocking(0) client. Descriptor. settimeout(15. 0) socket. setblocking(flags) flags = 0 flags = 1 for non blocking for blocking (default) Non blocking sockets have timeouts. . . socket. settimeout(value) value = 0. 0 is the default Copyright © 2000 -13 CRS Enterprises Ltd 39

. . . Non Blocking Sockets What if several blocking sockets are receiving data at the same time how do you know which socket has data available if you pick the wrong one you could block for a long time C API Unix. . . poll() and select() Windows. . . select() Python select module also look at asyncore and asynchat Copyright © 2000 -13 CRS Enterprises Ltd 40

Select Uses 3 lists of socket descriptors only active sockets get added to the lists sd 1 = socket(socket. AF_INET, socket. SOCK_DGRAM) sd 2 = socket(socket. AF_INET, socket. SOCK_DGRAM) sd 3 = socket(socket. AF_INET, socket. SOCK_DGRAM) def poll. Sockets(): timeout = 5 read. List, write. List, error. List = select( [sd 1, sd 2, sd 3], [ ], timeout) if [ read. List, write. List, error. List ] == [ [ ], [ ] ]: print timeout, "secs elapsed" else: for sd in read. List: print id(sd), sd. recvfrom(100) Copyright © 2000 -13 CRS Enterprises Ltd 41

Poll The poll() system call is only supported on Unix systems Provides better scalability for network servers that service a large number of clients the system call only requires listing the file descriptors of interest select() builds a bitmap, turns on bits for the fds of interest, and then afterward the whole bitmap has to be linearly scanned again Copyright © 2000 -13 CRS Enterprises Ltd 42

3

Socket. Server 3 Copyright © 2000 -13 CRS Enterprises Ltd 46

Socket. Server Module Socket. Server module simplifies the task of writing network servers Must create a Server object first from: 1. 2. 3. 4. TCPServer - server supporting the TCP UDPServer - server supporting the UDP Unix. Stream. Server - server implementing Unix domain stream-sockets (inherits from TCPServer) Unix. Datagram. Server - server implementing Unix domain datagrams (inherits from UDPServer) Then create a Handler provides a callback to process client requests Copyright © 2000 -13 CRS Enterprises Ltd 47

Servers Use method from Socket to send and receive Forking and threading versions of each type of server can be created using the Forking. Mix. In and Threading. Mix. In mix-in classes Base. Server TCPServer UDPServer Unix. Stream. Server Unix. Datagram. Server Copyright © 2000 -13 CRS Enterprises Ltd 48

Handlers Must override handle() method setup() and finish() are optional Derived classes provide convenience attributes rfile and wfile Base. Request. Handler Stream. Request. Handler setup() handle() finish() Datagram. Request. Handler self. rfile self. wfile Copyright © 2000 -13 CRS Enterprises Ltd 49

Plug-in Handlers Once you've written a handler it can be plugged into any server handlers are decoupled from servers class My. Request. Handler. A(Socket. Server. Stream. Request. Handler): . . class My. Request. Handler. B(Socket. Server. Datagram. Request. Handler): . . server = Socket. Server. TCPServer( ("localhost", 7001), My. Request. Handler. A) server = Socket. Server. UDPServer( ("localhost", 9001), My. Request. Handler. B) Copyright © 2000 -13 CRS Enterprises Ltd 50

Server Methods Base. Server. serve_forever(poll_interval=0. 5) handle requests until an explicit shutdown() request polls for shutdown every poll_interval seconds Base. Server. shutdown() tells the serve_forever() loop to stop and waits until it does Base. Server. allow_reuse_address allows server will allow the reuse (bind) of an address - defaults to False Base. Server. handle_error(request, client_address) called if the Request. Handler. Class‘s handle() method raises an exception default action is to print the traceback to standard output Copyright © 2000 -13 CRS Enterprises Ltd 51

Handler Methods Request. Handler. finish() called after the handle() method to perform any clean-up actions required Request. Handler. handle() must do all the work required to service a request self. request - client's request self. client_address - client address self. server - server instance Stream. Request. Handler or Datagram. Request. Handler also provide self. rfile - get the request data self. wfile - return data to the client Request. Handler. setup() called before the handle() method to perform initialization Copyright © 2000 -13 CRS Enterprises Ltd 52

Mix-in Classes To build asynchronous handlers use Threading. Mix. In- thread per client Forking. Mix. In - process per client class Threaded. TCPServer(Socket. Server. Threading. Mix. In, Socket. Server. TCPServer): . . . class Threaded. TCPServer(Socket. Server. Forkinging. Mix. In, Socket. Server. TCPServer): . . . Copyright © 2000 -13 CRS Enterprises Ltd 53

4

Secure Sockets 4 Copyright © 2000 -13 CRS Enterprises Ltd 56

SSL support in Python SSL module part of standard Python library M 2 Crypto SSL toolkit featuring RSA, DH, HMACs, message digests, symmetric ciphers (including AES) SSL functionality to implement clients and servers. HTTPS extensions to Python's httplib, urllib and xmlrpclib. Alternative implementations Paramiko py. Open. SSL TLS Lite Pyc. URL Copyright © 2000 -13 CRS Enterprises Ltd 57

SSL support in Python Secure Sockets Layer (SSL) support in Python wasn't secure before Python 2. 6 New ssl module in Python 2. 6 basically secure, but had some security issues Security improvements in Python 2. 7 can specify ciphers to use explicitly rather than just using what comes as default with the SSL module with Open. SSL 1. 0 using ssl. PROTOCOL_SSLv 23 is safe will not pick the insecure SSLv 2 protocol Copyright © 2000 -13 CRS Enterprises Ltd 58

Imports Several imports required import socket import ssl import Simple. HTTPServer import traceback Copyright © 2000 -13 CRS Enterprises Ltd # low level socket interface # ssl module # builds simple servers # stack trace on failures 59

$Wrapping Sockets on Client ssl. wrap_socket( sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version={see docs}, ca_certs=None,$

Wrapping Sockets on Client ssl. wrap_socket( sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version={see docs}, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True, ciphers=None) ssl_sock = ssl. wrap_socket( sd, ca_certs="PCA-3. pem", cert_reqs=ssl. CERT_REQUIRED) ssl_sock. connect(('www. verisign. com', 443)) Copyright © 2000 -13 CRS Enterprises Ltd 60

Other Client SSL Methods print "Peer: ", repr(ssl_sock. getpeername()) print "Cipher: ", ssl_sock. cipher() print "Peer cert: ", pprint. pformat(ssl_sock. getpeercert()) PEM_Certificate = ssl. get_server_certificate( ('www. verisign. com', 443), ca_certs="PCA-3. pem") DER_Certificate = ssl. PEM_cert_to_DER_cert(PEM_Certificate) Copyright © 2000 -13 CRS Enterprises Ltd 61

SSL Server. . . Server receives filename opens file and serves it to client sd = socket() sd. bind(('localhost', PORT)) sd. listen(5) def do_request(tcp_stream, from_addr): filename = tcp_stream. read() filename = "files/" + filename try: f = open(filename, "r") all = str(f. readlines()) f. close() except Exception, e: all = "Invalid File" tcp_stream. write(all) while True: try: newsocket, from_addr = sd. accept() ssl_socket = ssl. wrap_socket(newsocket, server_side=True, certfile='localhost. pem', ssl_version=ssl. PROTOCOL_TLSv 1) do_request(ssl_socket, from_addr) except Exception: traceback. print_exc() Copyright © 2000 -13 CRS Enterprises Ltd 62

. . . SSL Client sends filename to server serves contents of file import socket, ssl, pprint sd = socket(socket. AF_INET, socket. SOCK_STREAM) local_ssl_sock = ssl. wrap_socket(sd, ca_certs="localhost. pem", cert_reqs=ssl. CERT_NONE, ssl_version=ssl. PROTOCOL_TLSv 1) local_ssl_sock. connect(('localhost', 443)) # ask server to serve a file local_ssl_sock. write("f 2") data = local_ssl_sock. read() print data local_ssl_sock. close() Copyright © 2000 -13 CRS Enterprises Ltd 63

SSH with Paramiko implements the SSH 2 protocol for secure (encrypted and authenticated) connections to remote machines written entirely in python (no C or platform-dependent code) SSH 2 replaced telnet and rsh for secure access to remote shells Client is responsible for authenticating using a password or private key, and checking the server's host key Server is responsible for deciding which users, passwords, and keys to allow Key signature and verification is done by Paramiko Copyright © 2000 -13 CRS Enterprises Ltd 64

5

Other Web Modules 5 Copyright © 2000 -13 CRS Enterprises Ltd 68

Other Web Modules cgi webbrowser json used to implement CGI scripts on a webserver ftplib implements the client side of the FTP protocol (urllib package provides a higher-level interface) urllib provides a high-level interface for writing clients that need to interact with HTTP servers, FTP servers, and local files requests http smtplib an improved module to replace urllib high level control of browsers - often used in testing used to serialize and unserialize objects represented using Java. Script Object Notation (JSON) modules for writing HTTP clients and servers low-level SMTP client interface Copyright © 2000 -13 CRS Enterprises Ltd 69

Web Browsers webbrowser module is high level easy to control default browser import webbrowser url = 'http: //www. python. org' try: # Open URL in a new tab, if a browser window is already open. webbrowser. open_new_tab(url + '/doc') # Open URL in new window, raising the window if possible. webbrowser. open_new(url) except Exception, e: print e Copyright © 2000 -13 CRS Enterprises Ltd 70

Web Servers Simple. HTTPServer module provides all the code for a simple web server serves static pages Base. HTTPServer module allows customization via Handler modules can add support for CGI Copyright © 2000 -13 CRS Enterprises Ltd 71

Simple HTTP Server Simple. HTTPServer module is high level easy to serve pages to clients import Simple. HTTPServer import Socket. Server # minimal web server. # serves files relative to the current directory. PORT = 8000 Handler = Simple. HTTPServer. Simple. HTTPRequest. Handler httpd = Socket. Server. TCPServer(("", PORT), Handler) print "serving at port", PORT httpd. serve_forever() Copyright © 2000 -13 CRS Enterprises Ltd 72

Simple HTTP Server - fine control Write your own handler class derive from Base. HTTPServer define a go. GET() method to control how and what gets served to clients import Base. HTTPServer class Handler(Base. HTTPServer. Base. HTTPRequest. Handler): def do_GET(self): # code for GET subprotocol PORT = 8000 httpd = Base. HTTPServer(("", PORT), Handler) print "serving at port", PORT httpd. serve_forever() Copyright © 2000 -13 CRS Enterprises Ltd 73

FTP Clients Login to FTP server. . . Send a local file to FTP server import ftplib ftp_server = ftplib. FTP(host, username, password) # Open the file you want to send f = open(filename, "rb") # Send it to the FTP server resp = ftp_server. storbinary("STOR "+filename, f) # Close the connection ftp_server. close() Copyright © 2000 -13 CRS Enterprises Ltd 74