Net BSD Kernel Topics IP Processing mbuf structure

Net. BSD Kernel Topics: IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 1

Goals • IP – Understand the structure of IP processing in Net. BSD – Understand the IP packet format – Become familiar with IP packet field access » accessing src and dst addresses, etc… • mbufs – Understand the mbuf structure – Become familiar with mbuf access routines » packet size » data location • Stevens TCP/IP books – impress upon you the importance of these ref. books January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 2

What we are NOT going to talk about • IPv 6 – Everything we talk about will be in terms of IPv 4 – Router Plugins also supports IPv 6 – The code can be confusing » use IPv 6 structures since they are superset of IPv 4 • IP Options • Fast Forward Path • Details of interaction between device drivers and IP – actually, we’ll talk about some of this… • Packet Scheduler • Routing protocols • daemons: routed, rsvpd, … • January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 3

Net. BSD Kernel Topics: IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 4

Vocabulary: Terms Used in Stevens • message: Transport protocol data given to IP • datagram: message + IP header • fragment: if datagram is too large for network: IP splits it. Each fragment contains its own IP header • packet: fragment or datagram small enough for network • frame: packet + data-link layer header January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 5

Protocol messages, IP Packets, Fragments, … message IP UDP Application datagram fragment 1 IP UDP application data IP packet link IP UDP application link IP data Application data IP Application data fragment 2 fragment 3 Application data link IP Application data frame Derived from Figure 8. 7 from Wright/Stevens, Vol. 2 January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 6

IP Packet Format Vers HLen TOS Total length ID Flags Protocol TTL Fragment Offset Header Checksum Source Address Destination Address Source Port Destination Port. . . IP Header (with no Options fields) Transport Protocol (e. g. TCP, UDP) Header Packet Payload 0 January 9, 2001 8 16 24 31 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 7

IPv 4 Forwarding in the Net. BSD Kernel (APIC) T R A N S P O R T N E T W O R K socket layer tcp/udp_input ip 6_input ipintr tcp/udp_output ip 6_forward ip 6_output ip_forward ip_output ipintrq: D A T A L I N K packet_scheduler apic_isr input packets January 9, 2001 PS_Q atmc_output atmc_input U P P E R apic_intr interrupts L O W E R apic_start apic_devoutput packets Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 8

User IP in the Net. BSD Kernel User Program T R A N S P O R T N E T W O R K D A T A L I N K socket layer Waits in Socket tcp/udp_input Buffer for User Program to pick up ipintrq: ip_output Software Interrupt U P P E R PS_Q atmc_output atmc_input packet_scheduler apic_isr input packets January 9, 2001 ip_forward tcp/udp_output apic_intr interrupts L O W E R apic_start apic_devoutput packets Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 9

IP Packet Handling in the CB Kernel User Space T R A N S P O R T are the gates? This is. Where thethe gate that you use. Where Here are is the route rest of lookup the will gates? done? Kernel N E T W O R K socket layer tcp/udp_input ip 6_input ipintr tcp/udp_output ip 6_forward ip 6_output sgate ip_forward ip_output dgate ipintrq: SW Interrupt HW Interrupt D A T A L I N K packet_scheduler apic_isr input packets January 9, 2001 PS_Q atmc_output atmc_input U P P E R apic_intr interrupts L O W E R apic_start apic_devoutput packets Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 10

Important Source Files (usr/src/sys/netinet/ip*) • ip. h – struct ip { } – #define’s • ip_input. c – ipintr() (This is ip_input routine) – ip_forward() • ip_output. c – ip_output() January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 11

IP Packet Handling in CB Kernel with APIC • Device Driver -- Device Dependent – apic_intr() » read INTR_ACK: 0 => not for us, return » apic_isr() – apic_isr() » read NOTIFY_LIST, next rcv channel needing attention • Process descriptors for RCV channel – swap words in each received mbuf (APIC BUG) – link mbufs until we find end of frame – verify CRC (just check flag set by APIC) – atmc_input(packet) – repeat ipintr • repeat ip_forward ipintrq: atmc_output atmc_input apic_isr January 9, 2001 Router Plugins (Crossbow) ip_output apic_intr apic_devoutput Washington WASHINGTON UNIVERSITY IN ST LOUIS 12

IP Packet Handling in CB Kernel (continued) • Device Driver - Dev. Independent atmc_input(): – LLC/SNAP processing (if needed) » extract type (IP/RATM/…) – AAL 5 processing » extract length » strip trailer – aiu_getafix(packet) (flow table will be discussed later) » aiu_getafix stores a ptr to FTE in packet – Enqueue in IP Queue ipintr ip_forward ip_output ipintrq: atmc_output atmc_input apic_isr January 9, 2001 Router Plugins (Crossbow) apic_intr apic_devoutput Washington WASHINGTON UNIVERSITY IN ST LOUIS 13

IP Packet Handling in CB Kernel (continued) • IP Input -- ipintr(): – – – Get next packet from IP Queue Do some basic checks, header, length, checksum… process IP options aiu_dgate(&m, 2) Router Plugins Dynamic Gate #2 if packet is not for us and we can forward it: » forward packet ip_forward() -- upcoming slide. . . » return – aiu_dgate(&m, 4) Router Plugins Dynamic Gate #4 – protocol specific input routine e. g. tcp_input() ipintrq: atmc_input apic_isr January 9, 2001 ip_forward ip_output atmc_output apic_intrapic_devoutput Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 14

IP Packet Handling in CB Kernel (continued) • IP Forwarding -- ip_forward(): – – decide if we need to send any redirects to sender aiu_dgate(&m, 5) Router Plugins Dynamic Gate #5 route lookup ip_output() Update for Fred’s new Gate also check details of if/when route lookup is done ipintr ip_forward ipintrq: atmc_output atmc_input apic_isr January 9, 2001 Router Plugins (Crossbow) ip_output apic_intr apic_devoutput Washington WASHINGTON UNIVERSITY IN ST LOUIS 15

IP Packet Handling in CB Kernel (continued) • IP Output -- ip_output() – aiu_dgate(&m, 3) Router Plugins Dynamic Gate #3 – if (no route yet) ipintr » get route – check for special processing » ANEP Options ip_forward ip_output ipintrq: atmc_output atmc_input apic_isr apic_intr apic_devoutput • anep_output() – if DAN then afd_handle_dan_packet() : ACTIVE PROCESSING – send on interface January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 16

IP Packet Handling in CB Kernel (continued) • Device Driver Device Indep. -- atmc_output(): – – LLC/SNAP processing AAL 5 processing (if needed) if packet_scheduling: enqueue for PS if !packet_scheduling: send to devoutput() • Device Driver Device Dep. -- apic_devoutput() – configure an APIC descriptor for each mbuf in packet – Resume APIC TX channel ipintr ip_forward ip_output ipintrq: atmc_output atmc_input apic_isr January 9, 2001 Router Plugins (Crossbow) apic_intr apic_devoutput Washington WASHINGTON UNIVERSITY IN ST LOUIS 17

Note about Addresses • The address structures in the Crossbow Kernel are IPv 6 – IPv 6 address structures are a superset of IPv 4 – IPv 6 address are 128 bits • We will be using IPv 4 addresses – e. g. 192. 168. 5. 2 – IPv 4 addresses are 32 bits • Notation for using IPv 4 address in IPv 6: – Use double colon before address: : : 192. 168. 5. 2 – Double colon tells the utilities to set everything to the left to 0’s January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 18

Net. BSD Kernel Topics IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 19

The mbuf Data Structure mbuf{} 0 -108 0 -100 208 -2048 MT_xxx M_EXT M_PKTHDR | M_EXT m_nextpkt m_len m_data m_type m_flags 0 M_PKTHDR m_pkthdr. len m_pkthdr. rcvif m_ext. ext_buf m_ext. ext_free Normal mbuf January 9, 2001 Normal mbuf with Packet Hdr 2048 -byte cluster Cluster mbuf with Packet Hdr Router Plugins (Crossbow) m_ext. ext_size Washington WASHINGTON UNIVERSITY IN ST LOUIS 20

• Highlight: (add Stevens’ like figures…) – m_next vs m_nextpkt (figure 2. 2 page 35) » m_next: points to next mbuf in THIS packet » m_nextpkt: points to the first mbuf of the NEXT packet – m_len vs. m_pkthdr. len – m_ext. ext_buf vs. m_data – Which mbuf structure we use with APIC • Example: – – – Sorry: This was just one do m_pullup of my notes slides that get data pointer should have been removed printing print out src addr, dst addrbefore and protocol fields get Transport header pointer print out dst port January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 21

The mbuf Data Structure mbuf{} 0 -100 208 -2048 m_nextpkt m_len m_data m_type m_flags MT_DATAMT_xxx 0 M_PKTHDR MT_xxx M_EXT M_PKTHDR | M_EXT m_pkthdr. len Sorry: This was just one m_pkthdr. rcvif m_ext. ext_buf of my notes slides that m_ext. ext_free should have been removed 2048 m_ext. ext_size before printing 2048 -byte cluster January 9, 2001 Router Plugins (Crossbow) 2048 -byte cluster Washington WASHINGTON UNIVERSITY IN ST LOUIS 22

The mbuf Data Structure (example) head of queue tail of queue mbuf{} m_nextpkt next mbuf in chain m_len 42 m_data m_type MT_DATA m_flags M_PKTHDR mbuf m_pkthdr. len 192 m_pkthdr. rcvif NULL packet header Ethernet header, IP header, UDP header mbuf{} m_nextpkt m_len m_data m_type next mbuf in chain NULL 100 MT_DATA m_flags 0 m_pkthdr. len m_pkthdr. rcvif mbuf{} m_nextpkt m_len m_data m_type m_flags NULL 50 MT_DATA 0 50 bytes of data 100 bytes of data next packet in queue January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 23

The mbuf Data Structure (example, continued) mbuf{} tail of queue m_nextpkt next mbuf in chain NULL 54 m_len m_data m_type MT_HEADER m_flags M_PKTHDR mbuf m_pkthdr. len 1514 packet m_pkthdr. rcvif NULL header Ethernet header, IP header, TCP header January 9, 2001 mbuf{} m_nextpkt NULL m_len 1460 m_data m_type MT_DATA m_flags M_EXT m_pkthdr. len 2048–byte cluster m_pkthdr. rcvif m_ext. ext_buf m_ext. ext_free m_ext. ext_size 2048 Router Plugins (Crossbow) 1460 bytes of data Washington WASHINGTON UNIVERSITY IN ST LOUIS 24

Net. BSD Kernel Topics IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 25

Loadable Kernel Modules • Mechanism in Net. BSD to dynamically load code into running kernel • Net. BSD System Utilities: – modload(8) – modunload(8) – modstat(8) • Module Types supported: – System Call modules » When unloaded, returns to original system call » Any system call can be replace » Take care when replacing ioctl(2) since LKM uses it to load/unload modules!! – – Virtual File System modules Device Driver modules: Block and character device drivers Execution Interpreters: For binaries not normally usable by OS Miscellaneous modules » No defined interfaces » Up to user/developer to provide hooks to get to the code » This is what Router Plugins uses January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 26

Loadable Kernel Modules: modload • Open /dev/lkm – ioctl’s will be performed on the open file descriptor • Prelink module, open it and calculate size info – system(“ld -A /netbsd -e _<entry> -o <outfile> -T <addr=0> <module>”); • Reserve Kernel memory: – ioctl(lkm_fd, LMRESERV, size_info) – returns kernel load address • Relink at kernel load address • Open relinked module and load it into kernel – ioctl(lkm_fd, LMLOADBUF, . . . ) • • Adjusting symbol table entry pointers and load it. sync(); Call the module’s entry function to test it. Post-install if called for. . . – ioctl(lkm_fd, LMSTAT, . . . ) January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 27

Net. BSD Kernel Topics IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 28

Interrupts • 8 levels – – – – spl 0 splsoftclock splnet spltty splbio splimp splclock splhigh normal low-priority clock processing network protocol processing terminal I/O disk and tape I/O network device I/O high-priority clock processing all interrupts blocked • Typical usage: int i; i = splimp(); [do something at device I/O level] splx(i); January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 29

Net. BSD Kernel Topics IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 30

Other kernel details you must know • No floating point • No #include in header files • More caveats and info from Fred later… January 9, 2001 Router Plugins (Crossbow) Washington WASHINGTON UNIVERSITY IN ST LOUIS 31