The New MSR Plugin Download Environment jddarl wustl

The New MSR Plugin Download Environment jdd@arl. wustl. edu Washington WASHINGTON UNIVERSITY IN ST LOUIS http: //www. arl. wustl. edu/~jdd

First: The Old Way • Compile plugin code on Net. BSD • Copy plugin directory onto File System destined for SPC • Boot SPC • Log in to SPC – requires user space and serial line or telnet access • Use modload(1) on SPC to load plugin – links plugin against kernel in /netbsd on SPC • Use modstat(1) on SPC to see list of loaded plugins John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 2

The New Way • Compile plugin code on Net. BSD • Verify plugin has no undefined symbols – There is no link against running kernel anymore – msr_lkm. h provides macros to needed kernel fucntions and structures • We can add more as we need them. • Use plugin. Download utility from CP (Linux) to download plugin to SPC – Not sure I’ve even tried it on Net. BSD… • Use sendcmd utility from CP to configure plugin, filters, etc • Use plugin. Stat utility from CP to view list of loaded plugins John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 3

plugin. Download • Uses gnu binary utilities cross-linker – Runs on Linux, builds a Net. BSD binary – Used to: • prelink to get size information • final link to relocate • Uses new AAL 5 control data channels to download plugin directly into kernel memory. – uses sendcmd API to: • allocate kernel memory for plugin • set up AAL 5 data channel • call plugin entry function to complete load operation • Side Note: sendcmd API – should be expanded to include all of sendcmd, but right now we have about 15 “commands”. John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 4

• The rest of the slides are old notes John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 5

Plugin Loading Steps • • Compile Plugin code on Net. BSD Do one of these two, currently Linux seems to work better for later steps: – Pre-link Plugin on Net. BSD to get size info ready Ø ld -e example -T 0 -o pl_combined. o – Pre-link Plugin on Linux to get size info ready Ø ld-cross –o pl_combined. o –e _example –Ttext 0 combined. o John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 6

Plugin Loading Steps • ü On Linux CP: open file (pl_combined. o), get size information and allocate MSR/SPC memory for plugin Ø sendcmd –p # -c rp_pcu –s allocate –d <size> -d <sym_symsize> • sendcmd code in kernel calls msr_lkmiotcl(LMRESERV, <ptr> , FWRITE) • <ptr> points to a (struct lmc_reserv) which passes in: – size = a_text + a_data + a_bss (all from a. out header) » but we don’t need to load the bss since it is all 0 – sym_size = a_syms + stb. st_size – N_STROFF(info_buf) » a_syms is symbol table size » string table is at the end so, the string table size is full file size ( stb. st_size minus string tab offset) – sym_symsize = a_syms • and returns: – address of block of memory allocated in kernel (resrvp->addr) – slot in lkm table (resrvp->slot) – address of symbols (resrvp->sym_addr) • msr_lkmioctl() returns 0 if successful, non-zero if there was an error. – sendcmd will return • Kernel address (Pl. Addr) for plugin to be loaded at • Slot number • Address of location of symbols (what do we use this for? ) – This is used for adjusting the string table pointers before they are loaded (see modload. c for details) – useful command: Ø objdump John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 7

Plugin Loading Steps (continued) • On Linux CP: re-link plugin to relocate to Pl. Addr Ø ld-cross –o plugin –e _example –Ttext <Pl. Addr+0 x 20> combined. o • relocates to pl. Addr+0 x 20 and sets ‘_example’ as the entry point – the +0 x 20 is because we are currently downloading the a. out header – This sets the a_entry field in the a. out header of the file • file plugin is now ready for downloading • On Linux CP: request two data channels to MSR Kernel – One for plugin text and data: Ø sendcmd –p # -c dchan -s get -d <Pl. Addr> -d <size> – Another for the plugin symbols (sym_addr was returned by allocate command earlier) Ø sendcmd –p # -c dchan -s get -d <sym_addr> -d <sym_size> – sendcmd returns the dchan# in its responses • On Linux CP: download file plugin into MSR/SPC memory: Ø plugin. Download -f plugin –S <sym_addr> -d <dchan#1> -d <dchan#2> -V <vci> -L <lkm slot> – This needs to adjust the string table before downloading. Thus it needs the sym_addr which was returned by the allocate sendcmd above – text and data go to dchan#1 and symbol and string table to go dchan#2 John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 8

Plugin Loading Steps (continued) • On Linux CP: Finish the loading of the plugin Ø sendcmd –p # -c rp_pcu –s load -d <entry_addr> -d <size> -d <sym_size> – I don’t think we actually need the Pl. Addr here. – I think in this command (or an earlier one) we do need to send the entry symbol address – We are also sending the amount of data sent for the plugin (size) and symbols (sym_size) • the msr_lkm code will use these two values to update – curp->offset – curp->sym_offset – This sendcmd causes a call to msr_lkmioctl(LMREADY, <data>, FWRITE) – We may also need to do some other stuff. See page of ioctl’s used by modload. • On Linux CP: release the data channel Ø sendcmd –p # -c dchan -s release -d <dchan#> • When all done with plugin: – On Linux CP: unload and deallocate the plugin Ø sendcmd –p # -c rp_pcu –s unload –d <id> • We should never have to use the LMLOADBUF and LMLOADSYMS cmds John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 9

Data Transfer • Options for the download phase, CP side: – hack up modload from Net. BSD and build it on Linux • modload uses ioctl’s to load the module into the local kernel • we want to send AAL 5 frames – modify Stage 3 of AAL 5_download • Stage 3 already processes Net. BSD a. out files • Need to do one of these: – change Stage 3’s interaction with Stage 2, i. e. have it not expect any acks – add a Stage 2 like thing to the MSR kernel for accepting plugin frames – Merge modload and Stage 3 • modload does some manipulations to get the string table correct • Stage 3 has all the code to send AAL 5 frames and wait for ACK/NACK • Options for the downloading phase, MSR side: – Build a new AAL 5 control data channel structure • Uses AAL 5 frame buffer pool that the IP fwding channels use • Re-uses some of the frame processing code in kernel • Arriving frames are copied into previously allocated memory associated with the data channel. – It would be nice if APIC descriptors pointed directly at the allocated memory, but that breaks the current model and would make it hard to build in safeguards against corruption. John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 10

Data Transfer (continued) MSR Kernel Dchannel Table Vaddr Length Start of Virtual Length of block in address block bytes (32 b) where data frames Index(dchan#) should go (32 b) 0 Next Addr Seq # Where next frame should go. Initially set to <Vaddr> (32 b) Next expected frame sequence number (first frame should be 1) (16 b) . . . 1 DCHAN_MAX John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 11

Data Transfer (continued) MSR Control Data Channel RATM AAL 5 Frame Format (as it appears in the MSR Kernel) seq# (16 bits) framelength (16 bits) pad (16 bits) AAL 5 Frame Data (<= 1992 Bytes) 2016 B flags: 1: Data 2: Ack 4: Nack flags (8 bits) dchan#(8 bits) 2000 B APIC Rx desc points here 1992 B Shim Reserved space (8 Bytes) AAL 5 Trailer (8 Bytes) John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 12

External Symbols in Plugin • We will implement a structure to contain pointers to Kernel symbols that Plugins are allowed to use. – This structure will surely grow over time as we discover more kernel functions that we want to expose to Plugins • When a plugin is loaded, it will be given the pointer to this structure in the kernel via the call to its entry function • The plugin will therefore not need to link against the running kernel to resolve symbols. • This strategy will achieve two goals: – Put bounds on what plugins can do in the kernel – Make the linking and downloading of plugins easier John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 13

External Symbols in Plugin (continued) • Current kernel functions used in example plugin: – malloc related: • free() – via FREE() macro from sys/malloc. h • malloc() – via MALLOC() macro from sys/malloc. h – MSR Related: • msr_printf() • PCU Related: – pcu_deregister_class() – pcu_free_all_instances() – pcu_register_class() – LKM Related: • lkm_nofunc() • lkm_dispatch() – via DISPATCH() macro from sys/lkm. h • lkm_exists() John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 14

The rest of the slides are supporting notes some from other places John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 15

Data Transfer • Options for the download phase, CP side: – hack up modload from Net. BSD and build it on Linux • modload uses ioctl’s to load the module into the local kernel • we want to send AAL 5 frames – modify Stage 3 of AAL 5_download • Stage 3 already processes Net. BSD a. out files • Need to do one of these: – change Stage 3’s interaction with Stage 2, i. e. have it not expect any acks – add a Stage 2 like thing to the MSR kernel for accepting plugin frames • Options for the downloading phase, MSR side: – When we get a request to load a plugin of size <size> Ø sendcmd –p # -c rp_pcu –s allocate –d <size> • Allocate an APIC buffer of that size • associate the plugin load descriptor with that buffer • Tie the descriptor to the plugin load VCI • Go – Once we have the plugin in the allocated buffer: • in response to a Ø sendcmd –p # -c rp_pcu –s load –d <Pl. Addr> -S “example • use calls to lkmioctl() to do the rest John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 16

Implementing modload in MSR • Issues: – Where will we download plugin from? • presumably just from the CP – Will the plugin on the CP already be pre-linked with the current MSR kernel? • if not, we may have Linux vs. Net. BSD problems. • if so – how do we plan to keep plugins and MSR kernels in sync – how do we know the address where the plugin will go in the kernel – Dynamic scheme: • allocate memory on demand in the kernel for a new plugin – Static scheme: • predefined slots at predefined addresses in kernel • Each plugin is built to fit in a specific slot at a predefined address • Still need to resolve symbols with actual kernel – if we allow this at all… John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 17

Implementing modload in MSR • Option 1: Use sendcmd utility to execute modload on SPC – Download plugin binary to a file on MSR • e. g. /var/run/rp/plugin/file. ABC. o – and run modload • e. g. modload –o /var/run/rp/plugin/file. ABC. o –e example file. ABC. o – Something like: Ø sendcmd –p # –c rp_pcu –s load –S “file. ABC. o” –S “example” • uses proposed new option –S for including strings • causes file “file. ABC. o” to be downloaded to MSR and loaded as kernel module named “example” – Requires presence on MSR file system of: • /sbin/modload • /usr/bin/ld • up to date symbol version of kernel being executed John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 18

Implementing modload in MSR • Option 2: Prelink on CP, use sendcmd to download directly into memory – Link plugin against copy of kernel on CP Ø ld -A netbsd. MSR -e _<entry> -o <outfile> -T <addr=0> <module> – Reserve Kernel memory: Ø sendcmd –p # –c rp_pcu –s allocate –d <size> • returns kernel load address – Relink at kernel load address (LA) Ø ld -A netbsd. MSR -e _<entry> -o <outfile> -T <addr=LA> <module> – Open relinked module and load it into kernel Ø sendcmd –p # –c rp_pcu –s load –S <module> • uses proposed new option –S for including a strings – Adjusting symbol table entry pointers and load it. • What does this actually do in modload(1) – sync(); • What does this actually do in modload(1) – Call the module’s entry function to test it. – Post-install if called for. . . • ioctl(lkm_fd, LMSTAT, . . . ) John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 19

Implementing modload in MSR • Option 3: Prelink on CP, Plugin must use kernel function pointer array to access predefined, limited number of kernel functions, then use sendcmd to download directly into memory – plugin code uses local pointer for function pointer array which will be reset later when kernel calls entry function – Reserve Kernel memory: Ø sendcmd –p # –c rp_pcu –s allocate –d <size> • returns kernel load address – Relink at kernel load address (LA) Ø ld -A netbsd. MSR -e _<entry> -o <outfile> -T <addr=LA> <module> – Open relinked module and load it into kernel Ø sendcmd –p # –c rp_pcu –s load –S <module> • uses proposed new option –S for including strings – Call the module’s entry function to test it. • kernel will provide the actual location of the function pointer array – Post-install if called for. . . • ioctl(lkm_fd, LMSTAT, . . . ) John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 20

Implementing modload in MSR (continued) • Current ld commands issued by modload on Net. BSD: – Ø modload links it the first time so it can get size information: ld -A /netbsd -e _example -o /var/run/msr/example -T 0 combined. o modload links it the second time to resolve symbols and relocate: ld -A /netbsd -e _example -o /var/run/msr/example -T 100000 combined. o • Equivalent command on Linux: – Ø – First do this on Net. BSD (‘pl_’ implies pre-linked) ld -e _example -o pl_combined -T 0 combined. o This is just for relocating, everything else should be done on Net. BSD: ld-cross –o plugin –e _example –Ttext 0 x 100020 pl_combined. o Linux and Net. BSD seem to be off by 0 x 20 on their location of the text segment, still investigating that. – After this command, the file plugin should be ready for downloading John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 21

modload ioctl’s • We need to know exactly what each of these does: – ioctl(LMRESERV) • reserves kernel memory • reserves lkm slot for module • reserves kernel memory for the symbols – ioctl(LMLOADBUF) • loads module into memory, uses and affects lkm struct curp – ioctl(LMLOADSYMS) • loads symbol table into memory , uses and affects lkm struct curp – ioctl(LMLOADSYMS) • loads string table into memory , uses and affects lkm struct curp – ioctl(LMREADY) • if DDB is turned on, adds modules symbols to the DB symbol table • clears modules BSS space • calls entry procedure of module , uses and affects lkm struct curp – ioctl(LMSTAT) • post install retrieval of stats from module – ioctl(LMUNRESRV) • release slot for module if something goes wrong after LMRESERV. John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 22

(From Crossbow Tutorial) Net. BSD Kernel Topics IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 23

Loadable Kernel Modules • Mechanism in Net. BSD to dynamically load code into running kernel • Net. BSD System Utilities: – modload(8) – modunload(8) – modstat(8) • Module Types supported: – System Call modules • When unloaded, returns to original system call • Any system call can be replace • Take care when replacing ioctl(2) since LKM uses it to load/unload modules!! – – Virtual File System modules Device Driver modules: Block and character device drivers Execution Interpreters: For binaries not normally usable by OS Miscellaneous modules • No defined interfaces • Up to user/developer to provide hooks to get to the code • This is what Router Plugins uses John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 24

Loadable Kernel Modules: modload • Open /dev/lkm – ioctl’s will be performed on the open file descriptor • Prelink module, open it and calculate size info – system(“ld -A /netbsd -e _<entry> -o <outfile> -T <addr=0> <module>”); • Reserve Kernel memory: – ioctl(lkm_fd, LMRESERV, size_info) – returns kernel load address • Relink at kernel load address • Open relinked module and load it into kernel – ioctl(lkm_fd, LMLOADBUF, . . . ) • • Adjusting symbol table entry pointers and load it. sync(); Call the module’s entry function to test it. Post-install if called for. . . – ioctl(lkm_fd, LMSTAT, . . . ) John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 25

Overview (From MSR Tutorial) • Introduction to hardware environment • APIC core processing and buffer management • Overview of SPC kernel software architecture and processing steps • Plugin environment and filters • Command Facility John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 26

Packet Classification & Plugins • Classification provides and opportunity to bind flows to registered plugin instances. • General classifier - Network Management – classification using 5 -tuple • <saddr, sport, daddr, dport, proto> , • Prefix match on address, exact match port and proto • 0 is a wildcard for all fields – input and output ports – filters added/removed via the command facility John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 27

Flow Bound to a Plugin Frame/Buffer and IP Processing . . . General Match Classifier: Linear search of {src_addr, dst_addr, src_port, dst_port, proto}. Plugin. General Environment Classifier options: {First, Last, All} Rule Actions: {Deny, Permit, Frame Active}. Plugin AAL 5 plugin DQ/ plugin Rule flags {All, Copy, Stop}. . . pkt (struct ip *) Exact Match Classifier: Shim In Queuing Version H-len Total length Invoke TOS instance handler. Hash Identification flags Fragment offset Search handle_packet(inst, pkt, flags) {{src_addr, src_port, dst_port}, TTL dst_addr, protocol Header Rule 1 i 2 i 3 Address i 4 i 5 checksum Source Exact active processing same asfor /* Plugin may. Match: read and/or then flow spec. i 1 Destination i 2 Options i 3 Address i 4 i 5 Rule 2 linear search ? ? * modify content but not general match. The AAL 5 length. Classifier is and NM Flow Classifier/ Exact Match i 1 i 2 i 3 options: i 4 i 5 None. Rule 3 Ingress/ data (transport header and transport data) * delete. Filter it unless COPY. Egress IP ? {Deny, (channel Rule map)Rule i 1 so i 2 i 3 Active, i 4 i 5 Reserve}. 4 calculated Actions: Permit, IP header checksum are * On return the framework i 1 i 2 Idle, i 3 i 4 Remove} i 5 Rule 5 flags {Pinned, AAL 5 padding (0 - 40 bytes) plugin does not have to. Rule perform these * forwards packet */ i 1 i 2 i 3 i 4 i 5 Rule 6 CPCS-UU (0) Length (IP packet + LLC/SNAP). . . operations. Call CRC packet for i 2 bound instance hash i 3 i 4 i 5 Rule 7 handler i 1 Flow Table Instance 1 Route Lookup. Rule 8 return; } i 1 i 2 i 3 i 4 i 5 with pointer toto IP packet (struct ip *). DRR/. . . Send packet (Shim, FIPL, Simple, cache) flow exact flow i 1 Out i 2 Queuing i 3 i 4 i 5 Rule 9 instance->handle_packet(instance, flags) match classifier i 1 i 2 i 3 i 4 packet, i 5 Rule 10 {Active} Flow entry to plugin has a one-to-one relationship. John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 28

General Match Classifier Notes General Match Classifier: Linear search of {src_addr, dst_addr, src_port, dst_port, proto} • General Classifier options: {First, Last, All} • Rule Actions: {Deny, Permit, Active}. Search {All, Copy, Stop} • Rule flags Rule 1 Rule 2 Rule 3 Rule 4 Rule 5 Rule 6 Rule 7 Rule 8 Rule 9 Rule 10 John De. Hart - 12/22/2021 Invoke instance handler i 1 i 2 i 3 i 4 i 5 i 1 i 2 i 3 i 4 i 5 i 1 i 2 i 3 i 4 i 5 Washington WASHINGTON UNIVERSITY IN ST LOUIS 29

Exact Match Classifier Notes General Match Classifier: Linear search of - {src_addr, dst_addr, src_port, dst_port, proto}. • Exact Match Classifier options: None. • Rule Actions: {Deny, Permit, Active, Reserve}. • Rule flags {Pinned, Idle, Remove} hash Flow Table flow Instance 1 {Active} Flow entry to plugin has a one-to-one relationship John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 30

Active Processing Environment Class A “plugin x” Instance 1 {Active} Instance 2 {Active, All} Class B “plugin y” Class C “plugin z” Instance 1 {Deny} Instance 1 {Active} General/Exact Match Classifier Rule N Rule P • Plugin instance maps to at most one rule/filter. • General classifier: rule maps to at most 5 instances. • Exact match classifier: rule maps to at most 1 instance. John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 31

Creating an Instance Return reference to instance Class A classid = 100 inst_t *create_instance(class_t *, inst_id) create class instance Instance of Class A - (Base Class extended by Developer) create_instance() <Fields defined by the Base Class> class_t *class Called by PCU inst_t *next framework in inst_id id response to receiving command. fid_t bound_fid void (*handle_packet) (inst_t *, ip_t *, flag 32_t); void (*bind_instance) (inst_t *); void (*unbind_instance) (inst_t *); struct my_inst { void (*free_instance) (inst_t *); inst_t base; int (*handle_msg) (inst_t *, buf_t *, flag 8_t, seq_t, subclass defs len_t *) }; <Class Specific Data>. . . John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 32

Plugin Class Specific Interface • All plugins belong to a class. At run time a class (i. e. plugin) must be instantiated before it vcan be referenced. • Plugin is passed its instance pointer (like c++) as the first argument. • Developer may extend the base class (struct rp_instance) to include additional fields which are local to each instance. • Plugin developer must implement the following methods: – – void(*handle_packet)(struct rp_instance *, struct ip *, u_int 32_t); void(*bind_instance)(struct rp_instance *); void(*unbind_instance)(struct rp_instance *); void(*free_instance)(struct rp_instance *); – int (*handle_msg)(struct rp_instance *, void *, u_int 8_t, u_int 8_t); John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 33

Plugin Framework Enhancements • Integrated with Command framework – send command cells to PCU: • create instance, free instance, bind instance to filter, unbind instance – Send command cells to particular plugin instances – Send command cells to plugin base class • Enhanced interface to address limitation noticed in crossbow: – instance access to: plugin class, instance id, filter id – pcu reports describing any loaded classes, instances and filters John De. Hart - 12/22/2021 Washington WASHINGTON UNIVERSITY IN ST LOUIS 34