Design of a Diversified Router Line Card Jon

Design of a Diversified Router: Line Card Jon Turner, John De. Hart, Fred Kuhns jon. turner@wustl. edu, jdd@arl. wustl. edu, fredk@arl. wustl. edu http: //www. arl. wustl. edu/arl

Outline n What is NOT covered in these slides n JST’s original slides n Schedule n Model n Traffic Types n IP Addressing n Components » Switch » Meta. Link Loopback Block » LC Substrate Link Types n Packet Formats n LC Rx/Tx Design Implementation n n Common Router Framework (CRF) » Functional Blocks for implementing a Router 2 - JST/JDD/FK - 12/31/2021

Not Covered n Control » Installation » Configuration » Initialization » Monitoring » … n End Host » Substrate/Meta Model n Details about how Planet. Lab works. n These are very important topics but for now we will talk about the Data Path in the Network. 3 - JST/JDD/FK - 12/31/2021

Outline n JST’s original slides n Schedule n Model n Traffic Types n IP Addressing n Components » Switch » Meta. Link Loopback Block » LC Substrate Link Types n Packet Formats n LC Rx/Tx Design Implementation n n Common Router Framework (CRF) » Functional Blocks for implementing a Router 4 - JST/JDD/FK - 12/31/2021

Design Implementation n n We will now look at how our model might be implemented. Again, the primary focus will be on wired Ethernet as the access technology. We will look at RX and TX separately. First we will focus on the Substrate Router functionality in the LC. » As we do we will also show the packet formats as they traverse other parts of the Substrate Router. n Some of this will apply only to the Shared NP case. » For the sake of simpler diagrams we will reduce the IXP PE functional blocks as shown below: n Then we will look at the implementation of a common router framework in the IXP PE. Rx De. Mux Substrate RX 5 - JST/JDD/FK - 12/31/2021 Parse Lookup MR Header Format QM Tx Substrate TX

Rxsubstrate Txsubstrate IXP PE (Shared) RX 6 - JST/JDD/FK - 12/31/2021 MR TX Switch LC LC Packet leaving On Port M … … Packet arriving On Port N Switch Design Implementation

Substrate Link Configuration n RX Rules for determining type of Frame/Substrate Link: » P 2 P-DC is pre-configured on an interface by interface basis. » ((Ether. Type = Substrate) AND (VLAN=VLAN 0)) P 2 P-VLAN 0 SL n n n VLAN 0 is a predefined VLAN Id for use by the Substrate Network to connect peer substrate routers on a Multi-Access network SL id = Senders Ethernet Address Lookup MLI in SL Specific Table MR: MI » ((Ether. Type = Substrate) & (VLAN ≠ VLAN 0)) OR ((Ether. Type = ARP) and (Proto. Type = Substrate)) Multi-Access n n MLI is unique across a Multi-Access Substrate Link MLI Lookup in Multi-Access-SL Table MR: MI » ((Ether. Type = ARP) and (Proto. Type ≠ Substrate)) OR ((Ether. Type ≠ Substrate) AND (Ether. Type ≠ ARP)) Tunnel or Legacy n Substrate Link in a Tunnel (e. g. IPv 4) Ø n Legacy (e. g. IPv 4, ARP): Ø Ø n ((Ether. Type = IP) AND (IP_Proto = Substrate)) Substrate Tunnel in IPv 4 ((Ether. Type = IP) AND (IP_Proto = TCP)) Legacy IPv 4 ((Ether. Type = ARP) AND (Proto. Type ≠ Substrate)) Legacy ARP Etc. 7 - JST/JDD/FK - 12/31/2021

Substrate Link Configuration n TX: » Substrate Link uniquely identified by MR: MI tuple n MR: MI SL » Meta. Link uniquely identified by MR: MI tuple n MR: MI MLI » Hence: n MR: MI SL: MLI » Different Header formats may be defined per SL n n i. e. IPv 4 Tunnel header vs. P 2 P-VLAN 0 header Located with lookup based on MLI » P 2 P-VLAN 0 n One or more MLI per Meta. Net on the Multi-Access network » Multi-Access n n One MLI per Meta. Net on the Multi-Access network Meta-Destination Address, to get Ethernet Address use ARP » Legacy: n n n Substrate Link in a Tunnel IPv 4 Gateway Ø Legacy to/from Meta. Net 8 - JST/JDD/FK - 12/31/2021

Tables and Lookups n The following slides have some tables and lookups. » They are not meant as the only way to do things, just one way. » Some if not all of the tables and lookups will almost certainly be implemented using the TCAM. n For different types of frames we might need different types of headers. » For this a lookup result will probably indicate what format/template to use and some/all of the info to use. n The Point: » These slides are meant to show that we have the right information at the right place to do the needed lookup. » There is still some design work for the actual implementer(s) to do to get it all right and make it efficient. 9 - JST/JDD/FK - 12/31/2021

MR Txsubstrate Meta Frame n LC … MI (2 B) Mn. Flags(1 B) Nh. Addr (n. B) LEN (2 B) Switch LC Packet TX: All SL Types Mn. Flags: define what type of Next Hop Address (Nh. Addr) is being given: n or perhaps what type is needed (maybe Substrate provides it for broadcast…) » Type (2 b) Blade-to-Blade Ethernet Header (MR specific VLAN) Tx. MI (2 B) Mn. Flags (1 B) Nh. Addr (n. B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) 10 - JST/JDD/FK - 12/31/2021 n Nh. Mn. Addr: (01 b) Ø n MAC Address(10 b) Ø n We may need to use ARP to translate to MAC/Ethernet Address Could be used for Broadcast/Multicast NULL (00 b): No Next Hop Address given or needed. » Size (6 b): Length of Nh. Addr field in Bytes

MR Txsubstrate MI (2 B) Mn. Flags(1 B) Nh. Addr (n. B) LEN (2 B) VLAN MNid LC … Switch LC Packet TX: All SL Types Blade-to-Blade Ethernet Header (MR specific VLAN) Tx. MI (2 B) Mn. Flags (1 B) Nh. Addr (n. B) LEN (2 B) Meta Frame Mn. Id Mn. ID . . . Arp Cache Mn. ID Key Mn. ID: Mn. Addr If SL_TYPE=Multi-Access Lookup(Mn. ID, Mn. Addr) Mn. ID: Mn. Addr . . . PAD (n. B) CRC (4 B) SL_Type Port . . . SL_Type Port 11 - JST/JDD/FK - 12/31/2021 DAddr Mn. ID: Mn. Addr DAddr SL/ML Table VLAN: Tx. MI SL: MLI Result MLI MLI Hdr Info/Templates MLI Hdr Info/Templates

LC Packet TX – P 2 P-DC SL/ML Table SL_Type Port . . . n n Hdr Info/Templates Key Mn. ID: Mn. Addr . . . SL_Type Port Blade-to-Blade Ethernet Header MR specific VLAN Tx. MI (2 B) Mn. Flags (1 B) Nh. Mn. Addr (n. B) LEN (2 B) MLI MLI Arp Cache MLI Hdr Info/Templates n Result n DAddr Mn. ID: Mn. Addr Dst. Addr (6 B) P 2 P-DC should not need to give Nh. Mn. Addr Mn. Flags would indicate NULL Src. Addr (6 B) Type=802. 1 Q (2 B) TCI (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) 12 - JST/JDD/FK - 12/31/2021 P 2 P-DC does not use ARP Cache. DAddr is configured in the Hdr field of Per SL ML Tables Recalculate PAD (n. B) CRC (4 B)

LC Packet TX – P 2 P-Tunnel (IPv 4) SL/ML Table SL_Type Port . . . Hdr Info/Templates Key Result Mn. ID: Mn. Addr DAddr . . . SL_Type Port n Blade-to-Blade Ethernet Header MR specific VLAN Tx. MI (2 B) Mn. Flags (1 B) Nh. Mn. Addr (n. B) LEN (2 B) MLI MLI Arp Cache MLI Hdr Info/Templates n n Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI ≠ VLAN 0 (2 B) Type=IP (2 B) Ver/HLen/Tos/Len (4 B) ID/Flags/Frag. Off (4 B) TTL (1 B) Protocol=Substrate (1 B) Hdr Cksum (2 B) Src Addr (4 B) Dst Addr (4 B) MLI (2 B) LEN (2 B) » Is that reasonable? » Probably, Yes. Meta Frame 13 - JST/JDD/FK - 12/31/2021 » This may be an oversimplification. » The packet needs to be routed toward the other end of the Tunnel Mn. ID: Mn. Addr DAddr This seems to assume that we have a static first hop for the IP Tunnel. PAD (n. B) CRC (4 B) P 2 P-Tunnel does not use ARP Cache. DAddr is configured in the Hdr field of Per SL ML Tables Meta Frame Recalculate PAD (n. B) CRC (4 B)

LC Packet TX – P 2 P-VLAN 0 SL/ML Table SL_Type Port . . . SL_Type Port MLI MLI Hdr Info/Templates Arp Cache Key Mn. ID: Mn. Addr . . . MLI Hdr Info/Templates n Result DAddr n DAddr Do we want the P 2 PVLAN 0 to use the ARP Cache? Or is the DAddr configured in the Hdr fields of Per SL ML Tables? Mn. ID: Mn. Addr DAddr Blade-to-Blade Ethernet Header MR specific VLAN Tx. MI (2 B) Mn. Flags (1 B) Nh. Mn. Addr (n. B) LEN (2 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI=VLAN 0 (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) 14 - JST/JDD/FK - 12/31/2021 Dst. Addr (6 B) Recalculate PAD (n. B) CRC (4 B)

LC Packet TX – Multi-Access SL/ML Table SL_Type Port . . . SL_Type Port MLI MLI Hdr Info/Templates Arp Cache Key Result Mn. ID: Mn. Addr DAddr . . . MLI Hdr Info/Templates n n Mn. ID: Mn. Addr DAddr Multi-Access does use ARP Cache. If DAddr entry for Mn. Addr is missing, send packet up to XScale for it to perform ARP Request. Blade-to-Blade Ethernet Header MR specific VLAN Tx. MI (2 B) Mn. Flags (1 B) Nh. Mn. Addr (n. B) LEN (2 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI≠VLAN 0 (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) 15 - JST/JDD/FK - 12/31/2021 Dst. Addr (6 B) Recalculate PAD (n. B) CRC (4 B)

LC Packet RX: All SL Types Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) Meta Frame Type=802. 1 Q (2 B) TCI ≠ VLAN 0 (2 B) Type=IP (2 B) Ver/HLen/Tos/Len (4 B) ID/Flags/Frag. Off (4 B) TTL (1 B) Protocol=Substrate (1 B) Hdr Cksum (2 B) Src Addr (4 B) Dst Addr (4 B) MLI (2 B) LEN (2 B) Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI=VLAN 0 (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI≠VLAN 0 (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) P 2 P-DC 16 - JST/JDD/FK - 12/31/2021 PAD (n. B) CRC (4 B) P 2 P-Tunnel P 2 P-VLAN 0 Multi-Access

LC Packet RX: P 2 P-DC Per SL ML Tables VLANMR Rx. MI Blade QID VLAN Rx. MIBlade. QID MR MR VLAN Rx. MI Blade QID QID MR MR MR VLAN Blade QID VLAN Rx. MI Blade QID Rx. MI MR MR MR VLAN Rx. MI Blade QID Rx. MI MR MR (Port N P 2 P-DC Table) VLAN Blade QID Rx. MI MR N (Port P 2 P-DC Table) VLANMR Rx. MI Blade QID (Port N P 2 P-DC Table) VLANMR Rx. MI Blade QID Packet arriving On Port N Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) n n n There is one P 2 P-DC SL Table per Port (Physical Interface). Blade info in tables provides info so we can fill in templates for appropriate header(s) to get frame to necessary blade. QID is used on LC Blade-to-Blade Ethernet Header MR Specific VLAN Rx. MI (2 B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) Recalculate 17 - JST/JDD/FK - 12/31/2021 Do we want to include Sender’s Mn. Addr in Frame going to MR? » No. n But we may need to give it the MAC address. MI (2 B) LEN (2 B) Meta Frame Switch … Packet arriving On Port N LC n Rxsubstrate MR

LC Packet RX: P 2 P-Tunnel Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI ≠ VLAN 0 (2 B) Type=IP (2 B) Ver/HLen/Tos/Len (4 B) ID/Flags/Frag. Off (4 B) TTL (1 B) Protocol=Substrate (1 B) Hdr Cksum (2 B) Src Addr (4 B) Dst Addr (4 B) MLI (2 B) LEN (2 B) Per SL ML Tables Fc t(P IP ort. N Src , IP _A v dd 4, r) Packet arriving On Port N VLANMR Rx. MI Blade QID VLAN Rx. MIBlade. QID MR MR VLAN Rx. MI Blade QID QID MR MR MR VLAN Blade QID VLAN Rx. MI Blade QID Rx. MI MR MR MR VLAN Rx. MI Blade QID Rx. MI MR Tunnel(i) Table) QID (Port. MR N IPv 4 VLAN Blade QID Rx. MI MR N (Port P 2 P-DC Table) VLANMR Rx. MI Blade QID (Port N P 2 P-DC Table) VLANMR Rx. MI Blade QID n There is one P 2 P-Tunnel SL Table per tunnel SL. Blade-to-Blade Ethernet Header MR Specific VLAN Rx. MI (2 B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) … Packet arriving On Port N LC 18 - JST/JDD/FK - 12/31/2021 Switch Recalculate Rxsubstrate MR MI (2 B) LEN (2 B) Meta Frame n Do we want to include Sender’s Mn. Addr in Frame going to MR? » Probably not » IP Routers don’t

LC Packet RX: P 2 P-VLAN 0 Per SL ML Tables VLANMR Rx. MI Blade QID VLAN Rx. MIBlade. QID MR MR VLAN Rx. MI Blade QID QID MR MR MR VLAN Blade QID VLAN Rx. MI Blade QID Rx. MI MR MR MR VLAN Rx. MI Blade QID Rx. MI (Port. MR NMR P 2 P-VLAN 0 Table) VLAN Blade QID Rx. MI MR N (Port P 2 P-DC Table) VLANMR Rx. MI Blade QID (Port N P 2 P-DC Table) VLANMR Rx. MI Blade QID Packet arriving On Port N Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI=VLAN 0 (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) n There is one P 2 P-VLAN 0 SL Table per SL. » Located based on Src. Addr of sender MI (2 B) LEN (2 B) Meta Frame Blade-to-Blade Ethernet Header MR Specific VLAN Rx. MI (2 B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) … Packet arriving On Port N LC 19 - JST/JDD/FK - 12/31/2021 Switch Recalculate Rxsubstrate MR n Do we want to include Sender’s Mn. Addr in Frame going to MR?

LC Packet RX: Multi-Access Per SL ML Tables n VLANMR Rx. MI Blade QID VLAN Rx. MIBlade. QID MR MR VLAN Rx. MI Blade QID QID MR MR MR VLAN Blade QID VLAN Rx. MI Blade QID Rx. MI MR MR MR VLAN Rx. MI Blade QID Rx. MI (Port. MR N MR Multi-Access Table) VLAN Blade QID Rx. MI MR N (Port P 2 P-DC Table) VLANMR Rx. MI Blade QID (Port N P 2 P-DC Table) VLANMR Rx. MI Blade QID Packet arriving On Port N Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI≠VLAN 0 (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) There is one Multi-Access SL Table per port. » Contains the MLI entries for the Multi-Access SL MI (2 B) LEN (2 B) Meta Frame Blade-to-Blade Ethernet Header MR Specific VLAN Rx. MI (2 B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) … Packet arriving On Port N LC 20 - JST/JDD/FK - 12/31/2021 Switch Recalculate Rxsubstrate MR n Do we want to include Sender’s Mn. Addr in Frame going to MR?

Pure Legacy Traffic GE GE LC GE Dst. Addr (6 B) GE Src. Addr (6 B) GE Type=802. 1 Q (2 B) TCI ≠ VLAN 0 (2 B) Type=IP (2 B) Ver/HLen/Tos/Len (4 B) ID/Flags/Frag. Off (4 B) TTL (1 B) Protocol ≠ Substrate (1 B) Hdr Cksum (2 B) Src Addr (4 B) Dst Addr (4 B) IP Packet Body GE GE LC GE . . . GE IXP PE Blade GE GE GE IPv 4 MR GE GE GE PAD (n. B) CRC (4 B) Switch Blade n LC: » If ((Ether. Type=IPv 4) AND (IP Proto ≠ Substrate)) n then non-Tunnel Legacy Ø Send frame to pre-configured default IPv 4 MR 21 - JST/JDD/FK - 12/31/2021 IXP PE Blade GP Blade

LC Packet RX: Legacy IPv 4 Packet arriving Fct(Port. N) On Port N Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI ≠ VLAN 0 (2 B) Type=IP (2 B) Ver/HLen/Tos/Len (4 B) ID/Flags/Frag. Off (4 B) TTL (1 B) 4) v P (I t Fc Per SL ML Tables VLANMR Rx. MI Blade QID VLAN Rx. MIBlade. QID MR MR VLAN Rx. MI Blade QID QID MR MR MR VLAN Blade QID VLAN Rx. MI Blade QID Rx. MI MR MR MR VLAN Rx. MI Blade QID Rx. MI MRNMR (Port Legacy MR Table) VLAN Blade QID Rx. MI MR N (Port P 2 P-DC Table) VLANMR Rx. MI Blade QID (Port N P 2 P-DC Table) VLANMR Rx. MI Blade QID Protocol ≠ Substrate (1 B) Hdr Cksum (2 B) Src Addr (4 B) Dst Addr (4 B) IP Packet Body Recalculate PAD (n. B) CRC (4 B) n There is one Legacy MR Table per Port. » Entries point to Legacy MR for the specified Protocol. » There is at most one Legacy MR for each legacy protocol Blade-to-Blade Ethernet Header MR Specific VLAN Rx. MI (2 B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) … Packet arriving On Port N LC 22 - JST/JDD/FK - 12/31/2021 Switch Recalculate Rxsubstrate MR MI (2 B) LEN (2 B) Meta Frame

MR Txsubstrate MI (2 B) Mn. Flags(1 B) Nh. Addr (n. B) LEN (2 B) VLAN MNid LC … Switch LC Packet TX: All SL Types Blade-to-Blade Ethernet Header (MR specific VLAN) Tx. MI (2 B) Mn. Flags (1 B) Nh. Addr (n. B) LEN (2 B) Meta Frame Mn. Id Mn. ID . . . Arp Cache Mn. ID Key Mn. ID: Mn. Addr If SL_TYPE=Multi-Access Lookup(Mn. ID, Mn. Addr) Mn. ID: Mn. Addr . . . PAD (n. B) CRC (4 B) SL_Type Port . . . SL_Type Port 23 - JST/JDD/FK - 12/31/2021 DAddr Mn. ID: Mn. Addr DAddr SL/ML Table VLAN: Tx. MI SL: MLI Result MLI MLI Hdr Info/Templates MLI Hdr Info/Templates

LC Packet TX – Legacy IPv 4 SL/ML Table SL_Type Port . . . SL_Type Port MLI MLI Hdr Info/Templates MLI Hdr Info/Templates Arp Cache Key Result Mn. ID: Mn. Addr DAddr . . . n n Legacy IPv 4 does use ARP Cache. If DAddr entry for Mn. Addr is missing, send packet up to XScale for it to perform ARP Request. Mn. ID: Mn. Addr Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI ≠ VLAN 0 (2 B) Type=IP (2 B) Ver/HLen/Tos/Len (4 B) ID/Flags/Frag. Off (4 B) TTL (1 B) Blade-to-Blade Ethernet Header (MR specific VLAN) Tx. MI (2 B) Mn. Flags (1 B) Nh. Addr (n. B) LEN (2 B) Protocol ≠ Substrate (1 B) Hdr Cksum (2 B) Src Addr (4 B) Dst Addr (4 B) IP Packet Body Meta Frame PAD (n. B) CRC (4 B) 24 - JST/JDD/FK - 12/31/2021 Recalculate PAD (n. B) CRC (4 B)

Pure Legacy Traffic GE Dst. Addr (6 B) GE Src. Addr (6 B) GE Type=802. 1 Q (2 B) TCI ≠ VLAN 0 (2 B) Type=IP (2 B) Ver/HLen/Tos/Len (4 B) ID/Flags/Frag. Off (4 B) TTL (1 B) Protocol ≠ Substrate (1 B) Hdr Cksum (2 B) Src Addr (4 B) Dst Addr (4 B) IP Packet Body PAD (n. B) CRC (4 B) LC GE GE GE LC . . . IXP PE Blade GE GE GE IPv 4 MR GE GE GE Switch Blade n 25 - JST/JDD/FK - 12/31/2021 IXP PE Blade GP Blade

Substrate/Meta. Net Model n To the Substrate, some Meta Links “Pass Through” » Pass through a Substrate Router without visiting a Meta Router Meta. Link 1 Meta. Link 2 Meta. Link 3 Substrate Link MI MI MI MR_A MI MI MR_B MR_C MI MI Substrate Router X 26 - JST/JDD/FK - 12/31/2021 MR_A MI MI MR_A MI Pass Through Meta Link MI MR_B MI MI MR_C MI Substrate Router Y MR_C MI MI Substrate Router Z

Pass-Through Meta. Link n Packet arriving On Port N n For Pass-Through ML, we need to include Mn. Flags and Nh. Mn. Addr so Tx can handle Multi-Access case Allocate special set of MR/MI for use by Substrate when handling Pass-Through Meta. Links Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI=VLAN 0 (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) … Packet arriving On Port N Of LC X 27 - JST/JDD/FK - 12/31/2021 Blade-to-Blade Ethernet Header (MR specific VLAN) Rx. MI (2 B) Mn. Flags (1 B) Nh. Addr (n. B) LEN (2 B) Blade-to-Blade Ethernet Header (MR specific VLAN) Tx. MI (2 B) Mn. Flags (1 B) Nh. Addr (n. B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) Packet arriving At LC Y LC X Ethernet Switch LC Y

Rx: Pass-Through Meta. Link Per SL ML Tables MR Rx. MI Blade QID MRMR Rx. MIBlade. QID MRMRMR Rx. MI Blade QID QID Rx. MI Blade QID MRMRMR Blade QID Rx. MI MR MR Rx. MI Blade QID Rx. MI (Port N P 2 P-DC Table) MR Rx. MI Blade QID Packet arriving On Port N . . . Dst. Addr (6 B) Src. Addr (6 B) Type=802. 1 Q (2 B) TCI (2 B) Type=Substrate (2 B) MLI (2 B) LEN (2 B) Blade-to-Blade Ethernet Header (MR specific VLAN) Rx. MI (2 B) Mn. Flags (1 B) Nh. Addr (n. B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) Meta Frame Recalculate … Packet arriving On Port N Of LC X 28 - JST/JDD/FK - 12/31/2021 LC X Ethernet Switch LC Y PAD (n. B) CRC (4 B)

… Tx: Pass-Through Meta. Link LC X Ethernet Switch LC Y VLAN MNid Blade-to-Blade Ethernet Header (MR specific VLAN) Tx. MI (2 B) Mn. Flags (1 B) Nh. Addr (n. B) LEN (2 B) Meta Frame PAD (n. B) CRC (4 B) VLAN: Tx. MI SL: MLI Mn. Id Mn. ID . . . Mn. ID Key Mn. ID: Mn. Addr If SL_TYPE=Multi-Access Lookup(Mn. ID, Mn. Addr) Mn. ID: Mn. Addr SL/ML Table SL_Type Port . . . SL_Type Port n n Arp Cache MLI MLI . . . Result DAddr Mn. ID: Mn. Addr DAddr Hdr Info/Templates MLI Hdr Info/Templates This is exactly like the common case shown for all SL Types. After this things proceed as previously shown for the type of SL that is being used for the Transmit. 29 - JST/JDD/FK - 12/31/2021

LC: Functional Blocks Phy Int Rx (2 ME) Phy Int Tx (3 ME) n n Key Extract (1 ME) QM/Schd (1 ME) Hdr Format (1 ME) Lookup (1 ME) Hdr Format (1 ME) QM/Schd (1 ME) Rate Monitor (1 ME) Key Extract (1 ME) Switch Tx (3 ME) Switch Rx (2 ME) ME counts are my first guesses for number needed. For each block I’ll show: » Function: n What this block does. » Memory Accesses: n SRAM: DRAM: n Which fields in Buffer Descriptor the block needs to read and/or write. n » Buffer Descriptor Accesses: » Notes: n Additional thoughts about this block. n Next: » Analyze the SRAM Accesses and try to map them on to the available SRAM Channels. » Analyze the SRAM and DRAM accesses and calculate packet processing rates. 30 - JST/JDD/FK - 12/31/2021 S W I T C H

LC: Functional Blocks n Ingress (Physical Interface Switch): » Phy. Int Rx » Key. Extractor » Lookup » Hdr Format » QM/Scheduler » Switch Tx n Egress (Switch Physical Interface): » Switch Rx » Key. Extractor n This is only extracting the VLAN and the MI. Combine with previous or following block? Ø But it involves a DRAM Read so we probably want to leave it separate and use all 8 threads. » Rate. Monitor n n Before or after the Lookup? Is this different than what QM/Scheduler will/could do? » Lookup » Hdr Format » QM/Scheduler » Phy. Int Tx 31 - JST/JDD/FK - 12/31/2021

LC: Buffer Descriptor n n n Hopefully we can use the same buffer descriptor for the LC and the CRF Processing Engine. » There might be some fields that are used on one and not on the other but that’s ok (MR_ID, Tx. MI, VLAN not needed on LC) PE Buffer Descriptor: » LW 0: buffer_next 32 bits Next Buffer Pointer (in a chain of buffers) » LW 1: offset 16 bits Offset to start of data in bytes » LW 1: Buffer. Size 16 bits Length of data in the current buffer in bytes » LW 2: reserved 8 bits reserved/unused » LW 2: reserved 4 bits reserved/unused » LW 2: free_list 4 bits Freelist ID » LW 2: packet_size 16 bits (Total packet size across multiple buffers) » LW 3: MR_ID 16 bits Meta Router ID » LW 3: Tx. MI 16 bits Transmit Meta Interface » LW 4: VLAN 16 bits VLAN » LW 4: reserved 16 bits reserved/unused » LW 5: reserved 32 bits reserved/unused » LW 6: reserved 32 bits reserved/unused » LW 7: packet_next 32 bits pointer to next packet (unused in cell mode) Leave multi-buffer fields there as a template for the dedicated blade implementation of a jumbo-frame MR. » Also reduces changes to Rx, Tx, and QM and reduces potential problems. n So, far I haven’t found anything extra that we need on LC. 32 - JST/JDD/FK - 12/31/2021 Buffer Descriptor Buffer_Next Buffer_Size Offset Free_List Packet_Size MR_ID Tx. MI VLAN Packet_Next

LC RX: Functional Blocks Phy Int Rx RBUF n Key Extract Lookup Hdr Format QM/Schd Buf Handle(32 b) Rx: Port(8 b) » Function: n Coordinate transfer of packets from RBUF to DRAM n SRAM: » Memory Accesses: Ø Ø Write Buffer Descriptor Free List? n DRAM: Transfer from RBUF n Write/Initialize: Buffer_Next, Buffer_Size, Offset, Free_List, Packet_Size » Buffer Descriptor Accesses: » Notes: n Buffer Handle: Ø Ø n Switch Tx contains the SRAM address of the buffer descriptor. from the SRAM address of the descriptor we can calculate the DRAM address of the buffer data. Offset of where packet starts should be a constant. 33 - JST/JDD/FK - 12/31/2021 S W I T C H Buffer Descriptor Buffer_Next Buffer_Size Offset Free_List Packet_Size MR_ID Tx. MI VLAN Packet_Next

LC RX: Functional Blocks Phy Int Rx Key Extract Buf Handle(32 b) n Port(8 b) Key_Extract: » Function: n Lookup Hdr Format Buf Handle(32 b) Lookup Key(16 B) Really only needs to be max of 10 B (see next slide for view of Keys) SL Type (4 b) Extracts lookup key based on type of frame that was received. » Memory Accesses: n DRAM: Ø n Read as much of header as is necessary to extract key – May vary depending on type of Substrate Link SRAM: None » Buffer Descriptor Accesses: None » Notes: n n QM/Schd S W I T C Buffer H Switch Tx Descriptor Buffer_Next Buffer_Size Offset Free_List Packet_Size MR_ID Tx. MI VLAN Packet_Next Calculates DRAM Address based on SRAM descriptor address in buffer handle and the Offset passed to it by RX. Frame offset in buffer is a constant and does not need to be read from Buffer Descriptor 34 - JST/JDD/FK - 12/31/2021

LC RX: Functional Blocks n Substrate Link Type: n Lookup Keys: » Will be used as Database ID by Lookup Block SL Type (SL Type ID) (size) P 2 P-DC(0 x 0) (28 bits) Port(8 b) P 2 P-Tunnel - IPv 4(0 x 1) (76 bits) Port (8 b) MLI(16 b) Ether. Type (16 b) 0 x 0800 P 2 P-VLAN 0(0 x 2) (76 bits) Port (8 b) MA(0 x 3) (28 bits) Port(8 b) Physical Interface # Legacy IPv 4(0 x 4) (28 bits) Port (8 b) 35 - JST/JDD/FK - 12/31/2021 IP SAddr (32 b) Ethernet SAddr (48 b) MLI(16 b) Ether. Type (16 b) 0 x 0800 MLI (16 b)

LC RX: Functional Blocks Phy Int Rx n Lookup: Key Extract Lookup Buf Handle(32 b) Lookup Key(16 B) Lookup Result (16 B) » Function: n Hdr Format QM/Schd Switch Tx SL Type (4 b) Performs Lookup and passes result on to Hdr Format. » Memory Accesses: n n DRAM: None SRAM: Ø TCAM Lookup – Write Lookup command – Read Lookup Result (1 -5 words from TCAM SRAM Controller or other SRAM Controller) » Buffer Descriptor Accesses: None » Notes: n n Lookup does no processing on the lookup result. Need to decide how lookup result will be stored and retrieved. Ø See notes on TCAM for information about the issues involved. 36 - JST/JDD/FK - 12/31/2021 S W I T C H

LC RX: Functional Blocks Phy Int Rx n Key Extract Hdr Format: Lookup Hdr Format Buf Handle(32 b) Lookup Result (16 B) Size (16 b) » Function: n From lookup result: Ø Ø QM/Schd n re-writes headers in DRAM to make frame ready to transmit. Extract QID to pass on to QM/Scheduler DRAM: SRAM: Ø Ø Read Descriptor and Re-Write Descriptor – OR Atomic Increment/Decrement some fields in Descriptor » Buffer Descriptor Accesses: n Update Size and Offset fields » Notes: n n Pass Size on to QM/Scheduler so it does not have to read buffer descriptor for Enqueue to update Q Length. Buffer Offset should be a constant and should not need to be read from Buffer Descriptor 37 - JST/JDD/FK - 12/31/2021 S W I T C H QID(24 b) » Memory Accesses: n Switch Tx Buffer Descriptor Buffer_Next Buffer_Size Offset Free_List Packet_Size MR_ID Tx. MI VLAN Packet_Next

LC RX: Functional Blocks Phy Int Rx Key Extract Lookup Hdr Format QM/Schd Buf Handle(32 b) Size (16 b) QID(24 b) n QM/Scheduler (See Sailesh’s slides for more details) » Function: n n n Enqueue and Dequeue from queues Scheduling algorithm Drop Policy » Memory Accesses: n n DRAM: None SRAM: Ø Ø Q-Array Reads and Writes Scheduling Data Structure Reads and Writes QLength Data Structure Reads and Writes Dequeue: Read Buffer Descriptor to retrieve Packet Size » Buffer Descriptor Accesses: Read packet size » Notes: 38 - JST/JDD/FK - 12/31/2021 Switch Tx Buf Handle(32 b) S W I T C H

LC RX: Functional Blocks Phy Int Rx Key Extract Lookup Hdr Format QM/Schd Buf Handle(32 b) n Switch Tx TBUF Switch TX: » Function: n Coordinate transfer of packets from DRAM to TBUF » Memory Accesses: n n SRAM: Read Buffer Descriptor DRAM: Transfer to TBUF » Buffer Descriptor Accesses: n Read Size and Offset » Notes: n Calculate DRAM address based on SRAM Descriptor address in buffer handle 39 - JST/JDD/FK - 12/31/2021 S W I T C H

LC TX: Functional Blocks Phy Int Tx QM/Schd Hdr Format Lookup Rate Monitor Key Extract Buf Handle(32 b) n Switch Rx RBUF Offset (16 b) Rx: » Function: n Coordinate transfer of packets from RBUF to DRAM » Memory Accesses: n n SRAM: Write Buffer Descriptor DRAM: Transfer from RBUF » Buffer Descriptor Accesses: n Write/Initialize: Buffer_Next, Buffer_Size, Offset, Free_List, Packet_Size » Notes: n Buffer Handle: Ø Ø n contains the SRAM address of the buffer descriptor. from the SRAM address of the descriptor we can calculate the DRAM address of the buffer data. Passing the offset of where the packet starts in memory will save the next block from having to read the buffer descriptor. Ø Perhaps we should just pass the actual DRAM Buffer Pointer? 40 - JST/JDD/FK - 12/31/2021 S W I T C H Buffer Descriptor Buffer_Next Buffer_Size Offset Free_List Packet_Size MR_ID Tx. MI VLAN Packet_Next

LC TX: Functional Blocks Phy Int Tx QM/Schd Hdr Format Lookup Rate Monitor Buf Handle(32 b) Lookup Key n Key_Extract: Key Extract Switch Rx Buf Handle (32 b) VLAN (16 b) S W I T C H Tx. MI (16 b) » Function: n Extracts lookup key based on type of frame that was received. » Memory Accesses: n DRAM: Ø n Read VLAN and Tx. MI from Frame SRAM: None » Buffer Descriptor Accesses: None » Notes: n Calculates DRAM Address based on SRAM descriptor address in buffer handle and the Offset passed to it by RX. 41 - JST/JDD/FK - 12/31/2021

LC TX: Functional Blocks Phy Int Tx n QM/Schd Rate Monitor: Hdr Format Lookup Rate Monitor Key Extract Buf Handle(32 b) Offset (16 b) VLAN (16 b) Tx. MI (16 b) Switch Rx » Function: n n Ensures that MR/MI’s behave according to their Rate Specs. Does this need to be a separate function from the QM/Scheduler? » Memory Accesses: Unknown at this point n n DRAM: SRAM: » Buffer Descriptor Accesses: Unknown at this point » Notes: 42 - JST/JDD/FK - 12/31/2021 S W I T C H

LC TX: Functional Blocks Phy Int Tx n QM/Schd Lookup: Hdr Format Lookup Key Extract Buf Handle (32 b) Buf Handle(32 b) Offset (16 b) Lookup Result (16 B) VLAN (16 b) » Function: n Rate Monitor Switch Rx Tx. MI (16 b) Performs Lookup and passes result on to Hdr Format. » Memory Accesses: n n DRAM: None SRAM: Ø TCAM Lookup – Write Lookup command – Read Lookup Result (1 -5 words from TCAM SRAM Controller or other SRAM Controller) » Buffer Descriptor Accesses: None » Notes: n n Lookup does no processing on the lookup result. Need to decide how lookup result will be stored and retrieved. Ø See notes on TCAM for information about the issues involved. 43 - JST/JDD/FK - 12/31/2021 S W I T C H

LC TX: Functional Blocks Phy Int Tx n QM/Schd Hdr Format Lookup Buf Handle(32 b) Buf Handle (32 b) Size (16 b) Offset (16 b) QID(24 b) Lookup Result (16 B) Rate Monitor Key Extract Switch Rx Hdr Format: » Function: n From lookup result: Ø Ø re-writes headers in DRAM to make frame ready to transmit. Extract QID to pass on to QM/Scheduler » Memory Accesses: n n DRAM: SRAM: Ø Ø Read Descriptor and Re-Write Descriptor – OR Atomic Increment/Decrement some fields in Descriptor » Buffer Descriptor Accesses: n Update Size and Offset fields n Pass Size on to QM/Scheduler so it does not have to read buffer descriptor for Enqueue to update Q Length. » Notes: 44 - JST/JDD/FK - 12/31/2021 S W I T C H Buffer Descriptor Buffer_Next Buffer_Size Offset Free_List Packet_Size MR_ID Tx. MI VLAN Packet_Next

LC TX: Functional Blocks Phy Int Tx QM/Schd Buf Handle(32 b) Hdr Format Lookup Rate Monitor Key Extract Buf Handle(32 b) Size (16 b) QID(24 b) n QM/Scheduler (See Sailesh’s slides for more details) » Function: n n n Enqueue and Dequeue from queues Scheduling algorithm Drop Policy » Memory Accesses: n n DRAM: None SRAM: Ø Ø Q-Array Reads and Writes Scheduling Data Structure Reads and Writes QLength Data Structure Reads and Writes Dequeue: Read Buffer Descriptor to retrieve Packet Size » Buffer Descriptor Accesses: Read packet size » Notes: 45 - JST/JDD/FK - 12/31/2021 Switch Rx S W I T C H Buffer Descriptor Buffer_Next Buffer_Size Offset Free_List Packet_Size MR_ID Tx. MI VLAN Packet_Next

LC TX: Functional Blocks Phy Int Tx QM/Schd TBUF n Switch Hdr Format Lookup Rate Monitor Key Extract Buf Handle(32 b) TX: » Function: n Coordinate transfer of packets from DRAM to TBUF » Memory Accesses: SRAM: Read Buffer Descriptor n DRAM: Transfer to TBUF n » Buffer Descriptor Accesses: n Read Size and Offset » Notes: n Calculate DRAM address based on SRAM Descriptor address in buffer handle 46 - JST/JDD/FK - 12/31/2021 Switch Rx S W I T C H Buffer Descriptor Buffer_Next Buffer_Size Offset Free_List Packet_Size MR_ID Tx. MI VLAN Packet_Next

ML Loopback: Functional Blocks Phy Int Rx Key Extract Lookup Hdr Format QM/Schd Switch Tx S W I T C H Loopback Hdr Re-Format Phy Int Tx n n QM/Schd Hdr Format Lookup Rate Monitor Key Extract Switch Rx Loopback path should be able to re-use some of the blocks implemented for the LC Loopback Hdr Re-Format: » Needs to be able to strip off the previous MR’s Header. » For Plain-IP IPv 4 MR Y n n When frame arrives at Loopback (between IPv 4 MR and MR Y) it will still have IP Header which should be stripped off before frame is sent to MR Y. Lookup Result should probably include a length field or a Buffer offset field that indicates where new Meta Frame should start. 47 - JST/JDD/FK - 12/31/2021

LC: Notes on TCAM Lookups n See tech. X_Design_TCAM_Usage. ppt slides for notes on how the TCAM will be used by the Lookup Block 48 - JST/JDD/FK - 12/31/2021

Extra n The next set of slides are for templates or extra information if needed 49 - JST/JDD/FK - 12/31/2021

Text Slide Template 50 - JST/JDD/FK - 12/31/2021

Image Slide Template 51 - JST/JDD/FK - 12/31/2021

LC: SRAM Accesses SRAM Ch#A n n n n n SRAM Ch#C Ch#D SRAM Ch#A Ch#F Ch#H SRAM Ch#A Hdr Switch Phy Int Key Lookup Format Tx QM/Schd Rx Extract (1 ME) (2 ME) (1 ME) SRAM Ch #A: SRAM Buffer Descriptors for Ingress (Phy Int Switch) SRAM Ch #B: SRAM Buffer Descriptors for Egress (Switch Phy Int) SRAM Ch #C: TCAM Access SRAM Ch #D: Lookup Associated Data for Ingress (Phy Int Switch) SRAM Ch #E: Lookup Associated Data for Egress (Switch Phy Int) SRAM Ch #F: Q-Array for Ingress (Phy Int Switch) QM SRAM Ch #G: Q-Array for Egress (Switch Phy Int) QM SRAM Ch #H: QM Scheduling/Dequeue Data Structure SRAM Ch #I: QM Scheduling/Dequeue Data Structure Phy Int Tx (1 ME) SRAM Ch#B QM/Schd (1 ME) Hdr Format (1 ME) Lookup (1 ME) SRAM SRAM Ch#B Ch#G Ch#I Ch#B Ch#C Ch#E 52 - JST/JDD/FK - 12/31/2021 Rate Monitor (1 ME) Key Extract (1 ME) Switch Rx (2 ME) SRAM Ch#B S W I T C H

LC: Number of Per Pkt SRAM Accesses n Rx: n Lookup » 1 SRAM Buffer Descriptor WRITE (A, B) » TCAM: n n 1 SRAM WRITE of TCAM Lookup Command (C, C) 1 -8 SRAM READs of Results Mailbox (C, C) Ø Ø 1 READ if we are just retrieving an Index or Pointer Up to 8 locations to read if we are retrieving the actual result data from Results Mailbox » Associated Data: n Read of up to 8 32 -bit words in consectutive SRAM locations (D, E) n Hdr Format: n QM/Sched » 1 SRAM Buffer Descriptor WRITE (A, B) » Q-Array: n Enqueue: <= 1 SRAM Read Queue Descriptor (F, G) Dequeue: <= 1 SRAM Read Queue Descriptor (F, G) n Enqueue: n » SRAM Buffer Descriptors: Ø 1 Buffer Descriptor Read (A, B) 1 Buffer Descriptor Write (A, B) Ø 1 Buffer Descriptor Read (A, B) Ø n Dequeue » SRAM for Scheduling Data Structure n Enqueue: n Dequeue Ø Ø n Tx: Ø 1 Write to add new queue which just became active (H, I) 1 Read (H, I) 1 Write (H, I) » 1 SRAM Buffer Descriptor READ (A, B) » Return Buffer Descriptor to Free List 53 - JST/JDD/FK - 12/31/2021

LC: SRAM Accesses n Buffer Descriptor Accesses (A, B): 3 Writes, 3 Reads » Rx: Write initial descriptor » Hdr Format: Write descriptor to update sizes, offsets, … » QM/Scheduler: n n n Enqueue Read Descriptor to get pkt size to do drop policy check Enqueue Write Descriptor to update next pkt pointer (how does Q-Array do this for us? ) Dequeue Read Descriptor to get pkt size to do credit check? » Tx: Read Descriptor 54 - JST/JDD/FK - 12/31/2021

LC: Number of Per Pkt SRAM Accesses n Channel A: n Channel B: n Channel C: n Channel D: n Channel E: n Channel F: n Channel G n Channel H: n Channel I: » Ingress: W, R, R » Egress: W, R, R » Ingress: W, 1 -8 R, » Egress: W, 1 -8 R, » Ingress: <=8 R » Egress: <= 8 R » Ingress: R, W, R, W » Egress: R, W, R, W » Ingress: W, R, W » Egress: W, R, W 55 - JST/JDD/FK - 12/31/2021

LC: Functional Blocks Phy Int Rx (2 ME) Key Extract (1 ME) Hdr Format (1 ME) Lookup (1 ME) Input Hlpr (1 ME) QM/Schd (1 ME) . . . Output Hlpr (1 ME) Switch Tx (2 ME) QM/Schd (1 ME) Scratch Rings NN Ring Phy Int Tx (1 ME) QM/ Sch Hlpr (1 ME) QM/Schd (1 ME) . . . QM/ Sch Hlpr (1 ME) Hdr Format (1 ME) QM/Schd (1 ME) Scratch Rings NN Ring 56 - JST/JDD/FK - 12/31/2021 NN Ring n Lookup (1 ME) Rate Monitor (1 ME) Multiple QM/Scheds Key Extract (1 ME) Switch Rx (2 ME) » Meta. Links assigned to QM/Sched to balance load based on their allocated BW » Input Helper ME demuxes from NN ring from Hdr Format to multiple Scratch Rings » Output Helper ME muxes from multiple Scratch Rings to NN to TX S W I T C H

OLD n The rest of these are old slides that should be deleted at some point. 57 - JST/JDD/FK - 12/31/2021

LC: Notes on TCAM Lookups n The following slides show a way to use the TCAM for the lookups. n Slight adjustments might be desirable depending on: » Ease of doing operations on non-byte length bit fields » What we learn about methods for using the TCAM. n Field and Identifier sizes: » MN id: 32 bits » MI id: 16 bits (64 K Meta Interfaces per Meta Router) » MLI: 16 bits (64 K Meta Links per Substrate Link) » Port: 8 bits (256 Physical Interfaces per Line Card) » QID: 20 bits (1 M Queues per Queue Manager) » QM ID: 4 bits (16 Queue Managers per LC or PE. ) n We probably can only support 4 QMs (2 bits) Ø (64 Q-Array Entries) / (16 CAM entries) 4 QMs per SRAM Controller. 58 - JST/JDD/FK - 12/31/2021

LC: Notes on TCAM Lookups n Lookup Key size options: » 32/36, 64/72, 128/144, 256/288, 512/576 (all in bits) n Lookup Result options: » Absolute Index: relative to beginning of TCAM array » Database Relative Index: relative to beginning of selected database » Memory Pointer Index: points to SRAM location of result data » Associated Data: 32, 64 or 128 bits of data associated with the lookup result. n Associated Data is stored in ZBT SRAM attached to TCAM. n TCAM Databases » How many to use? 1 for TX and 1 for RX? n 1 for TX and 1 for each of the SL Types on Rx (5 types)? n Other… n 59 - JST/JDD/FK - 12/31/2021

LC: TCAM Lookup Keys on RX n n n P 2 P-DC(28 b): SL_Type(4)/Port(8)/MLI(16) P 2 P-Tunnel(IPv 4)(76 b): SL_Type(4)/Port(8)/Ether. Type(16)/IPSrc(32)/MLI(16) P 2 P-VLAN 0(76 b): SL_Type(4)/Port(8)/Eth. SAddr(48)/MLI(16) MA(28 b): SL_Type(4)/Port(8)/MLI(16) Legacy(28 b): SL_Type(4)/Port(8)/EType(16) Fields: » SL_Type (4 b): Substrate Link Type n n n 0000: 0001: 0010: 0011: 0100: 0101: DC IPv 4 Tunnel VLAN 0 MA Legacy (non-substrate) without VLAN Legacy (non-substrate) with VLAN » Port(8 b): Physical Interface number » MLI(16 b): Meta Link Identifier » Ethertype(16 b): Ethernet Header Type field » IPSrc(32 b): IP Source Address » Eth. SAddr(48 b): Ethernet Header Source Address 60 - JST/JDD/FK - 12/31/2021

LC: TCAM Lookup Keys on RX SL(4 b) 0000 Port(8 b) SL (4 b) Port 0001 (8 b) SL (4 b) Port 0010 (8 b) SL(4 b) 0011 Port(8 b) SL (4 b) Port 0100 (8 b) MLI(16 b) Ether. Type (16 b) 0 x 0800 PAD(100 b) IP SAddr (32 b) Ethernet SAddr (48 b) MLI (16 b) PAD (52 b) MLI(16 b) PAD(100 b) Ether. Type (16 b) 0 x 0800 PAD (100 b) 61 - JST/JDD/FK - 12/31/2021

LC: TCAM Lookup Results on RX n Standard Fields (116 b): » Type (4 b) n n n 0000: Default, not Pass Through, ignore Mn. Flags 1000: Pass Through with no extra lookup needed for Nh. Addr(n. B), Mn. Flags(8 b) should be 0 x 00 1001: Pass Through with extra lookup needed for Nh. Addr(n. B) » VLAN (16 b) » MI (16 b) » Blade Eth Hdr (48 b) n n Only needs to have the DAddr. SAddr can be configured and constant per LC First Ether. Type can be constant: 802. 1 Q Second Ether. Type can be constant: Substrate » QID (24 b) » Mn. Flags(8 b) n Can we say there will be no Pass Through Meta Links where one side will be on a Multi Access and hence might need a Nh. Addr field? Pass Through Meta Link Fields: n Even a 32 b IP Nh. Addr would not fit in a 128 b TCAM Result. n » Mn. Flags(8 b) » Nh. Addr(n. B) » The TCAM supports Multi-Hit Lookups: Need to better understand what this means. » Do a second lookup with same key except SL_Type=1111 b, to retrieve the Next Hop bits? » Or do a second lookup with the following fields to retrieve the Next Hop bits? n n SL_Type (4 b) = 1111 b Port (8 b) VLAN (16 b) MI (16 b) 62 - JST/JDD/FK - 12/31/2021

LC: TCAM Lookups on TX n Key: » VLAN(16 b) » Tx. MI(16 b) n Result » The Lookup Result for TX will consist of several parts: n n Lookup Result Constant fields Calculated fields Fields that can be stored in Local Memory » Some of these are common across all SL Types » Other fields are specific to each SL Type » Common across all SL Types (100 b): n From Result (44 b) Ø Ø n SL Type(4 b) Port(8 b) MLI(16 b) QID (24 b) Local Memory (48 b) Ø Eth Hdr SA (48 b) : tied to Port » SL Type Specific Headers n n P 2 P-DC Hdr: P 2 P-MA Hdr: P 2 P-VLAN 0 Hdr P 2 P-Tunnel Hdr for IPv 4 Tunnel 63 - JST/JDD/FK - 12/31/2021

LC: TCAM Lookups on TX n Key: » VLAN(16 b) » Tx. MI(16 b) n Result » Common across all SL Types (100 b): n From Result (52 b) Ø Ø n SL Type(4 b) Port(8 b) MLI(16 b) QID (24 b) Local Memory (48 b) Ø Eth Hdr SA (48 b) : tied to Port » SL Type Specific Headers n P 2 P-DC Hdr (64 b) Ø Ø Ø Constant (16 b) – Ether. Type (16 b) = Substrate Calculated (0 b) From Result (48 b) – Eth DA (48 b) Lookup Result Total (Common From Result + Specific From Result): 100 bits Total (Common + Specific) : 164 bits 64 - JST/JDD/FK - 12/31/2021

LC: TCAM Lookups on TX n Key: » VLAN(16 b) » Tx. MI(16 b) n Result » Common across all SL Types (100 b): n From Result (52 b) Ø Ø n SL Type(4 b) Port(8 b) MLI(16 b) QID (24 b) Local Memory (48 b) Ø Eth Hdr SA (48 b) : tied to Port » SL Type Specific Headers n P 2 P-MA Hdr (64 b) : Ø Ø Ø Constant (16 b) – Ether. Type (16 b) = Substrate Calculated (0 b) From Result (48 b) – Eth DA (48 b) Lookup Result Total (Common From Result + Specific From Result): 100 bits Total (Common + Specific) : 164 bits 65 - JST/JDD/FK - 12/31/2021

LC: TCAM Lookups on TX n Key: n Result » VLAN(16 b) » Tx. MI(16 b) » Common across all SL Types (100 b): n From Result (52 b) Ø SL Type(4 b) Port(8 b) MLI(16 b) QID (24 b) Ø Eth Hdr SA (48 b) : tied to Port Ø Ø Ø n Local Memory (48 b) » SL Type Specific Headers n P 2 P-VLAN 0 Hdr (80 b): Ø Ø Ø Constant (16 b) – Ether. Type 1 (16 b) = 802. 1 Q – Ether. Type 2 (16 b) = Substrate Calculated (0 b) From Result (64 b) – Eth DA (48 b) – TCI (16 b) Lookup Result Total (Common From Result + Specific From Result): 116 bits Total (Common + Specific) : 180 bits 66 - JST/JDD/FK - 12/31/2021

LC: TCAM Lookups on TX n Result (continued) » Common across all SL Types (100 b): n From Result (52 b) Ø SL Type(4 b) Port(8 b) MLI(16 b) QID (24 b) Ø Eth Hdr SA (48 b) : tied to Port Ø Ø Ø n Local Memory (48 b) » SL Type Specific Headers n P 2 P-Tunnel Hdr for IPv 4 Tunnel (224 b): Ø Ø Ø Constant (64 b) – Eth Hdr Ether. Type (16 b) = 0 x 0800 – IPHdr Version(4 b)/HLen(4 b)/Tos(8 b) (16 b): All can be constant? – IP Hdr Flags(3 b)/Frag. Off(13 b) (16 b) : what is our stance on Fragments? If never used, these are constants, if it is possible we will have to use them, then this has to be calculated. Either way, shouldn’t be in Result – IP Hdr TTL (8 b): Constant – IP Hdr Proto (8 b) = Substrate Calculated (48 b) – IP Hdr Len(16 b) : needs to be calculated for each packet sent, so shouldn’t be in Result. – IP Hdr ID(16 b): should be unique for each packet sent, so shouldn’t be in Result. – IP Hdr Checksum (16 b): Needs to be calculated, so shouldn’t be in Result. Local Memory (32 b) – IP Hdr Src Addr (32 b) : tied to port From Result (80 b) – Eth Hdr DA (48 b) – IP Hdr Dst Addr (32 b) Lookup Result Total (Common From Result + Specific From Result): Total (Common + Specific) : 324 bits 67 - JST/JDD/FK - 12/31/2021 132 bits

LC: TCAM Lookups on TX n Key: n Result » VLAN(16 b) » Tx. MI(16 b) » Common across all SL Types (100 b): n From Result (52 b) Ø SL Type(4 b) Port(8 b) MLI(16 b) – Ignored for Legacy Traffic QID (24 b) Ø Eth Hdr SA (48 b) : tied to Port Ø Ø Ø n Local Memory (48 b) » SL Type Specific Headers n Legacy (IPv 4) with VLAN Hdr (96 b): Ø Ø Ø Constant (16 b) – Ether. Type 1 (16 b) = 802. 1 Q Calculated (0 b) ARP Lookup on Nh. Addr (48 b) – Eth DA (48 b) From Result (32 b) – Ether. Type 2 (16 b) = IPv 4 – TCI (16 b) Lookup Result Total (Common From Result + Specific From Result): 84 bits Total (Common + Specific) : 196 bits 68 - JST/JDD/FK - 12/31/2021

LC: TCAM Lookups on TX n Key: n Result » VLAN(16 b) » Tx. MI(16 b) » Common across all SL Types (100 b): n From Result (52 b) Ø SL Type(4 b) Port(8 b) MLI(16 b) – Ignored for Legacy Traffic QID (24 b) Ø Eth Hdr SA (48 b) : tied to Port Ø Ø Ø n Local Memory (48 b) » SL Type Specific Headers n Legacy (IPv 4) without VLAN Hdr (64 b): Ø Ø Ø Constant (0 b) Calculated (0 b) ARP Lookup on Nh. Addr (Is ARP cache another database in TCAM? ) (48 b) – Eth DA (48 b) From Result (16 b) – Ether. Type (16 b) = IPv 4 Lookup Result Total (Common From Result + Specific From Result): 68 bits Total (Common + Specific) : 164 bits 69 - JST/JDD/FK - 12/31/2021

SUMMARY: LC: TCAM Lookups DC Key RX MA Legacyw / vlan Legacy w/o vlan 76 76 28 28 28 116 116 116 32 32 32 100 132 116 100 84 68 Result in Loc. Mem 48 80 48 48 Combined 148 212 164 148 132 116 Key n n VLAN 0 28 Result TX Tunnel Rx Key Size: 128 bits Rx Result Size: 128 bits Tx Key Size: 32 bits Tx Result Size: 256 bits (we should try to squeeze this in to 128 bits!) » What if QID went from 24 to 22 bits. n n 2 bits for QM id 20 bits for QID: 1 M queues per QM instance, 4 M Queues total. » And Port went from 8 bits to 6 bits (64 physical interfaces per Line Card) n If we can’t use the 128 bit Result size for Tx, we might as well include the Local Memory result in the TCAM lookup. » The Local Memory result was going to be the data that was only dependent on the Physical Interface id and was going to go into Local memory to try to keep the TCAM result below 128 bits. 70 - JST/JDD/FK - 12/31/2021