Trident 3 X 7 ASIC Packet Walk and

Trident 3 -X 7 ASIC Packet Walk and Platform Overview Faraz Taifehesmatian Technical Marketing Engineer, INSBU September 2018

Agenda © 2017 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1 ASIC Architecture 2 Forwarding 3 Buffering 4 CT vs SF 5 Platform Overview

Trident 3 ASIC Architecture © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident 3 ASIC © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • ASIC Block and Architecture • ASIC Port Group • Forwarding Pipeline • Port Numbering • Single Pipeline Block • Fle. XGS Ingress Pipeline • Memory Management Unit • Cross-Point Element (XPE) • Queuing and Scheduling • ASIC Comparison

Trident 3 ASIC Block 10 G XFI Falcon. Core 18 Falcon. Core 17 25 G/50 G Falcon. Core 19 Falcon. Core 20 Falcon. Core 21 Falcon. Core 22 100 G KR 4/SR 4 40 G XLAUI Falcon. Core 23 Falcon. Core 24 Falcon. Core 25 Falcon. Core 26 Falcon. Core 27 Falcon. Core 29 Falcon. Core 28 Falcon. Core 30 Falcon. Core 31 Falcon. Core 32 Multistage Content aware Engine Packet Buffer L 2 Processing VLAN PFC Virtual Machine L 3 Processing FCOE Forwarding Engine Virtual Machine Switching Merlin. Core x 4 PCIe 10 -100 G MAC Falcon. Core 1 Falcon. Core 2 Falcon. Core 3 Falcon. Core 4 Falcon. Core 5 Falcon. Core 6 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Falcon. Core 7 Falcon. Core 8 HG (42) Falcon. Core 9 Falcon. Core 10 Falcon. Core 11 Falcon. Core 12 Falcon. Core 13 HG (106) Falcon. Core 14 Falcon. Core 15 Falcon. Core 16

Trident 3 ASIC Architecture • BCM 56870 from Strata. XGS family • 3. 2 Tbps Single Chip Ethernet Switch • 2 Pipes @1. 6 Tbps (BCM 56870) Dynamic Memory Manager (8 Co. S, SP/WRR/WDRR) Fully Shared 32 MB Buffer Multistage Content Aware Engine (Classify, Filter, Modify, Parallel Lookup) • 32 MB of Buffer • 32 Falcon Cores • 1 Merlin Core L 2 MAC L 2/L 3 Processing VLAN, VRF, VSI, ECMP Hashing/ Lead Balancing Engine 128 x 25 G Ser. Des Merlin. Core L 2/L 3 Host/ARP L 3 Processing Flex Counters CPU PCIe ………… 32 x 100 GE, 64 x 40 GE/50 GE 128 x 10 GE/25 GE © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10 GE, 2. 5 GE, 1 GE 4 x Gen 3

Trident 3 ASIC Port Groups • Trident 3 has two packet processing pipelines • Each pipeline has 16 Falcon. Cores Ser. Des • The group of 16 Falcon. Core Ser. Des are further subdivided into two groups of eight Falcon. Core Ser. Des • Each group of 8 Falcon. Core Ser. Des shares ¼ of the device’s total forwarding bandwidth © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential FC 4 FC 5 FC 6 FC 7 FC 8 FC 9 FC 10 FC 11 FC 3 FC 12 FC 13 Packet Processing Pipeline 0 FC 14 FC 0 Group 1 FC 15 FC 31 Group 3 Group 2 FC 16 FC 30 FC 17 Packet Processing Pipeline 1 FC 29 FC 18 FC 28 FC 19 FC 27 FC 26 FC 25 FC 24 FC 23 FC 22 FC 21 FC 20

Trident 3 ASIC Forwarding Pipeline Pipe 1 Ser. Des x 8 Falcon. Core x 8 Ser. Des x 8 1 1 8 8 IP 0 Falcon. Core EP 0 9 9 16 Pipe 2 Ser. Des Falcon. Core 17 Ser. Des 24 Falcon. Core 25 32 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Ser. Des x 8 17 IP 1 Ser. Des x 8 Falcon. Core 16 MMU x 8 Falcon. Core EP 1 Falcon. Core 24 25 x 8 Falcon. Core 32 Ser. Des

Trident 3 ASIC Port Numbering MMU EP 0 CPU Port Macro (0) … Port(7)Macro IP 0 Port Macro (7) Local Port # 0 1 -4 29 -32 33 -36 Global Port # 0 1 -4 … 29 -32 33 -36 Group 0 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential EP 1 Port Macro (0) … Port(0)Macro LPBK 61 -64 65 0 -3 61 -64 130 65 -68 … Group 1 … Port(7)Macro … Group 2 IP 1 Port Macro (7) 28 -31 32 -35 93 -96 97 -100 … Port(0)Macro … Group 3 MGT LPBK 60 -63 63* 64 65 125 -128 128* 129 131

Trident 3 ASIC Single Pipeline Block Fle. XGS Ingress Pipeline 0 Port Macro Parser Lookup Engine(s) Special Functions, ECMP, Hash Memory Management Unit (MMU) Buffer Scheduler Fle. XGS Egress Pipeline 0 Port Macro © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Editor Control Lookup Engine(s)

Trident 3 Flex Database Reconfiguration Fle. XGS Ingress Pipeline 0 Parser Lookup Engine(s) Unified Adaptation Table Adaptation/Tunneling Forwarding Translations Lookup Engine(s) Unified Forwarding Table Forwarding Translations Classification/Policy Lookup Engine(s) Special Functions, ECMP, Hash Filed Processor Classification/Policy Adaptation/Tunneling Forwarding Each Centralized Database in the Programmable Ingress Pipe can be repurposed for Multiple functions © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident 3 ASIC Memory Management Unit (MMU) Buffering Cells Queuing Scheduling Egress Pipeline 0 Ingress Pipeline 0 … … Ingress Pipeline 1 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Egress Pipeline 1

Trident 3 MMU Cross-Point Element (XPE) • Provides inputs for two sets of ingress ports • Provides full outputs for two sets of egress ports XP E Ingress • Consists of a dedicated 32 MB data buffer space Egress controls, service pools and other accounting resources Ingress Egress • Keeps its own Ingress and egress admission • Consists of separate 10 UC and 10 MC queues for each egress port. These are called physical queues © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Egress

Trident 3 Physical and Logical Queues • Each egress port has: • 10 physical UC queues • 10 physical MC queues • For each UC queues or MC queues, a logical queue is formed using Deficit Round Robin (DRR) scheduler • Scheduling capabilities are then applied to logical queues Logical UC Queue UC COS 0 UC Traffic UC COS 0 DR R Ingress Ports MC Traffic Logical MC Queue MC COS 0 DR R © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Unicast Logical Queue Multicast Logical Queue

Trident 3 Egress Port Scheduling Hierarchy • There are 4 Levels: • Level 2 QM • Level 1 SC • Level 0 COS 7 • Port Level COS 6 Physical Logical Queue L 2 Queue L 1 COS L 0. 9 SP+ WDRR L 0. 8 SP+ WDRR […] SP+ WDRR UC COS 0 DR R Ingress Port MC COS 0 DR R © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Port L 0. 0 SP+ WDRR

Trident 3 Egress Port Scheduler Summary Level Nodes/Queues Per Port Total Per ASIC Total L 2 Physical Queues 20 2580 L 1 Logical Queues 20 2580 L 0 Cos Level 10 1290 Port Level 1 129 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Egress Port Scheduler, Level 2 • Physical queues • Separate UC and MC queues • • 20 physical queues per (egress port, XPE), 10 UC queues and 10 MC queues It takes two XPEs to make up a complete egress port. Therefore, each egress port consists of 10 pairs of UC queues and 10 pairs of MC queues © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • 25% There are 5, 200 physical queues for 128 egress port and 2 management ports. • • (128 + 2) * 40= 5, 200 physical queues 2, 600 are UC queues and 2, 600 are MC queues

Egress Port Scheduler, Level 1 • • • Logical Queue Level Schedule packets from two physical queues belonging to two XPEs (XPE A and XPE B in the same MMU slice) for a given egress port. DRR is used as a scheduling discipline at each logical queue level scheduling node. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 50%

Egress Port Scheduler, Level 0 • • • Co. S Level Each Cos level scheduling node has two children ( Level 1 scheduling nodes). One is for logical UC queue and the other one is for logical MC queue. Support baseline scheduling disciplines © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • 75% When Strict priority scheduling discipline is used, the logical UC queue has higher priority than logical MC queue.

Egress Port Scheduler, Port Level • • • Each port-level scheduling node has 10 children ( L 0 -level scheduling nodes). Support baseline scheduling disciplines. The order of the Co. S-level (L 0) scheduling nodes that are bound to the port-level scheduling node is fixed. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • 100% If Strict priority scheduling discipline is used at the port level, the order of the priority will be in descending order from L 09 to L 0 -0

Trident 3 Baseline Scheduling Disciplines • Strict Priority (SP) • Weighted. Round. Robin(WRR) • Weighted. Deficit. Round. Robin(WDRR) • Strict Priority + WRR • Strict Priority + WDRR • • When a scheduling node is configured as WRR or WDRR, its child’s weight can be set between 1 to 1127 If the weight is set to 0, the queue or scheduling node is scheduled using strict priority © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential L 2 Child L 1 Parent Child

Replication Queuing Engine (RQE) • • There is one RQE per XPE In each RQE there are 11 queues which supports 8 multicast class of service, two type of CPU traffic (High and Low) and SPAN. The RQE scheduling is only one level One RQUE scheduling schedule packet for replication among 11 RQE queues © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

RQE Scheduler • • • RQE scheduling node has 11 children (RQE queues) The order of the RQE queues that are bound to the RQE scheduling node is fixed When the RQE scheduling node is configured to utilize a strict priorityscheduling discipline, the order of the priority will be in descending order from RQE queue 10 to RQE queue 0. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

RQE Scheduling Hierarchy RQE Queues L 0 RQE Scheduler M irror CPU High CPU Low COS 7 COS 1 COS 0 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential SP+ (Rep licat io n -W RR)

RQE Scheduling Discipline • Strict Priority (SP) • Replication – Weighted Round Robin (WRR) • Strict Priority + (Replication – Weighted Round Robin (WRR)) • • • The Replication – WRR scheduler selects packets for replication such that the total number of replications provided to an RQE queue is proportional to its weight. When the RQE scheduler selects one RQE queue for service, it finishes replication of the packet at the head of the queue before it selects the next eligible queue for replication. The weight of the Replication – WRR scheduling discipline is in range [1 -127] © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident ASIC Comparison ASIC T 2+ T 3 -X 7 AKA BCM 56860 BCM 56870 Buffer 12. 2 MB 16 MB 32 MB Cell Count 59076 80596 131072 Cell Size 208 Bytes 256 Bytes Bandwidth 1. 28 Tbps 3. 2 Tbps Max-Port Number 104 x 10 G 128 x 25 G © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident ASIC Unified Forwarding Comparison ASIC T 2+ T 3 -X 7 AKA BCM 56860 BCM 56870 Host 128 K 168 K IPv 4 ALPM 128 K 360 K IPv 6/64 ALPM 80 K 192 K Next Hops 48 K 64 K IPMC Groups 16 K 16 K © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident 3 New Features © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • Transient Capture Buffer • Packet Time Stamping • Packetized MMU Stats • In-band Telemetry

Trident 3 Transient Capture Buffer (TCB) • • • Identify offending flows when queues suddenly fill up – Analyze packet drops Enable centralized controller, applications to be traffic aware when congestion/drops events occur Offers an on-chip “logical Analyzer” for operator insight into network congestion MMU Ingress/Egress Admission No Free Cells Qualified Set Unicast queue/group Ingress/Egress port Ingress/Egress pipe WRED CPU Freeze Pre-Trigger. Capture Events Meta-data 80 B pkt hdr Critical Event Buffer © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Capture Scope Packet Drop Masks 1 K TCB DMA TCB + Event Buffer to CPU Post-Trigger-Bound to offending Queue FIFO buffer (Record drop packets meta data and Drop reason) Circular Buffer, It records packet information and meta data (side band information on queue arrival)

Transient Capture Buffer State Machine • • TCB configuration is done in Idle phase where TCB is disabled Idle Phase Pre-Trigger Phase starts after enabling TCB where circular buffer will capture data from qualified set queues Admin Enable Admin Disable • • When a trigger (drop) event occurs for a queue in the capturing scope, Capture Data Within the TCB process transitions to the post-trigger phase Pre-Trigger Qualified Set In the post-trigger phase, the TCB process will focus on the queue within the capturing scope The TCB process will continue to record until a pre-determined number of transactions have been recorded or when the watchdog timer expires, TCB process then transitions to the frozen phase The CPU can only read the TCB in the frozen state. Once the CPU reads the capture buffer, it transitions back to the pre-trigger phase Trigger Event CPU Handshake Enough Post-Trigger Records Captured Frozen Signal Buffer Ready © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Post-Trigger Selective Capture Data

TCB Configurable attributes • Capture Scope • Monitor Scope Type • Pre/Post Trigger Phase Sample Probability Freeze condition Unicast Queue (UCQ) • Ingress port • Egress port • • Monitor Scope Entity • Threshold Profiles • Drop Event Trigger • Threshold Profile Map • Ingress Admission Drop • Egress Admission Drop • Weighted Random Drop • © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Pre Freeze Capture Number • Pre Freeze Capture Time • Egress Admission Drop • Weighted Random Drop •

Trident 3 Packet Time Stamping • • • Packet time stamping enables precise, scalable network traffic monitoring • Perform latency measurement thru the Device or End-to-End • Re-construct path topology using Origin_id Packet arrival and/or departure times are programmatically tagged to frames along the network path • Post Queue –EP applies Ingress and/or Egress Time Stamps to front port traffic • Only Egress TS supported on HG Port Programmatic control is at the flow or port level • • IFP can override port level controls and deny time stamp action Frames can carry a series of Time Stamps inserted by switches along the path UTC format 1588 Synchronized Timestamp Original Packet FCS 48 -bit UTC time-stamp (16 b sec + 32 b Nanosecond) 8 -bit unused 32 -bit Origin ID rx/tx Rx/Tx Indicates if time stamp is pre -queue or post-queue 32 -bit reserved/FCS © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32 -bit origin IS identifies the source of the time stamp shim

Trident 3 Packetized MMU Stats (PSTATS) • • Real-Time Congestion Queue Monitoring supports packetized MMU Stats SDN Friendly method of tracking buffer occupancy over time • • Send occupancy data to local CPU, CPU then will redirect to remote collectors Reports instantons or high watermark (peak usecount) values per queue • Functions independent of BST • Mutually exclusive operation with OOB Stats Interface • Either OOB or PSTATS can be enabled at a time • By default OOB stats is enabled • To enable the PSTATS configure “[no] pstat enable” -Reports any subset of service pools and unicast queue use-count -Support mutually exclusive instantaneous or high watermark reporting of total UCQ and pool total use-counts -Provides an UTC format, 1588 time-stamp per each round of reporting © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident 3 Packetized MMU Stats (PSTATS) Agent polls stats table repetitively and forms multiple PSTATS frame per report Occupancy 100 90 Pkt. Stats Agent 80 Collector Local CPU 70 Trident III PCIe 60 Packetized Stats Message 50 8 14 B Pkt. Stats Header Src_ID, Time. Stamps, Seq# 40 1 8 ENET+. 1 Q IPv 4/IPv 6 UDP 30 1200 B Stats Payload carries 400 items 20 17 b Stat_Value + 7 b rsvd 10 0 Collector re-arranges payload based on message type and Sequence id 1 2 3 4 5 6 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17 b Stat_Value + 7 b rsvd 8 Packets required to DMA 3 K queues Stats Payload FCS 1 Pkt Order Agnostic Stats Messages

In-band Telemetry • • T 3 Spine INT data-plane probe is new IETF draft protocol for In Band Telemetry which provides visibility into the network using high frequency probes T 3 Spine Per-Flow trace (LAG/ECMP paths) with congestion metrics used for troubleshooting and/or traffic engineering Unique set of parsing, header validation, editing requirement mapped to Trident 3 pipeline T 3 Leaf Host Trident 3’s in-band telemetry is capable of embedding metadata, such as queue-id, ingress and egress timestamps, transmit rate, and drop count in the forwarding plane packets so that a remote analytics application, can build histograms for latency and burst characterization at microsecond granularity © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Analytics Application

In-band Telemetry and Programmability • • Trident 3 programmability enables in-field upgradability of the pipeline to support new packet processing capabilities The ASIC delivers flex-parsing, flex-table lookup, and flex-metadata insertion in the pipeline, all through programmability © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

In-band Telemetry and Programmability Fle. XGS Ingress Pipeline 0 Port Macro Add new rules to parse INT header Parser Lookup Engine(s) Normal forwarding lookups Normal tunnel terminations lookups Lookup Engine(s) Special Functions, ECMP, Hash Identify INT actions -Transit Action: Insert MD -Egress Action: Mirror/Loopback Editor Construct and Insert transport and/or INT MD headers © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Editor Control Generate INT edit instructions Buffer Scheduler Fle. XGS Egress Pipeline 0 Port Macro Memory Management Unit (MMU) Lookup Engine(s) Generate Metadata -Device ID -Ingress/Egress Port ID -Ingress/Egress Time stamp -Congestion Status

Agenda © 2017 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1 ASIC Architecture 2 Forwarding 3 Buffering 4 CT vs SF 5 Platform Overview

Trident 3 Forwarding © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident 3 Longest Prefix Match (LPM) • • • The memories and interfaces are similar in structure to previous Strata. XGS devices. The hardware is designed to allow retaining legacy LPM prefix matching while offering a means to scale the number of prefixes supported. There are two modes supported for prefix matching: Legacy mode • ALPM mode • © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident 3 Legacy LPM Prefix Matching • Backward compatibility, where a larger number of LPM entries are not required • 16 K max entries. • Direct mapping of TCAM match index to associative data in SRAM • Individual VRF for each TCAM entry © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident 3 ALPM Mode • A larger SRAM is used in conjunction with a TCAM to store a list of entries which share a common prefix • The number of entries supported is increased. • VRF is per TCAM entry and shared for all prefixes within the bucket. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Nexus C 3132 C-Z Scalability Feature Host(default) L 3(ALPM) MAC addresses 98304 32768 IPv 4 host routes 212992 16384 IPv 6 host routes 106496 8192 IPv 4 unicast routes (LPM) 28672 131072 IPv 6 unicast routes (<=64 b) (LPM) 12288 16384 IPv 6 unicast routes (>64 b & <128 b) (LPM) 1024 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Agenda © 2017 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1 ASIC Architecture 2 Forwarding 3 Buffering 4 CT vs SF 5 Platform Overview

Trident 3 Buffer Management © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident 3 Buffer Architecture • • • Superior Burst Absorption (Single flow up to full 32 MB) Dynamic thresholding for Fair Access XP E Ingress Egress • 32 MB Smart Buffer fully shared across all ports Egress • Keeps its own Ingress and egress admission controls, service pools and other accounting resources Consists of separate 10 UC and 10 MC queues for each egress port. These are called physical queues © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Egress

Buffer Management on Nexus 3132 C-Z XPE Shared Buffer (32 MB) Control Traffic Default All user traffic (unicast + multicast) and span traffic • Total buffer is 131072 cells = 32 MB (each cell is 256 Bytes) • The buffer is shared for both ingress and egress and by all ports • The buffer is carved into two service pools, control and default pools. • The buffer reserved for control pool is 4804 Cells = 1230 KB • The total shared available buffer is 95092 cells = 24 MB per • Default service pool supports 8 user-defined. Each user-defined class can have one unicast queue and one multicast queue © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Accounting Basics • • • Trident 3 uses shared buffer architecture 32 MB Packet is stored only once in the buffer, but accounted twice (ingress, egress) When a packet enters through a port, ingress accounting information for that ingress port increments © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • • When a packet gets queued on an egress port, egress accounting information for that egress port increments When a packet egresses the chip, both ingress/egress accounting decrements

Trident 3 ASIC Oversubscription • • Trident 3 provides 3200 G of aggregate switch bandwidth using 2 pipeline To allow greater I/O connectivity without increasing frequency and power required by the pipeline, oversubscription mode is supported • • • © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Oversubscribe means that the input demand to a packet processing pipeline exceeds the provided packet processing throughput This results in a non-line rate throughput for certain small packet size All ports has to operate either in oversubscribe or nonoversubscribe mode

Trident 3 ASIC Oversubscription Aggregate MMU Throughput (RFC 2544) Ingress 800 G (all 64 B packets) Arbitration 1600 G (packet size >251 B) 1. 6 Tb/s Input Bandwidth 1. 6 Tb/s Output Bandwidth Ingress Pipeline 0 Egress Pipeline 0 Ingress Packet Processing Ingress Pipeline 1 Ingress Packet Processing 1. 6 Tb/s Input Bandwidth © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential MMU Egress Pipeline 1 1. 6 Tb/s Output Bandwidth

RFC 2544 Throughput vs Packet size 120 100 80 60 % Line Rate 40 20 0 64 101 192 193 251 32 x 100 Gb. E without FEC Aggregate © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 512 1024 1280 1518 2048 4096 8192 9216

Agenda © 2017 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1 ASIC Architecture 2 Forwarding 3 Buffering 4 CT vs SF 5 Platform Overview

Trident 3 Cut-Through Mode © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident 3 Cut-Through Vs Store-and-Forward • • Trident 3 MMU supports both storeand-forward (SF) and cut-through (CT) modes In SF mode, an entire incoming packet is written into the buffer first. The packet is held in the buffer until the scheduler selects that particular egress port's queue © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • • • CT mode is used in latencysensitive applications In CT mode, the packet is scheduled through the cut-through path and dequeued to the EP before it has been completely received from the ingress pipeline. In CT mode, After one or more packet cells are received by the MMU, the packet becomes eligible for dequeening

MMU Cut-Through Block Diagram SF Queues No Ingress Pipe Queue Scheduler Admission Control Cut Through Egress Pipe CT FIFOs Yes © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Trident 3 Cut-Through switching Matrix Min Ingress Speed Max Egress Speed Destination port Speed 10 G ->10 G 50 G 10 G 40 G -> 10 G 25 G 50 G 25 G 40 G 100 G 40 G 50 G 100 G © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25 G -> 10 G 50 G -> 10 G

Agenda © 2017 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1 ASIC Architecture 2 Forwarding 3 Buffering 4 CT vs SF 5 Platform Overview

N 3 K-C 3132 -Z Platform Overview © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

N 3 K-C 3132 C-Z © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • Switch Overview • System Characteristics • Power Characteristics • Switch Architecture • ASIC Port-map • Performance

N 3 K-C 3132 C-Z Lane-selected LED Beacon LED Status LED Environment LED 32 40/100 GE QSFP 28 2 x 10 GE SPF+ Each QSFP 28 port can operate at 10, 25, 40, 50, and 100 G Up to a maximum of 128 x 25 -Gbps ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1 RU

Nexus C 3132 C-Z System Characteristics Internal Code Name Haggan ASIC Trident 3 -X 7 Port Density 32 x 100 G Port speed support 10 / 25 / 40 / 50 / 100 Gbps CPU 4 Core, 1. 8 GHz Intel Xeon System Memory 16 GB Boot-flash memory 128 GB © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Nexus C 3132 C-Z Power Characteristics Number of Power Supply 2 (1+1) Typical Power Usage 312 W Maximum Power Usage 493 W Power-supply types AC (forward and reverse airflow) Input Voltage (AC) 100 to 240 VAC Efficiency (AC) 89 to 91% at 220 V Frequency (AC) 50 to 60 Hz © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Nexus C 3132 C-Z Mechanical Characteristics Fans 4 ( 3 + 1 redundancy ) Air Flow Port Side Intake and Exhaust Physical dimension ( H x W x D) 17. 3 W x 1. 72 H x 18. 5 D” (43. 9 x 4. 36 x 46. 99 cm) Weight (without PS and fan) 13. 8 lb. Weight (with PS and fan) 19. 2 lb. Ro. HS Compliance Yes Temperature: Operating 32 to 104°F (0 to 40°C) Altitude: Operating 0 ft. to 10, 000 ft. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

N 3 K-C 3132 C-Z Switch Architecture Fan 1 Fan 2 Management Console USB DIMM CPU Leffe Board 1. 8 GHz Intel Xeon Fan 3 Fan 4 650 W PSU G 1 x 1 PCIe 650 W PSU Fan Interface Board G 2 x 4 PCIe G 1 x 1 PCIe CTRL/Status Haggan Board x 4 PCIe Trident 3 MC 2 x 10 G SFP © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential FCs 32 x QSPF 28 MI FPGA CPLD

4(3+1) FANs Connected to FAN interface Board N 3 K-C 3132 C-Z Switch Architecture 2(1+1) 560 W PSUs FANs IO card PSU 2(1+1) 560 W PSUs FAN Interface Board 1 x 16 GB DIMM Main System Board Trident 3 -X 7 ASIC © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential D I M M Intel(R) Xeon(R) CPU D-1526 4 C @ 1. 80 GHz C P U T 3 x 7 ASIC

N 3 K-C 3132 C-Z ASIC Port-map Trident 3 Falcon. Cores M C 1 M C 2 FC 1 FC 5 FC 9 FC 13 FC 17 FC 21 FC 25 FC 29 FC 3 FC 7 FC 11 FC 15 FC 19 FC 23 FC 27 FC 31 Port Group 0 FC 6 FC 10 FC 14 FC 18 FC 22 FC 26 FC 30 FC 4 FC 8 FC 12 FC 16 FC 20 FC 24 FC 28 FC 32 FC 2 Pipe 0 Port Group 1 Port Group 2 Pipe 1 Port Group 3 3 3 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 3 4 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 C 3132 C-Z Front Panel Ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

N 3 K-C 3132 C-Z Performance © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • 32 x 100 g L 2 • 32 x 100 g V 4 • 32 x 100 g V 6

N 3 K-C 3132 C-Z 32 x 100 g • N 3132 C-Z is directly connected(32*100 GE) to Spirent traffic generator 100 GE ports • Spirent Test center 4. 78 ver with MX 3 100 ge modules • Switching Mode – Cut Through • FIFO Latency Measurement is used. • Test duration for each frame size per load for each iteration is 120 secs • FEC enabled on all the ports • For unicast tests – control packets such as multicast or any L 3 routing protocols were not configured • Adapter Latency is removed © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

N 3 K-C 3132 C-Z L 2 Performance © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • L 2 Unicast Port-Pair • L 2 Unicast Full-mesh • L 2 Mcast 1 to 1 • L 2 Mcast 1 to N • L 2 Mcast N to N

C 3132 C-Z 32 x 100 G L 2 Unicast Port-Pair Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 64 42. 17355 0. 895 2. 116 16. 878 128 74. 28651 0. 905 1. 147 8. 588 256 99. 995 0. 93 0. 989 1. 01 1024 99. 995 0. 935 0. 995 1. 005 1280 99. 995 0. 935 0. 967 1. 015 1518 99. 995 0. 935 0. 966 0. 98 2034 99. 995 0. 94 0. 966 0. 98 4368 99. 995 0. 94 0. 967 0. 978 9216 99. 995 0. 935 0. 965 0. 978 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G L 2 Unicast Full-mesh Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 64 42. 1678 0. 895 1. 218 9. 258 128 74. 28925 0. 905 1. 168 7. 7 256 99. 995 0. 935 0. 991 1. 02 1024 99. 995 0. 94 0. 996 1. 02 1280 99. 995 0. 935 0. 968 0. 98 1518 99. 995 0. 943 0. 968 0. 985 2034 99. 995 0. 945 0. 968 0. 98 4368 99. 995 0. 945 0. 968 0. 983 9216 99. 995 0. 935 0. 967 0. 98 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G L 2 Mcast 1 to 1 Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 64 63. 5640 0. 91 1. 096 8. 753 128 99. 995 0. 935 0. 955 1. 265 256 99. 995 0. 969 1. 285 1024 99. 995 0. 96 0. 979 1. 293 1280 99. 995 0. 955 0. 977 1. 298 1518 99. 995 0. 955 0. 977 1. 29 2034 99. 995 0. 955 0. 977 1. 29 4368 99. 995 0. 955 0. 977 1. 298 9216 99. 995 0. 955 0. 976 1. 288 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G L 2 Mcast 1 to N Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 64 42. 167788 0. 905 1. 263 11. 35 128 74. 289135 0. 918 1. 212 10. 2 256 99. 995 0. 93 0. 975 1. 293 1024 99. 995 0. 94 0. 983 1. 303 1280 99. 995 0. 935 0. 985 1. 308 1518 99. 995 0. 935 0. 985 1. 303 2034 99. 995 0. 935 0. 985 1. 303 4368 99. 995 0. 935 0. 985 1. 305 9216 99. 995 0. 935 0. 984 1. 303 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G L 2 Mcast N to N Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 64 63. 56 0. 9 1. 135 4. 843 128 99. 995 0. 905 1. 135 1. 318 256 99. 995 0. 915 1. 304 1. 645 1024 99. 995 0. 925 1. 613 2. 265 1280 99. 995 0. 925 2. 244 3. 51 1518 99. 995 0. 925 2. 559 4. 138 2034 99. 995 0. 92 2. 856 4. 715 4368 99. 995 0. 923 3. 494 5. 975 9216 99. 995 0. 93 6. 029 10. 948 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

N 3 K-C 3132 C-Z V 4 Performance © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • V 4 Unicast Port-Pair • V 4 Unicast Full-mesh • V 4 Mcast 1 to 1 • V 4 Mcast 1 to N • V 4 Mcast N to N

C 3132 C-Z 32 x 100 G L 3 V 4 -Unicast Port-Pair Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 64 42. 16779 0. 643 1. 196 14. 97 128 74. 23787 0. 653 0. 994 4. 265 256 99. 995 0. 668 0. 965 1. 235 1024 99. 995 0. 678 0. 974 1. 24 1280 99. 995 0. 678 0. 996 1. 26 1518 99. 995 0. 67 1. 002 1. 26 2034 99. 995 0. 678 1. 018 1. 288 4368 99. 995 0. 678 1. 032 1. 293 9216 99. 995 0. 675 1. 109 1. 378 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G L 3 V 4 -Unicast Full-mesh Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 64 40. 9301 0. 63 0. 907 1. 195 128 73. 97422 0. 645 0. 958 1. 328 256 99. 995 0. 678 1. 026 1. 383 1024 99. 995 0. 683 1. 02 1. 47 1280 99. 995 0. 683 1. 028 1. 475 1518 99. 995 0. 688 1. 006 1. 438 2034 99. 995 0. 693 1. 038 1. 498 4368 99. 995 0. 693 1. 011 1. 42 9216 99. 995 0. 698 1. 008 1. 365 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G V 4 Mcast 1 to 1 Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 64 42. 16779 0. 878 1. 226 7. 61 128 74. 28181 0. 895 1. 132 8. 545 256 99. 995 0. 91 0. 989 1. 015 1024 99. 995 0. 915 1 1. 025 1280 99. 995 0. 915 1. 018 1. 04 1518 99. 995 0. 923 1. 029 1. 055 2034 99. 995 0. 915 1. 035 1. 068 4368 99. 995 0. 915 1. 04 1. 093 9216 99. 995 0. 92 1. 007 1. 168 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G V 4 Mcast 1 to N Frame Size (bytes) 64 Load (%) Min Latency Avg Latency Max Latency 63. 55998 0. 9 1. 027 6. 11 128 99. 995 0. 923 0. 989 1 256 99. 995 0. 935 1. 008 1. 023 1024 99. 995 0. 95 1. 011 1. 025 1280 99. 995 0. 945 1. 02 1. 03 1518 99. 995 0. 945 1. 025 1. 04 2034 99. 995 0. 95 1. 03 1. 04 4368 99. 995 0. 95 1. 04 1. 055 9216 99. 995 0. 945 1. 079 1. 098 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G V 4 Mcast N to N Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 64 63. 56 0. 895 1. 172 5. 685 128 99. 995 0. 905 1. 133 1. 318 256 99. 995 0. 958 1. 303 1. 645 1024 99. 995 0. 923 1. 607 2. 265 1280 99. 995 0. 915 2. 233 3. 513 1518 99. 995 0. 913 2. 543 4. 138 2034 99. 995 0. 915 2. 84 4. 718 4368 99. 995 0. 915 3. 474 5. 983 9216 99. 995 0. 93 6. 028 10. 953 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

N 3 K-C 3132 C-Z V 6 Performance © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential • V 6 Unicast Port-Pair • V 6 Unicast Full-mesh • V 6 Mcast 1 to 1 • V 6 Mcast 1 to N • V 6 Mcast N to N

C 3132 C-Z 32 x 100 G L 3 V 6 -Unicast Port-Pair Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 78 49. 19113 0. 645 1. 133 7. 958 128 74. 26716 0. 658 1. 046 6. 535 256 99. 995 0. 675 0. 965 1. 23 512 99. 995 0. 68 0. 973 1. 235 1024 99. 995 0. 678 0. 997 1. 26 1280 99. 995 0. 675 1. 002 1. 26 1518 99. 995 0. 678 1. 018 1. 283 2034 99. 995 0. 678 1. 033 1. 295 4368 99. 995 0. 675 1. 105 1. 378 9216 99. 995 0. 675 1. 27 1. 593 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G L 3 V 6 -Unicast Full-mesh Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 78 49. 13254 0. 64 0. 949 2. 623 128 74. 28181 0. 65 1. 117 7. 058 256 99. 995 0. 678 1. 028 1. 388 512 99. 995 0. 683 1. 018 1. 465 1024 99. 995 0. 683 1. 03 1. 475 1280 99. 995 0. 69 1. 005 1. 433 1518 99. 995 0. 693 1. 059 1. 498 2034 99. 995 0. 688 1. 016 1. 423 4368 99. 995 0. 698 1. 015 1. 378 9216 99. 995 0. 703 1. 305 1. 608 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G L 3 V 6 -Mcast 1 to 1 Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 86 53. 21179 0. 893 1. 234 10 128 74. 28914 0. 9 1. 177 9. 8 256 99. 995 0. 915 0. 993 1. 015 512 99. 995 0. 92 1. 003 1. 02 1024 99. 995 0. 92 1. 023 1. 04 1280 99. 995 0. 92 1. 031 1. 05 1518 99. 995 0. 92 1. 037 1. 065 2034 99. 995 0. 92 1. 03 1. 088 4368 99. 995 0. 92 1. 016 1. 163 9216 99. 995 0. 923 1. 05 1. 385 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G L 3 V 6 -Mcast 1 to N Frame Size (bytes) Load (%) Min Latency Avg Latency Max Latency 80. 20661 0. 903 1. 031 6. 013 128 99. 995 0. 925 0. 99 1. 005 256 99. 995 0. 938 1. 008 1. 025 512 99. 995 0. 95 1. 013 1. 03 1024 99. 995 0. 945 1. 022 1. 04 1280 99. 995 0. 95 1. 026 1. 045 1518 99. 995 0. 95 1. 03 1. 05 2034 99. 995 0. 95 1. 042 1. 06 4368 99. 995 0. 95 1. 083 1. 103 9216 99. 995 0. 945 1. 224 1. 28 86 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

C 3132 C-Z 32 x 100 G L 3 V 6 -Mcast N to N Frame Size (bytes) 86 Load (%) Min Latency Avg Latency Max Latency 80. 20938 0. 9 1. 114 2. 545 128 99. 995 0. 905 1. 134 1. 318 256 99. 995 0. 92 1. 305 1. 645 512 99. 995 0. 923 1. 609 2. 263 1024 99. 995 0. 915 2. 236 3. 518 1280 99. 995 0. 915 2. 542 4. 133 1518 99. 995 0. 915 2. 838 4. 718 2034 99. 995 0. 915 3. 474 5. 978 4368 99. 995 0. 93 6. 03 10. 953 9216 99. 995 0. 925 12. 42 23. 528 FEC enabled on all the ports © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Agenda © 2017 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1 ASIC Architecture 2 Forwarding 3 Buffering 4 CT vs SF 5 Platform Overview