PMD Auto Load Balancing Nitin Katiyar Pradeep Venkatesan

PMD Auto Load Balancing Nitin Katiyar Pradeep Venkatesan Jan Scheurich 2018 -12 -04 | Page 1 (20)

Agenda › Background › Auto Load Balancing – Proposal › Example › Summary › Questions 2018 -12 -04 | Page 2 (20)

Background › Port Rx queues assignment to PMDs is realized – statically by configuration – dynamically - on demand or when certain events occur › Static port queue assignment: – Simple – Works best if traffic patterns are well understood and unchanging 2018 -12 -04 | Page 3 (20)

Background(contd…) › Dynamic port queue assignment: – triggered by port addition, deletion etc. – triggered by CLI pmd-rxq-rebalance command – configurable assignment algorithm i. e. cycles or roundrobin – can better handle unpredictable traffic patterns but still not adaptive 2018 -12 -04 | Page 4 (20)

Background(contd…) › Round Robin Assignment – Simple algorithm – Load distribution not uniform – Reassignment may or may not result in better load distribution › Cycles based Assignment – Assignment aims to keep load distribution uniform across PMDs – Each PMD keeps track of the processing cycles spent for each Rx Queue – Six snapshots of the queue processing cycles each spanning 10 seconds are maintained – When reassignment requested, the queue statistics are used to get a uniform distribution 2018 -12 -04 | Page 5 (20)

Automatic Load Balancing Proposal › Trigger cycles based assignment automatically (disabled by default) › Maintain additional per-queue statistics to help detect if any PMD is overloaded › Additional configuration parameters to fine tune the behavior › Reassignment will be triggered only if such a reassignment will result in better load distribution 2018 -12 -04 | Page 6 (20)

Automatic Load Balancing Proposal Details › Additional per-queue statistics to be maintained – Number of Rx drops for physical port queues – Number of times vhost ports exceeded the qfill threshold › New configuration parameters › Enable/Disable automatic load balancing › PMD Overload Threshold (utilization %) › Whether Q Drops should factor into PMD Overload determination › Minimum improvement in load distribution expected as a result of reassignment 2018 -12 -04 | Page 7 (20)

Automatic Load Balancing Proposal Details (contd…) › Modifications to Cycles based reassignment function – A new option termed "dry-run" – Dry-run option will return the poll list resulting from running the algorithm without actually performing the reassignment. 2018 -12 -04 | Page 8 (20)

Automatic Load Balancing Details (contd…) › A PMD will consider itself overloaded, if for six consecutive 10 second intervals: – the PMD utilization is more than the configured Overload threshold AND – if Q drops are to be factored, number of Rx queue drops or number of times qfillthreshold exceeded is non-zero 2018 -12 -04 | Page 9 (20)

Automatic Load Balancing Details (contd…) › OVS Main thread changes: – if any PMD considers itself overloaded, perform a dry-run of the reassignment – if dry-run poll list identical to current poll list, nothing more to do – if dry-run poll list different from current poll list, reassignment may offer benefits 2018 -12 -04 | Page 10 (20)

Automatic Load Balancing Details (contd…) › OVS Main thread changes: – Evaluate if the reassignment will meet the configured minimum improvement as follows › Calculate the total processing cycles for each PMD based on current poll list of RX queue › Calculate the average variance in processing cycles for each PMD › Repeat above calculations using dry-run poll list › If dry-run average variance is less, the reassignment will likely result in a more uniform load distribution - Calculate the percentage improvement of the new average variance over the old average variance - Trigger reassignment if the difference is more the configured improvement threshold 2018 -12 -04 | Page 11 (20)

Automatic Load Balancing Details (contd…) Record processing cycles and checks if PMD is overloaded PMD Thread Record processing cycles and checks if PMD is overloaded Monitor PMD load Main Thread Is any of the PMD thread overloaded? Continue monitoring PMD load No Yes Reload PMD with new “RX queue mapping” Does dry run indicate improvement ? Yes 2018 -12 -04 | Page 12 (20) Reload PMD with new “RX queue mapping” Initiate PMD dry run Initiate data path reconfiguration No

Automatic Load Balancing Example PMD Processing Load 100 35 80 60 30 40 25 20 20 10 5 0 PMD 1 PMD 2 Initial Configuration › › Configured pmd-auto-lb-thresh = 80% Processing load of PMD 1 = 90% (Q 0=25%, Q 1=30%, Q 2=35%) Processing load of PMD 2 = 35% (Q 3=10%, Q 4=20%, Q 5=5%) Mean load = 62% and Variance = 756 2018 -12 -04 | Page 13 (20)

Automatic Load Balancing Example(contd…) PMD Processing Load 100 80 60 5 30 40 20 25 20 10 35 0 PMD 1 PMD 2 PMD Dry Run › › Processing load of PMD 1 = 60% (Q 0=25%, Q 1=30%, Q 5=5%) Processing load of PMD 2 = 65% (Q 2=35%, Q 3=10%, Q 4=20%) Mean load = 62% and Variance = 6 Improvement = ((756 – 6) * 100) / 756 = 99% 2018 -12 -04 | Page 14 (20)

OVS CLI changes › We propose to have following knobs for Automatic PMD load balancing. – ovs-vsctl set open_vswitch. other_config: pmd-auto-lb="true" – ovs-vsctl set open_vswitch. other_config: pmd-auto-lb-thresh=“<>" – ovs-vsctl set open_vswitch. other_config: pmd-auto-lb-minimprovement=“<>" – ovs-vsctl set open_vswitch. other_config: pmd-auto-lb-dropcheck=“<>“ 2018 -12 -04 | Page 15 (20)

RFC Patch › RFC Patch v 1 is available in ML: – https: //patchwork. ozlabs. org/patch/982444/ 2018 -12 -04 | Page 16 (20)

Future Enhancements › Frequent change in RX queue to PMD assignment can be avoided by adding a dampening logic in main thread. › Time interval and frequency for PMD auto load balancing can be made configurable via CLI. › The threshold for RX Drops for physical ports and RX qfill for vhost user ports can be made configurable via CLI. 2018 -12 -04 | Page 17 (20)

Summary › PMD auto load balancing feature provides a mechanism to efficiently utilize the PMDs if traffic pattern changes after RX queue assignment to PMDs. › This feature can be enabled/disabled dynamically via OVS CLI and no additional cost in processing when this feature is disabled. › Provides various tunable parameters via OVS CLI to setup automatic load balancing according to the needs. 2018 -12 -04 | Page 18 (20)

Questions 2018 -12 -04 | Page 19 (20)

2018 -12 -04 | Page 20 (20)