Messages in this thread | | | Date | Fri, 17 Feb 2023 14:40:38 +0100 | From | Felix Fietkau <> | Subject | Re: [RFC v2] net/core: add optional threading for rps backlog processing |
| |
On 17.02.23 13:57, Eric Dumazet wrote: > On Fri, Feb 17, 2023 at 1:35 PM Felix Fietkau <nbd@nbd.name> wrote: >> >> On 17.02.23 13:23, Eric Dumazet wrote: >> > On Fri, Feb 17, 2023 at 11:06 AM Felix Fietkau <nbd@nbd.name> wrote: >> >> >> >> When dealing with few flows or an imbalance on CPU utilization, static RPS >> >> CPU assignment can be too inflexible. Add support for enabling threaded NAPI >> >> for RPS backlog processing in order to allow the scheduler to better balance >> >> processing. This helps better spread the load across idle CPUs. >> >> >> >> Signed-off-by: Felix Fietkau <nbd@nbd.name> >> >> --- >> >> >> >> RFC v2: >> >> - fix rebase error in rps locking >> > >> > Why only deal with RPS ? >> > >> > It seems you propose the sofnet_data backlog be processed by a thread, >> > instead than from softirq ? >> Right. I originally wanted to mainly improve RPS, but my patch does >> cover backlog in general. I will update the description in the next >> version. Does the approach in general make sense to you? >> > > I do not know, this seems to lack some (perf) numbers, and > descriptions of added max latencies and stuff like that :) I just ran some test where I used a MT7621 device (dual-core 800 MHz MIPS, 4 threads) as a router doing NAT without flow offloading.
Using the flent RRUL test between 2 PCs connected through the router, I get these results:
rps_threaded=0: (combined CPU idle time around 27%) avg median 99th % # data pts Ping (ms) ICMP : 26.08 28.70 54.74 ms 199 Ping (ms) UDP BE : 1.96 24.12 37.28 ms 200 Ping (ms) UDP BK : 1.88 15.86 27.30 ms 200 Ping (ms) UDP EF : 1.98 31.77 54.10 ms 200 Ping (ms) avg : 1.94 N/A N/A ms 200 TCP download BE : 69.25 70.20 139.55 Mbits/s 200 TCP download BK : 95.15 92.51 163.93 Mbits/s 200 TCP download CS5 : 133.64 129.10 292.46 Mbits/s 200 TCP download EF : 129.86 127.70 254.47 Mbits/s 200 TCP download avg : 106.97 N/A N/A Mbits/s 200 TCP download sum : 427.90 N/A N/A Mbits/s 200 TCP totals : 864.43 N/A N/A Mbits/s 200 TCP upload BE : 97.54 96.67 163.99 Mbits/s 200 TCP upload BK : 139.76 143.88 190.37 Mbits/s 200 TCP upload CS5 : 97.52 94.70 206.60 Mbits/s 200 TCP upload EF : 101.71 106.72 147.88 Mbits/s 200 TCP upload avg : 109.13 N/A N/A Mbits/s 200 TCP upload sum : 436.53 N/A N/A Mbits/s 200
rps_threaded=1: (combined CPU idle time around 16%) avg median 99th % # data pts Ping (ms) ICMP : 13.70 16.10 27.60 ms 199 Ping (ms) UDP BE : 2.03 18.35 24.16 ms 200 Ping (ms) UDP BK : 2.03 18.36 29.13 ms 200 Ping (ms) UDP EF : 2.36 25.20 41.50 ms 200 Ping (ms) avg : 2.14 N/A N/A ms 200 TCP download BE : 118.69 120.94 160.12 Mbits/s 200 TCP download BK : 134.67 137.81 177.14 Mbits/s 200 TCP download CS5 : 126.15 127.81 174.84 Mbits/s 200 TCP download EF : 78.36 79.41 143.31 Mbits/s 200 TCP download avg : 114.47 N/A N/A Mbits/s 200 TCP download sum : 457.87 N/A N/A Mbits/s 200 TCP totals : 918.19 N/A N/A Mbits/s 200 TCP upload BE : 112.20 111.55 164.38 Mbits/s 200 TCP upload BK : 144.99 139.24 205.12 Mbits/s 200 TCP upload CS5 : 93.09 95.50 132.39 Mbits/s 200 TCP upload EF : 110.04 108.21 207.00 Mbits/s 200 TCP upload avg : 115.08 N/A N/A Mbits/s 200 TCP upload sum : 460.32 N/A N/A Mbits/s 200
As you can see, both throughput and latency improve because load can be better distributed across CPU cores.
- Felix
| |