Messages in this thread | | | From | Eric Dumazet <> | Date | Wed, 19 Sep 2018 07:55:11 -0700 | Subject | Re: [RFC] net;sched: Try to find idle cpu for RPS to handle packets |
| |
On Wed, Sep 19, 2018 at 5:29 AM Kirill Tkhai <ktkhai@virtuozzo.com> wrote: > > Many workloads have polling mode of work. The application > checks for incomming packets from time to time, but it also > has a work to do, when there is no packets. This RFC > tries to develop an idea to queue RPS packets on idle > CPU in the the L3 domain of the consumer, so backlog > processing of the packets and the application can execute > in parallel. > > We require this in case of network cards does not > have enough RX queues to cover all online CPUs (this seems > to be the most cards), and get_rps_cpu() actually chooses > remote cpu, and SMP interrupt is sent. Here we may try > our best, and to find idle CPU nearly the consumer's CPU. > Note, that in case of consumer works in poll mode and it > does not waits for incomming packets, its CPU will be not > idle, while CPU of a sleeping consumer may be idle. So, > not polling consumers will still be able to have skb > handled on its CPU. > > In case of network card has many queues, the device > interrupts will come on consumer's CPU, and this patch > won't try to find idle cpu for them. > > I've tried simple netperf test for this: > netserver -p 1234 > netperf -L 127.0.0.1 -p 1234 -l 100 > > Before: > 87380 16384 16384 100.00 60323.56 > 87380 16384 16384 100.00 60388.46 > 87380 16384 16384 100.00 60217.68 > 87380 16384 16384 100.00 57995.41 > 87380 16384 16384 100.00 60659.00 > > After: > 87380 16384 16384 100.00 64569.09 > 87380 16384 16384 100.00 64569.25 > 87380 16384 16384 100.00 64691.63 > 87380 16384 16384 100.00 64930.14 > 87380 16384 16384 100.00 62670.15 > > The difference between best runs is +7%, > the worst runs differ +8%. > > What do you think about following somehow in this way?
Hi Kirill
In my experience, scheduler has a poor view of softirq processing happening on various cpus. A cpu spending 90% of its cycles processing IRQ might be considered 'idle'
So please run a real workload (it is _very_ uncommon anyone set up RPS on lo interface !)
Like 400 or more concurrent netperf -t TCP_RR on a 10Gbit NIC.
Thanks.
PS: Idea of playing with L3 domains is interesting, I have personally tried various strategies in the past but none of them demonstrated a clear win.
| |