Messages in this thread | | | Subject | Re: [PATCH v3] lib: optimize cpumask_local_spread() | From | Shaokun Zhang <> | Date | Fri, 8 Nov 2019 13:50:05 +0800 |
| |
Hi Andrew,
On 2019/11/8 11:49, Andrew Morton wrote: > On Thu, 7 Nov 2019 09:44:08 +0800 Shaokun Zhang <zhangshaokun@hisilicon.com> wrote: > >> In the multi-processors and NUMA system, I/O driver will find cpu cores >> that which shall be bound IRQ. When cpu cores in the local numa have >> been used, it is better to find the node closest to the local numa node, >> instead of choosing any online cpu immediately. >> >> On Huawei Kunpeng 920 server, there are 4 NUMA node(0 -3) in the 2-cpu >> system(0 - 1). We perform PS (parameter server) business test, the >> behavior of the service is that the client initiates a request through >> the network card, the server responds to the request after calculation. >> When two PS processes run on node2 and node3 separately and the >> network card is located on 'node2' which is in cpu1, the performance >> of node2 (26W QPS) and node3 (22W QPS) was different. >> It is better that the NIC queues are bound to the cpu1 cores in turn, >> then XPS will also be properly initialized, while cpumask_local_spread >> only considers the local node. When the number of NIC queues exceeds >> the number of cores in the local node, it returns to the online core >> directly. So when PS runs on node3 sending a calculated request, >> the performance is not as good as the node2. It is considered that >> the NIC and other I/O devices shall initialize the interrupt binding, >> if the cores of the local node are used up, it is reasonable to return >> the node closest to it. >> >> Let's optimize it and find the nearest node through NUMA distance for the >> non-local NUMA nodes. The performance will be better if it return the >> nearest node than the random node. >> >> After this patch, the performance of the node3 is the same as node2 >> that is 26W QPS when the network card is still in 'node2'. Since it will >> return the closest non-local NUMA code rather than random node, it is no >> harm to others at least. > > This is a little nicer: > > --- a/lib/cpumask.c~lib-optimize-cpumask_local_spread-v3-fix > +++ a/lib/cpumask.c > @@ -254,7 +254,6 @@ static unsigned int __cpumask_local_spre > BUG(); > } > > -static DEFINE_SPINLOCK(spread_lock); > /** > * cpumask_local_spread - select the i'th cpu with local numa cpu's first > * @i: index number > @@ -270,6 +269,7 @@ unsigned int cpumask_local_spread(unsign > { > static int node_dist[MAX_NUMNODES]; > static bool used[MAX_NUMNODES]; > + static DEFINE_SPINLOCK(spread_lock);
Good catch, thanks for fixing it.
Shaokun.
> unsigned long flags; > int cpu, j, id; > > _ > > > . >
| |