Messages in this thread | | | From | Pratyush Yadav <> | Subject | Re: [PATCH] nvme-pci: do not set the NUMA node of device if it has none | Date | Wed, 26 Jul 2023 21:32:33 +0200 |
| |
On Wed, Jul 26 2023, Keith Busch wrote:
> On Wed, Jul 26, 2023 at 05:30:33PM +0200, Pratyush Yadav wrote: >> On Wed, Jul 26 2023, Christoph Hellwig wrote: >> > On Wed, Jul 26, 2023 at 10:58:36AM +0300, Sagi Grimberg wrote: >> >>>> For example, AWS EC2's i3.16xlarge instance does not expose NUMA >> >>>> information for the NVMe devices. This means all NVMe devices have >> >>>> NUMA_NO_NODE by default. Without this patch, random 4k read performance >> >>>> measured via fio on CPUs from node 1 (around 165k IOPS) is almost 50% >> >>>> less than CPUs from node 0 (around 315k IOPS). With this patch, CPUs on >> >>>> both nodes get similar performance (around 315k IOPS). >> >>> >> >>> irqbalance doesn't work with this driver though: the interrupts are >> >>> managed by the kernel. Is there some other reason to explain the perf >> >>> difference? >> >> Hmm, I did not know that. I have not gone and looked at the code but I >> think the same reasoning should hold, just with s/irqbalance/kernel. If >> the kernel IRQ balancer sees the device is on node 0, it would deliver >> its interrupts to CPUs on node 0. >> >> In my tests I can see that the interrupts for NVME queues are sent only >> to CPUs from node 0 without this patch. With this patch CPUs from both >> nodes get the interrupts. > > Could you send the output of: > > numactl --hardware
$ numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 node 0 size: 245847 MB node 0 free: 245211 MB node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 node 1 size: 245932 MB node 1 free: 245328 MB node distances: node 0 1 0: 10 21 1: 21 10
> > and then with and without your patch: > > for i in $(cat /proc/interrupts | grep nvme0 | sed "s/^ *//g" | cut -d":" -f 1); do \ > cat /proc/irq/$i/{smp,effective}_affinity_list; \ > done
Without my patch:
$ for i in $(cat /proc/interrupts | grep nvme0 | sed "s/^ *//g" | cut -d":" -f 1); do \ > cat /proc/irq/$i/{smp,effective}_affinity_list; \ > done 40 40 33 33 44 44 9 9 32 32 2 2 6 6 11 11 1 1 35 35 39 39 13 13 42 42 46 46 41 41 46 46 15 15 5 5 43 43 0 0 14 14 8 8 12 12 7 7 10 10 47 47 38 38 36 36 3 3 34 34 45 45 5 5
With my patch:
$ for i in $(cat /proc/interrupts | grep nvme0 | sed "s/^ *//g" | cut -d":" -f 1); do \ > cat /proc/irq/$i/{smp,effective}_affinity_list; \ > done 9 9 15 15 5 5 23 23 38 38 52 52 21 21 36 36 13 13 56 56 44 44 42 42 31 31 48 48 5 5 3 3 1 1 11 11 28 28 18 18 34 34 29 29 58 58 46 46 54 54 59 59 32 32 7 7 56 56 62 62 49 49 57 57
-- Regards, Pratyush Yadav
Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879
| |