lkml.org 
[lkml]   [2024]   [Feb]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH 00/15] Coalesced Interrupt Delivery with posted MSI
Hi Jens,

On Mon, 12 Feb 2024 11:36:42 -0700, Jens Axboe <axboe@kernel.dk> wrote:

> > For the record, here is my set up and performance data for 4 Samsung
> > disks. IOPS increased from 1.6M per disk to 2.1M. One difference I
> > noticed is that IRQ throughput is improved instead of reduction with
> > this patch on my setup. e.g. BEFORE: 185545/sec/vector
> > AFTER: 220128
>
> I'm surprised at the rates being that low, and if so, why the posted MSI
> makes a difference? Usually what I've seen for IRQ being slower than
> poll is if interrupt delivery is unreasonably slow on that architecture
> of machine. But ~200k/sec isn't that high at all.

Even at ~200k/sec, I am seeing around 75% ratio between posted interrupt
notification and MSIs. i.e. for every 4 MSIs, we save one CPU notification.
That might be where the savings come from.

I was expecting an even or reduction in CPU notifications but more MSI
throughput. Instead, Optane gets less MSIs/sec as your data shows.

Is it possible to get the interrupt coalescing ratio on your set up? ie.
PMN count in cat /proc/interrupts divided by total NVME MSIs.

Here is a summary of my testing on 4 Samsung Gen 5 drives:
test cases IOPS*1000 ints/sec(MSI)*
=================================================
aio 6348 182218
io_uring 6895 207932
aio w/ posted MSI 8295 185545
io_uring w/ post MSI 8811 220128
io_uring poll_queue 13000 0
================================================


Thanks,

Jacob

\
 
 \ /
  Last update: 2024-05-27 15:00    [W:0.770 / U:0.008 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site