Messages in this thread | | | From | Angus Chen <> | Subject | RE: IRQ affinity problem from virtio_blk | Date | Wed, 16 Nov 2022 01:02:37 +0000 |
| |
> -----Original Message----- > From: Michael S. Tsirkin <mst@redhat.com> > Sent: Wednesday, November 16, 2022 7:37 AM > To: Thomas Gleixner <tglx@linutronix.de> > Cc: Angus Chen <angus.chen@jaguarmicro.com>; linux-kernel@vger.kernel.org; > Ming Lei <ming.lei@redhat.com>; Jason Wang <jasowang@redhat.com> > Subject: Re: IRQ affinity problem from virtio_blk > > On Wed, Nov 16, 2022 at 12:24:24AM +0100, Thomas Gleixner wrote: > > On Wed, Nov 16 2022 at 00:04, Thomas Gleixner wrote: > > > > > On Tue, Nov 15 2022 at 17:44, Michael S. Tsirkin wrote: > > >> On Tue, Nov 15, 2022 at 11:19:47PM +0100, Thomas Gleixner wrote: > > >>> > We can see global_available drop from 15354 to 15273, is 81. > > >>> > And the total_allocated increase from 411 to 413. One config irq,and > > >>> > one vq irq. > > >>> > > >>> Right. That's perfectly fine. At the point where you looking at it, the > > >>> matrix allocator has given out 2 vectors as can be seen via > > >>> total_allocated. > > >>> > > >>> But then it also has another 79 vectors put aside for the other queues, > > >> > > >> What makes it put these vectors aside? pci_alloc_irq_vectors_affinity ? > > > > > > init_vq() -> virtio_find_vqs() -> vp_find_vqs() -> > > > vp_request_msix_vectors() -> pci_alloc_irq_vectors_affinity() > > > > > > init_vq() hands in a struct irq_affinity which means that > > > pci_alloc_irq_vectors_affinity() will spread out interrupts and have one > > > for config and one per queue if vp_request_msix_vectors() is invoked > > > with per_vq_vectors == true, which is what the first invocation in > > > vp_find_vqs() does. > > > > I just checked on a random VM. The PCI device as advertised to the guest > > does not expose that many vectors. One has 2 and the other 4. > > > > But as the interrupts are requested 'managed' the core ends up setting > > the vectors aside. That's a fundamental property of managed interrupts. > > > > Assume you have less queues than CPUs, which is the case with 2 vectors > > and tons of CPUs, i.e. one ends up for config and the other for the > > actual queue. So the affinity spreading code will end up having the full > > cpumask for the queue vector, which is marked managed. And managed > means > > that it's guaranteed e.g. in the CPU hotplug case that the interrupt can > > be migrated to a still online CPU. > > > > So we end up setting 79 vectors aside (one per CPU) in the case that the > > virtio device only provides two vectors. > > > > But that's not the end of the world as you really would need ~200 such > > devices to exhaust the vector space...
Provide the VECTOR Information: [root@localhost domains]# cat VECTOR name: VECTOR size: 0 mapped: 2015 flags: 0x00000003 Online bitmaps: 80 Global available: 0 Global reserved: 154 Total allocated: 1861 System: 39: 0-19,29,32,50,128,236,240-242,244,246-255 | CPU | avl | man | mac | act | vectors 0 0 180 2 23 33-46,48,110,132,162,185,206-207,228-229 1 0 180 2 23 33-37,41,44,124,134,156-157,167,180,186-187,198,225,228-233 2 0 180 2 23 33-40,123,154-155,164,177,186,202,221-224,227,232-233,235 3 0 180 2 23 33-36,70,123-124,140,156,168,174,197,199,201,207,225-228,232-235 4 0 180 2 23 33-39,101,122,133,147,207,217-221,227-228,231,233-235 5 0 180 2 23 33-38,83,115,156,165-166,177,207-209,220-222,228,231-234 6 0 180 2 23 33-38,55,91,146,154,160,164,187-188,209,217-218,221-222,232-235 7 0 180 2 23 33-37,81-82,113,145,154,186-188,207,221-224,226,229,232,234-235 8 0 180 2 23 33-37,81,91,148-149,189,198-199,201,210,217-218,222,229-232,234-235 9 0 180 2 23 33-38,59,133,146,157,165,174,196,205,207,220-221,225-226,232-235 10 0 180 2 23 33-36,87,133-134,142,174,188,198-199,206,214,217-220,228-230,234-235 11 0 180 2 23 33-35,83,94,113,127,129,157,187-188,209,219-224,229-230,233-235 12 0 180 2 23 33-34,36,55,113-114,129,158-159,168,175,189-190,197,208-209,219-220,227,232-235 13 0 180 2 23 33-34,37-38,83,94,156-158,186-187,207,221-222,225-227,230-235 14 0 180 2 23 33-35,43,70,101-102,170,175-177,215,217-218,220,226-230,232-233,235 15 0 180 2 23 33-35,104,112,134,144,158,167-168,170,175-176,187,198,208,221-222,228-229,233-235 16 0 180 2 23 34-36,71,91,146,155-156,189-190,217-219,223,225-228,231-235 17 0 180 2 23 33-34,49,92,101,134,144,187,195-197,207-209,216-217,221,230-235 18 0 180 2 23 33-34,135-136,146,174,198,206-209,217,224-231,233-235 19 0 180 2 23 33-34,58,91,101,113,122,135,165,197-199,206,221-223,228-229,231-235 20 0 180 2 23 33-34,215-235 21 0 180 2 23 33-34,214,216-235 22 0 180 2 23 33-34,215-235 23 0 180 2 23 33-34,215-235 24 0 180 2 23 33-35,216-235 25 0 180 2 23 33-35,216-235 26 0 180 2 23 33-35,216-235 27 0 180 2 23 33-35,216-235 28 0 180 2 23 33-35,216-235 29 0 180 2 23 33-35,216-235 30 0 180 2 23 33-35,216-235 31 0 180 2 23 33-35,216-235 32 0 180 2 23 33-34,215-235 33 0 180 2 23 33-34,215-235 34 0 180 2 23 33-34,215-235 35 0 180 2 23 33-34,215-235 36 0 180 2 23 33-34,211,216-235 37 0 180 2 23 33-34,215-235 38 0 180 2 23 33-34,215-235 39 0 180 2 23 33-34,215-235 40 0 180 2 23 33-34,56,65,134,170,176-178,207-210,225-229,231-235 41 0 180 2 23 33-34,54,113,135-137,143,169,195-198,216-217,224,228-230,232-235 42 0 180 2 23 33,36,57,111-112,126,164,175-176,199-200,207-210,225-226,230-235 43 0 180 2 23 33-34,70,82,133-135,145,155,166,174,188-189,207,209,218,226-229,233-235 44 0 180 2 23 33-34,59,103,111,126,166-167,185-186,207-208,217-218,226-232,234-235 45 0 180 2 23 33,35-36,81,106,145-146,165,176,187,195,220-221,226-235 46 0 180 2 23 33-34,69,137,143,155,176,180,185-187,197,206-207,212-213,225-228,230,234-235 47 0 180 2 23 34,36,71,91-92,103-104,143,165,179,185-186,195,208-209,220-221,230-235 48 0 180 2 23 33-34,36,93,122,157,174,186-188,198,208-209,225,227-235 49 0 180 2 23 34-35,132-133,147-148,156,176-177,194-197,212,226-228,230-235 50 0 180 2 23 33-34,45,123,138,162,164-166,195-196,208-209,219,224-226,228,230-231,233-235 51 0 180 2 23 33-34,55,69-70,110,167,179-181,197-198,217-220,228-230,232-235 52 0 180 2 23 33-34,70,132,145,156,178,186-188,190,210-212,218-219,228-230,232-235 53 0 180 2 23 33,35,70,111,144,194-195,197,209,216-219,224,226-231,233-235 54 0 180 2 23 33-34,102,115,147,154,164-166,181,188,200,210-211,219-220,228-229,231-235 55 0 180 2 23 33-36,55,114,154-156,174,187,198,207-209,224-225,227-229,233-235 56 0 180 2 23 33-34,54,104,113,132,154,175,188,209,216-221,226-227,230-233,235 57 0 180 2 23 34-35,47,100,127,132-133,176-178,196-197,208,220,224-226,230-235 58 0 180 2 23 34,37,42,100,110-111,143,164-165,185,198,206-208,216-218,228-229,231,233-235 59 0 180 3 24 33-35,39,43,81-82,111,126,164-165,184,186,211-212,219-221,223,231-235 60 0 180 3 24 33-35,215-235 61 0 180 3 24 33-35,215-235 62 0 180 3 24 33-35,215-235 63 0 180 3 24 33-35,215-235 64 0 180 3 24 33-35,215-235 65 0 180 3 24 33-35,215-235 66 0 180 3 24 33-35,215-235 67 0 180 3 24 33-35,215-235 68 0 180 3 24 33-35,211,216-235 69 0 180 3 24 33-35,215-235 70 0 180 3 24 33-35,215-235 71 0 180 3 24 33-35,215-235 72 0 180 3 24 33-35,215-235 73 0 180 3 24 33-35,215-235 74 0 180 3 24 33-35,215-235 75 0 180 3 24 33-35,215-235 76 0 180 3 24 33-35,215-235 77 0 180 3 24 33-35,215-235 78 0 180 3 24 33-35,214,216-235 79 0 180 3 24 33-35,215-235
crash_cts> p *vector_matrix $98 = { matrix_bits = 256, alloc_start = 32, alloc_end = 236, alloc_size = 204, global_available = 0, global_reserved = 154, systembits_inalloc = 3, total_allocated = 1861, online_maps = 80, maps = 0x46100, scratch_map = {18446744069952503807, 18446744073709551615, 18446744073709551615, 18435229987943481343}, system_map = {1125904739729407, 0, 1, 18435221191850459136} } Any other information I need to provide,pls tell me. Thanks. > > > > Thanks, > > > > tglx > > Let's say we have 20 queues - then just 10 devices will exhaust the > vector space right? > > -- > MST
| |