Messages in this thread | | | Date | Mon, 16 Oct 2023 15:54:19 -0500 | From | Michael Roth <> | Subject | Re: [PATCHv2] efi/unaccepted: Fix soft lockups caused by parallel memory acceptance |
| |
On Mon, Oct 16, 2023 at 07:31:22PM +0300, Kirill A. Shutemov wrote: > Michael reported soft lockups on a system that has unaccepted memory. > This occurs when a user attempts to allocate and accept memory on > multiple CPUs simultaneously. > > The root cause of the issue is that memory acceptance is serialized with > a spinlock, allowing only one CPU to accept memory at a time. The other > CPUs spin and wait for their turn, leading to starvation and soft lockup > reports. > > To address this, the code has been modified to release the spinlock > while accepting memory. This allows for parallel memory acceptance on > multiple CPUs. > > A newly introduced "accepting_list" keeps track of which memory is > currently being accepted. This is necessary to prevent parallel > acceptance of the same memory block. If a collision occurs, the lock is > released and the process is retried. > > Such collisions should rarely occur. The main path for memory acceptance > is the page allocator, which accepts memory in MAX_ORDER chunks. As long > as MAX_ORDER is equal to or larger than the unit_size, collisions will > never occur because the caller fully owns the memory block being > accepted. > > Aside from the page allocator, only memblock and deferered_free_range() > accept memory, but this only happens during boot. > > The code has been tested with unit_size == 128MiB to trigger collisions > and validate the retry codepath. > > Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> > Reported-by: Michael Roth <michael.roth@amd.com
Tested-by: Michael Roth <michael.roth@amd.com>
This seems to improve things pretty dramatically for me. Previously I saw soft-lockups with 16 vCPUs and 16 processes faulting into memory, and now I can do 128+ vCPUs/processes.
I can still trigger soft lock-ups on occassion if the number of processes faulting in memory exceeds the number of vCPUs available to the guest, but with a 32 vCPU guest even something like this:
stress --vm 128 --vm-bytes 2G --vm-keep --cpu 255
still seems to avoid the soft lock-up messages. So that's probably well into "potential future optimization" territory and this patch fixes the more immediate issues.
Thanks!
-Mike
> Fixes: 2053bc57f367 ("efi: Add unaccepted memory support") > Cc: <stable@kernel.org> > Reviewed-by: Nikolay Borisov <nik.borisov@suse.com> > --- > > v2: > - Fix deadlock (Vlastimil); > - Fix comments (Vlastimil); > - s/cond_resched()/cpu_relax()/ -- cond_resched() cannot be called > from atomic context; >
| |