lkml.org 
[lkml]   [2020]   [Oct]   [29]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: arm64: dropping prevent_bootmem_remove_notifier


Hi Anshuman, David,

Thanks for all the detailed explanations for the reasoning to have
bootmem protected from being removed. Also, I do agree drivers being
able to mark memory sections isn't the right thing to do.

We went ahead with the approach of using "mem=" as you suggested to
limit the bootmem and add remaining blocks using
add_memory_driver_managed() so that driver has ownership of these
blocks.

We do have some follow-up questions regarding this - will initiate a
discussion soon.


On 2020-10-18 22:37, Anshuman Khandual wrote:
> Hello Sudarshan,
>
> On 10/17/2020 04:41 AM, Sudarshan Rajagopalan wrote:
>>
>> Hello Anshuman,
>>
>> In the patch that enables memory hot-remove (commit bbd6ec605c0f
>> ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier
>> put in place that prevents boot memory from being offlined and
>> removed. Also commit text mentions that boot memory on arm64 cannot be
>> removed. We wanted to understand more about the reasoning for this.
>> X86 and other archs doesn’t seem to do this prevention. There’s also
>> comment in the code that this notifier could be dropped in future if
>> and when boot memory can be removed.
>
> Right and till then the notifier cannot be dropped. There was a lot of
> discussions
> around this topic during multiple iterations of memory hot remove
> series. Hence, I
> would just request you to please go through them first. This list here
> is from one
> such series (https://lwn.net/Articles/809179/) but might not be
> exhaustive.
>
> -----------------
> On arm64 platform, it is essential to ensure that the boot time
> discovered
> memory couldn't be hot-removed so that,
>
> 1. FW data structures used across kexec are idempotent
> e.g. the EFI memory map.
>
> 2. linear map or vmemmap would not have to be dynamically split, and
> can
> map boot memory at a large granularity
>
> 3. Avoid penalizing paths that have to walk page tables, where we can
> be
> certain that the memory is not hot-removable
> -----------------
>
> The primary reason being kexec which would need substantial rework
> otherwise.
>
>>
>> The current logic is that only “new” memory blocks which are hot-added
>> can later be offlined and removed. The memory that system booted up
>> with cannot be offlined and removed. But there could be many usercases
>> such as inter-VM memory sharing where a primary VM could offline and
>> hot-remove a block/section of memory and lend it to secondary VM where
>> it could hot-add it. And after usecase is done, the reverse happens
>> where secondary VM hot-removes and gives it back to primary which can
>> hot-add it back. In such cases, the present logic for arm64 doesn’t
>> allow this hot-remove in primary to happen.
>
> That is not true. Each VM could just boot with a minimum boot memory
> which can
> not be offlined or removed but then a possible larger portion of memory
> can be
> hot added during the boot process itself, making them available for any
> future
> inter VM sharing purpose. Hence this problem could easily be solved in
> the user
> space itself.
>
>>
>> Also, on systems with movable zone that sort of guarantees pages to be
>> migrated and isolated so that blocks can be offlined, this logic also
>> defeats the purpose of having a movable zone which system can rely on
>> memory hot-plugging, which say virt-io mem also relies on for fully
>> plugged memory blocks.
> ZONE_MOVABLE does not really guarantee migration, isolation and
> removal. There
> are reasons an offline request might just fail. I agree that those
> reasons are
> normally not platform related but core memory gives platform an
> opportunity to
> decline an offlining request via a notifier. Hence ZONE_MOVABLE offline
> can be
> denied. Semantics wise we are still okay.
>
> This might look bit inconsistent that
> movablecore/kernelcore/movable_node with
> firmware sending in 'hot pluggable' memory (IIRC arm64 does not really
> support
> this yet), the system might end up with ZONE_MOVABLE marked boot memory
> which
> cannot be offlined or removed. But an offline notifier action is
> orthogonal.
> Hence did not block those kernel command line paths that creates
> ZONE_MOVABLE
> during boot to preserve existing behavior.
>
>>
>> I understand that some region of boot RAM shouldn’t be allowed to be
>> removed, but such regions won’t be allowed to be offlined in first
>> place since pages cannot be migrated and isolated, example reserved
>> pages.
>>
>> So we’re trying to understand the reasoning for such a prevention put
>> in place for arm64 arch alone.
>
> Primary reason being kexec. During kexec on arm64, next kernel's memory
> map is
> derived from firmware and not from current running kernel. So the next
> kernel
> will crash if it would access memory that might have been removed in
> running
> kernel. Until kexec on arm64 changes substantially and takes into
> account the
> real available memory on the current kernel, boot memory cannot be
> removed.
>
>>
>> One possible way to solve this is by marking the required sections as
>> “non-early” by removing the SECTION_IS_EARLY bit in its
>> section_mem_map.
>
> That is too intrusive from core memory perspective.
>
> This puts these sections in the context of “memory hotpluggable”
> which can be offlined-removed and added-onlined which are part of boot
> RAM itself and doesn’t need any extra blocks to be hot added. This way
> of marking certain sections as “non-early” could be exported so that
> module drivers can set the required number of sections as “memory
> hotpluggable”. This could have certain checks put in place to see
> which sections are allowed, example only movable zone sections can be
> marked as “non-early”.
>
> Giving modules the right to mark memory hotpluggable ? That is too
> intrusive
> and would still not solve the problem with kexec.
>
>>
>> Your thoughts on this? We are also looking for different ways to solve
>> the problem without having to completely dropping this notifier, but
>> just putting out the concern here about the notifier logic that is
>> breaking our usecase which is a generic memory sharing usecase using
>> memory hotplug feature.
>
> Completely preventing boot memory offline and removal is essential for
> kexec
> to work as expected afterwards. As suggested previously, splitting the
> VM
> memory into boot and non boot chunks during init can help work around
> this
> restriction effectively in userspace itself and would not require any
> kernel
> changes.
>
> - Anshuman


Sudarshan

--
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a
Linux Foundation Collaborative Project

\
 
 \ /
  Last update: 2020-10-29 22:04    [W:0.088 / U:0.076 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site