lkml.org 
[lkml]   [2023]   [Jul]   [26]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
SubjectRe: collision between ZONE_MOVABLE and memblock allocations
From
On 26.07.23 10:44, David Hildenbrand wrote:
> On 20.07.23 00:48, Ross Zwisler wrote:
>> On Wed, Jul 19, 2023 at 08:14:48AM +0200, Michal Hocko wrote:
>>> On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
>>> [...]
>>>> I do think that we need to fix this collision between ZONE_MOVABLE and memmap
>>>> allocations, because this issue essentially makes the movablecore= kernel
>>>> command line parameter useless in many cases, as the ZONE_MOVABLE region it
>>>> creates will often actually be unmovable.
>>>
>>> movablecore is kinda hack and I would be more inclined to get rid of it
>>> rather than build more into it. Could you be more specific about your
>>> use case?
>>
>> The problem that I'm trying to solve is that I'd like to be able to get kernel
>> core dumps off machines (chromebooks) so that we can debug crashes. Because
>> the memory used by the crash kernel ("crashkernel=" kernel command line
>> option) is consumed the entire time the machine is booted, there is a strong
>> motivation to keep the crash kernel as small and as simple as possible. To
>> this end I'm trying to get away without SSD drivers, not having to worry about
>> encryption on the SSDs, etc.
>
> Okay, so you intend to keep the crashkernel area as small as possible.
>
>>
>> So, the rough plan right now is:
>> > 1) During boot set aside some memory that won't contain kernel
> allocations.
>> I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.
>>
>> We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE
>> region (or whatever non-kernel region) will be set aside as PMEM in the crash
>> kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line
>> parameter passed to the crash kernel.
>>
>> So, in my sample 4G VM system, I see:
>>
>> # lsmem --split ZONES --output-all
>> RANGE SIZE STATE REMOVABLE BLOCK NODE ZONES
>> 0x0000000000000000-0x0000000007ffffff 128M online yes 0 0 None
>> 0x0000000008000000-0x00000000bfffffff 2.9G online yes 1-23 0 DMA32
>> 0x0000000100000000-0x000000012fffffff 768M online yes 32-37 0 Normal
>> 0x0000000130000000-0x000000013fffffff 256M online yes 38-39 0 Movable
>>
>> Memory block size: 128M
>> Total online memory: 4G
>> Total offline memory: 0B
>>
>> so I'll pass "memmap=256M!0x130000000" to the crash kernel.
>>
>> 2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set
>> aside only contains user data, which we don't want to store anyway.
>
> I raised that in different context already, but such assumptions are not
> 100% future proof IMHO. For example, we might at one point be able to
> make user page tables movable and place them on there.
>
> But yes, most kernel data structures (which you care about) will
> probably never be movable and never end up on these regions.
>
>> We make a
>> filesystem in there, and create a kernel crash dump using 'makedumpfile':
>>
>> mkfs.ext4 /dev/pmem0
>> mount /dev/pmem0 /mnt
>> makedumpfile -c -d 31 /proc/vmcore /mnt/kdump
>>
>> We then set up the next full kernel boot to also have this same PMEM region,
>> using the same memmap kernel parameter. We reboot back into a full kernel.
>>
>> 3) The next full kernel will be a normal boot with a full networking stack,
>> SSD drivers, disk encryption, etc. We mount up our PMEM filesystem, pull out
>> the kdump and either store it somewhere persistent or upload it somewhere. We
>> can then unmount the PMEM and reconfigure it back to system ram so that the
>> live system isn't missing memory.
>>
>> ndctl create-namespace --reconfig=namespace0.0 -m devdax -f
>> daxctl reconfigure-device --mode=system-ram dax0.0
>>
>> This is the flow I'm trying to support, and have mostly working in a VM,
>> except up until now makedumpfile would crash because all the memblock
>> structures it needed were in the PMEM area that I had just wiped out by making
>> a new filesystem. :)
>
>
> Thinking out loud (and remembering that some architectures relocate the
> crashkernel during kexec, if I am not wrong), maybe the following would
> also work and make your setup eventually easier:
>
> 1) Don't reserve a crashkernel area in the traditional way, instead
> reserve that area using CMA. It can be used for MOVABLE allocations.
>
> 2) Let kexec load the crashkernel+initrd into ordinary memory only
> (consuming as much as you would need there).

Oh, I realized that one might be able to place the kernel+initrd
directly in the area by allocating via CMA.

--
Cheers,

David / dhildenb

\
 
 \ /
  Last update: 2023-07-26 15:10    [W:0.086 / U:0.916 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site