[lkml]   [2012]   [Nov]   [28]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: [PATCH v2 0/5] Add movablecore_map boot option
On Wed, Nov 28, 2012 at 04:47:42PM +0800, Jiang Liu wrote:
>Hi all,
> Seems it's a great chance to discuss about the memory hotplug feature
>within this thread. So I will try to give some high level thoughts about memory
>hotplug feature on x86/IA64. Any comments are welcomed!
> First of all, I think usability really matters. Ideally, memory hotplug
>feature should just work out of box, and we shouldn't expect administrators to
>add several extra platform dependent parameters to enable memory hotplug.
>But how to enable memory (or CPU/node) hotplug out of box? I think the key point
>is to cooperate with BIOS/ACPI/firmware/device management teams.
> I still position memory hotplug as an advanced feature for high end
>servers and those systems may/should provide some management interfaces to
>configure CPU/memory/node hotplug features. The configuration UI may be provided
>by BIOS, BMC or centralized system management suite. Once administrator enables
>hotplug feature through those management UI, OS should support system device
>hotplug out of box. For example, HP SuperDome2 management suite provides interface
>to configure a node as floating node(hot-removable). And OpenSolaris supports
>CPU/memory hotplug out of box without any extra configurations. So we should
>shape interfaces between firmware and OS to better support system device hotplug.
> On the other hand, I think there are no commercial available x86/IA64
>platforms with system device hotplug capabilities in the field yet, at least only
>limited quantity if any. So backward compatibility is not a big issue for us now.
>So I think it's doable to rely on firmware to provide better support for system
>device hotplug.
> Then what should be enhanced to better support system device hotplug?
>1) ACPI specification should be enhanced to provide a static table to describe
>components with hotplug features, so OS could reserve special resources for
>hotplug at early boot stages. For example, to reserve enough CPU ids for CPU
>hot-add. Currently we guess maximum number of CPUs supported by the platform
>by counting CPU entries in APIC table, that's not reliable.
>2) BIOS should implement SRAT, MPST and PMTT tables to better support memory
>hotplug. SRAT associates memory ranges with proximity domains with an extra
>"hotpluggable" flag. PMTT provides memory device topology information, such
>as "socket->memory controller->DIMM". MPST is used for memory power management
>and provides a way to associate memory ranges with memory devices in PMTT.
>With all information from SRAT, MPST and PMTT, OS could figure out hotplug
>memory ranges automatically, so no extra kernel parameters needed.
>3) Enhance ACPICA to provide a method to scan static ACPI tables before
>memory subsystem has been initialized because OS need to access SRAT,
>MPST and PMTT when initializing memory subsystem.
>4) The last and the most important issue is how to minimize performance
>drop caused by memory hotplug. As proposed by this patchset, once we
>configure all memory of a NUMA node as movable, it essentially disable
>NUMA optimization of kernel memory allocation from that node. According
>to experience, that will cause huge performance drop. We have observed
>10-30% performance drop with memory hotplug enabled. And on another
>OS the average performance drop caused by memory hotplug is about 10%.
>If we can't resolve the performance drop, memory hotplug is just a feature
>for demo:( With help from hardware, we do have some chances to reduce
>performance penalty caused by memory hotplug.
> As we know, Linux could migrate movable page, but can't migrate
>non-movable pages used by kernel/DMA etc. And the most hard part is how
>to deal with those unmovable pages when hot-removing a memory device.
>Now hardware has given us a hand with a technology named memory migration,
>which could transparently migrate memory between memory devices. There's
>no OS visible changes except NUMA topology before and after hardware memory
> And if there are multiple memory devices within a NUMA node,
>we could configure some memory devices to host unmovable memory and the
>other to host movable memory. With this configuration, there won't be
>bigger performance drop because we have preserved all NUMA optimizations.
>We also could achieve memory hotplug remove by:
>1) Use existing page migration mechanism to reclaim movable pages.
>2) For memory devices hosting unmovable pages, we need:
>2.1) find a movable memory device on other nodes with enough capacity
>and reclaim it.
>2.2) use hardware migration technology to migrate unmovable memory to

Hi Jiang,

Could you give an explanation how hardware migration technology works?


>the just reclaimed memory device on other nodes.
> I hope we could expect users to adopt memory hotplug technology
>with all these implemented.
> Back to this patch, we could rely on the mechanism provided
>by it to automatically mark memory ranges as movable with information
>from ACPI SRAT/MPST/PMTT tables. So we don't need administrator to
>manually configure kernel parameters to enable memory hotplug.
> Again, any comments are welcomed!
>On 2012-11-23 18:44, Tang Chen wrote:
>> [What we are doing]
>> This patchset provide a boot option for user to specify ZONE_MOVABLE memory
>> map for each node in the system.
>> movablecore_map=nn[KMG]@ss[KMG]
>> This option make sure memory range from ss to ss+nn is movable memory.
>> [Why we do this]
>> If we hot remove a memroy, the memory cannot have kernel memory,
>> because Linux cannot migrate kernel memory currently. Therefore,
>> we have to guarantee that the hot removed memory has only movable
>> memoroy.
>> Linux has two boot options, kernelcore= and movablecore=, for
>> creating movable memory. These boot options can specify the amount
>> of memory use as kernel or movable memory. Using them, we can
>> create ZONE_MOVABLE which has only movable memory.
>> But it does not fulfill a requirement of memory hot remove, because
>> even if we specify the boot options, movable memory is distributed
>> in each node evenly. So when we want to hot remove memory which
>> memory range is 0x80000000-0c0000000, we have no way to specify
>> the memory as movable memory.
>> So we proposed a new feature which specifies memory range to use as
>> movable memory.
>> [Ways to do this]
>> There may be 2 ways to specify movable memory.
>> 1. use firmware information
>> 2. use boot option
>> 1. use firmware information
>> According to ACPI spec 5.0, SRAT table has memory affinity structure
>> and the structure has Hot Pluggable Filed. See " Memory
>> Affinity Structure". If we use the information, we might be able to
>> specify movable memory by firmware. For example, if Hot Pluggable
>> Filed is enabled, Linux sets the memory as movable memory.
>> 2. use boot option
>> This is our proposal. New boot option can specify memory range to use
>> as movable memory.
>> [How we do this]
>> We chose second way, because if we use first way, users cannot change
>> memory range to use as movable memory easily. We think if we create
>> movable memory, performance regression may occur by NUMA. In this case,
>> user can turn off the feature easily if we prepare the boot option.
>> And if we prepare the boot optino, the user can select which memory
>> to use as movable memory easily.
>> [How to use]
>> Specify the following boot option:
>> movablecore_map=nn[KMG]@ss[KMG]
>> That means physical address range from ss to ss+nn will be allocated as
>> And the following points should be considered.
>> 1) If the range is involved in a single node, then from ss to the end of
>> the node will be ZONE_MOVABLE.
>> 2) If the range covers two or more nodes, then from ss to the end of
>> the node will be ZONE_MOVABLE, and all the other nodes will only
>> 3) If no range is in the node, then the node will have no ZONE_MOVABLE
>> unless kernelcore or movablecore is specified.
>> 4) This option could be specified at most MAX_NUMNODES times.
>> 5) If kernelcore or movablecore is also specified, movablecore_map will have
>> higher priority to be satisfied.
>> 6) This option has no conflict with memmap option.
>> Tang Chen (4):
>> page_alloc: add movable_memmap kernel parameter
>> page_alloc: Introduce zone_movable_limit[] to keep movable limit for
>> nodes
>> page_alloc: Make movablecore_map has higher priority
>> page_alloc: Bootmem limit with movablecore_map
>> Yasuaki Ishimatsu (1):
>> x86: get pg_data_t's memory from other node
>> Documentation/kernel-parameters.txt | 17 +++
>> arch/x86/mm/numa.c | 11 ++-
>> include/linux/memblock.h | 1 +
>> include/linux/mm.h | 11 ++
>> mm/memblock.c | 15 +++-
>> mm/page_alloc.c | 216 ++++++++++++++++++++++++++++++++++-
>> 6 files changed, 263 insertions(+), 8 deletions(-)
>> .
>To unsubscribe, send a message with 'unsubscribe linux-mm' in
>the body to For more info on Linux MM,
>see: .
>Don't email: <a href=mailto:""> </a>

 \ /
  Last update: 2012-11-29 03:21    [W:0.408 / U:0.932 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site