Messages in this thread | | | Date | Wed, 30 Jan 2013 18:18:54 +0800 | From | Tang Chen <> | Subject | Re: [PATCH v6 00/15] memory-hotplug: hot-remove physical memory |
| |
On 01/30/2013 06:15 PM, Tang Chen wrote: > Hi Simon, > > Please see below. :) > > On 01/29/2013 08:52 PM, Simon Jeons wrote: >> Hi Tang, >> >> On Wed, 2013-01-09 at 17:32 +0800, Tang Chen wrote: >>> Here is the physical memory hot-remove patch-set based on 3.8rc-2. >> >> Some questions ask you, not has relationship with this patchset, but is >> memory hotplug stuff. >> >> 1. In function node_states_check_changes_online: >> >> comments: >> * If we don't have HIGHMEM nor movable node, >> * node_states[N_NORMAL_MEMORY] contains nodes which have zones of >> * 0...ZONE_MOVABLE, set zone_last to ZONE_MOVABLE. >> >> How to understand it? Why we don't have HIGHMEM nor movable node and >> node_staes[N_NORMAL_MEMORY] contains 0...ZONE_MOVABLE, IIUC, >> N_NORMAL_MEMORY only means the node has regular memory. >> > > First of all, I think we need to understand why we need N_MEMORY. > > In order to support movable node, which has only ZONE_MOVABLE (the last > zone), > we introduce N_MEMORY to represent the node has normal, highmem and > movable memory. > > Here, "we have movable node" means you configured CONFIG_MOVABLE_NODE.
Sorry, should be "we don't have movable node" means you didn't configured CONFIG_MOVABLE_NODE.
> This config option doesn't mean we don't have movable pages, (NO) > it means we don't have a node which has only movable pages (only have > ZONE_MOVABLE). (YES) > > Here, if we don't have CONFIG_MOVABLE_NODE (we don't have movable node), > we don't need a > separate node_states[] element to represent a particular node because we > won't have a node > which has only ZONE_MOVABLE. > > So, > 1) if we don't have highmem nor movable node, N_MEMORY == N_HIGH_MEMORY > == N_NORMAL_MEMORY, > which means N_NORMAL_MEMORY effects as N_MEMORY. If we online pages as > movable, we need > to update node_states[N_NORMAL_MEMORY]. > > Please refer to the definition of enum zone_type, if we don't have > CONFIG_HIGHMEM, we won't > have ZONE_HIGHMEM, but ZONE_NORMAL and ZONE_MOVABLE will always there. > So we can have movable > pages, and the zone_last should be ZONE_MOVABLE. > > Again, because we won't have a node only having ZONE_MOVABLE, so we just > need to update > node_states[N_NORMAL_MEMORY]. > >> * If we don't have movable node, node_states[N_NORMAL_MEMORY] >> * contains nodes which have zones of 0...ZONE_MOVABLE, >> * set zone_last to ZONE_MOVABLE. >> >> How to understand? > > 2) this code is in #ifdef CONFIG_HIGHMEM, which means we have highmem, > so if we don't have > movable node, N_MEMORY == N_HIGH_MEMORY, and N_HIGH_MEMORY effects as > N_MEMORY. If we > online pages as movable, we need to update node_states[N_NORMAL_MEMORY]. > >> >> 2. In function move_pfn_range_left, why end<= z2->zone_start_pfn is not >> correct? The comments said that must include/overlap, why? >> > > This one is easy, if I understand you correctly. > move_pfn_range_left() is used to move the left most part [start_pfn, > end_pfn) of z2 to z1. > So if end_pfn<= z2->zone_start_pfn, it means [start_pfn, end_pfn) is not > part of z2. > Then it fails. > >> 3. In function online_pages, the normal case(w/o online_kenrel, >> online_movable), why not check if the new zone is overlap with adjacent >> zones? >> > > Can a zone overlap with the others ? I don't think so. > > One pfn could only be in one zone, > zone = page_zone(pfn_to_page(pfn)); > > it could overlap with others, I think. :) > > But maybe I misunderstand you. :) > >> 4. Could you summarize the difference implementation between hot-add and >> logic-add, hot-remove and logic-remove? > > Sorry, I don't quite understand what do you mean by logic-add/remove. > Would you please explain more ? > > If you meant the sys fs interfaces, I think they are just another set of > entrances > of memory hotplug. > > Thanks. :) > >> >> >>> >>> This patch-set aims to implement physical memory hot-removing. >>> >>> The patches can free/remove the following things: >>> >>> - /sys/firmware/memmap/X/{end, start, type} : [PATCH 4/15] >>> - memmap of sparse-vmemmap : [PATCH 6,7,8,10/15] >>> - page table of removed memory : [RFC PATCH 7,8,10/15] >>> - node and related sysfs files : [RFC PATCH 13-15/15] >>> >>> >>> Existing problem: >>> If CONFIG_MEMCG is selected, we will allocate memory to store page >>> cgroup >>> when we online pages. >>> >>> For example: there is a memory device on node 1. The address range >>> is [1G, 1.5G). You will find 4 new directories memory8, memory9, >>> memory10, >>> and memory11 under the directory /sys/devices/system/memory/. >>> >>> If CONFIG_MEMCG is selected, when we online memory8, the memory >>> stored page >>> cgroup is not provided by this memory device. But when we online >>> memory9, the >>> memory stored page cgroup may be provided by memory8. So we can't >>> offline >>> memory8 now. We should offline the memory in the reversed order. >>> >>> When the memory device is hotremoved, we will auto offline memory >>> provided >>> by this memory device. But we don't know which memory is onlined >>> first, so >>> offlining memory may fail. >>> >>> In patch1, we provide a solution which is not good enough: >>> Iterate twice to offline the memory. >>> 1st iterate: offline every non primary memory block. >>> 2nd iterate: offline primary (i.e. first added) memory block. >>> >>> And a new idea from Wen Congyang<wency@cn.fujitsu.com> is: >>> allocate the memory from the memory block they are describing. >>> >>> But we are not sure if it is OK to do so because there is not >>> existing API >>> to do so, and we need to move page_cgroup memory allocation from >>> MEM_GOING_ONLINE >>> to MEM_ONLINE. And also, it may interfere the hugepage. >>> >>> >>> >>> How to test this patchset? >>> 1. apply this patchset and build the kernel. MEMORY_HOTPLUG, >>> MEMORY_HOTREMOVE, >>> ACPI_HOTPLUG_MEMORY must be selected. >>> 2. load the module acpi_memhotplug >>> 3. hotplug the memory device(it depends on your hardware) >>> You will see the memory device under the directory >>> /sys/bus/acpi/devices/. >>> Its name is PNP0C80:XX. >>> 4. online/offline pages provided by this memory device >>> You can write online/offline to >>> /sys/devices/system/memory/memoryX/state to >>> online/offline pages provided by this memory device >>> 5. hotremove the memory device >>> You can hotremove the memory device by the hardware, or writing 1 to >>> /sys/bus/acpi/devices/PNP0C80:XX/eject. >> >> Is there a similar knode to hot-add the memory device? >> >>> >>> >>> Note: if the memory provided by the memory device is used by the >>> kernel, it >>> can't be offlined. It is not a bug. >>> >>> >>> Changelogs from v5 to v6: >>> Patch3: Add some more comments to explain memory hot-remove. >>> Patch4: Remove bootmem member in struct firmware_map_entry. >>> Patch6: Repeatedly register bootmem pages when using hugepage. >>> Patch8: Repeatedly free bootmem pages when using hugepage. >>> Patch14: Don't free pgdat when offlining a node, just reset it to 0. >>> Patch15: New patch, pgdat is not freed in patch14, so don't allocate >>> a new >>> one when online a node. >>> >>> Changelogs from v4 to v5: >>> Patch7: new patch, move pgdat_resize_lock into >>> sparse_remove_one_section() to >>> avoid disabling irq because we need flush tlb when free pagetables. >>> Patch8: new patch, pick up some common APIs that are used to free >>> direct mapping >>> and vmemmap pagetables. >>> Patch9: free direct mapping pagetables on x86_64 arch. >>> Patch10: free vmemmap pagetables. >>> Patch11: since freeing memmap with vmemmap has been implemented, the >>> config >>> macro CONFIG_SPARSEMEM_VMEMMAP when defining __remove_section() is >>> no longer needed. >>> Patch13: no need to modify acpi_memory_disable_device() since it was >>> removed, >>> and add nid parameter when calling remove_memory(). >>> >>> Changelogs from v3 to v4: >>> Patch7: remove unused codes. >>> Patch8: fix nr_pages that is passed to free_map_bootmem() >>> >>> Changelogs from v2 to v3: >>> Patch9: call sync_global_pgds() if pgd is changed >>> Patch10: fix a problem int the patch >>> >>> Changelogs from v1 to v2: >>> Patch1: new patch, offline memory twice. 1st iterate: offline every >>> non primary >>> memory block. 2nd iterate: offline primary (i.e. first added) memory >>> block. >>> >>> Patch3: new patch, no logical change, just remove reduntant codes. >>> >>> Patch9: merge the patch from wujianguo into this patch. flush tlb on >>> all cpu >>> after the pagetable is changed. >>> >>> Patch12: new patch, free node_data when a node is offlined. >>> >>> >>> Tang Chen (6): >>> memory-hotplug: move pgdat_resize_lock into >>> sparse_remove_one_section() >>> memory-hotplug: remove page table of x86_64 architecture >>> memory-hotplug: remove memmap of sparse-vmemmap >>> memory-hotplug: Integrated __remove_section() of >>> CONFIG_SPARSEMEM_VMEMMAP. >>> memory-hotplug: remove sysfs file of node >>> memory-hotplug: Do not allocate pdgat if it was not freed when >>> offline. >>> >>> Wen Congyang (5): >>> memory-hotplug: try to offline the memory twice to avoid dependence >>> memory-hotplug: remove redundant codes >>> memory-hotplug: introduce new function arch_remove_memory() for >>> removing page table depends on architecture >>> memory-hotplug: Common APIs to support page tables hot-remove >>> memory-hotplug: free node_data when a node is offlined >>> >>> Yasuaki Ishimatsu (4): >>> memory-hotplug: check whether all memory blocks are offlined or not >>> when removing memory >>> memory-hotplug: remove /sys/firmware/memmap/X sysfs >>> memory-hotplug: implement register_page_bootmem_info_section of >>> sparse-vmemmap >>> memory-hotplug: memory_hotplug: clear zone when removing the memory >>> >>> arch/arm64/mm/mmu.c | 3 + >>> arch/ia64/mm/discontig.c | 10 + >>> arch/ia64/mm/init.c | 18 ++ >>> arch/powerpc/mm/init_64.c | 10 + >>> arch/powerpc/mm/mem.c | 12 + >>> arch/s390/mm/init.c | 12 + >>> arch/s390/mm/vmem.c | 10 + >>> arch/sh/mm/init.c | 17 ++ >>> arch/sparc/mm/init_64.c | 10 + >>> arch/tile/mm/init.c | 8 + >>> arch/x86/include/asm/pgtable_types.h | 1 + >>> arch/x86/mm/init_32.c | 12 + >>> arch/x86/mm/init_64.c | 390 +++++++++++++++++++++++++++++ >>> arch/x86/mm/pageattr.c | 47 ++-- >>> drivers/acpi/acpi_memhotplug.c | 8 +- >>> drivers/base/memory.c | 6 + >>> drivers/firmware/memmap.c | 96 +++++++- >>> include/linux/bootmem.h | 1 + >>> include/linux/firmware-map.h | 6 + >>> include/linux/memory_hotplug.h | 15 +- >>> include/linux/mm.h | 4 +- >>> mm/memory_hotplug.c | 459 +++++++++++++++++++++++++++++++--- >>> mm/sparse.c | 8 +- >>> 23 files changed, 1094 insertions(+), 69 deletions(-) >>> >>> -- >>> To unsubscribe, send a message with 'unsubscribe linux-mm' in >>> the body to majordomo@kvack.org. For more info on Linux MM, >>> see: http://www.linux-mm.org/ . >>> Don't email:<a href=mailto:"dont@kvack.org"> email@kvack.org</a> >> >> >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-acpi" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >
| |