Messages in this thread | | | Date | Mon, 6 Jun 2022 11:31:20 +0530 | Subject | Re: [RFC PATCH v4 1/7] mm/demotion: Add support for explicit memory tiers | From | Aneesh Kumar K V <> |
| |
On 6/6/22 11:03 AM, Ying Huang wrote: > On Mon, 2022-06-06 at 09:26 +0530, Aneesh Kumar K V wrote: >> On 6/6/22 8:19 AM, Ying Huang wrote: >>> On Thu, 2022-06-02 at 14:07 +0800, Ying Huang wrote: >>>> On Fri, 2022-05-27 at 17:55 +0530, Aneesh Kumar K.V wrote: >>>>> From: Jagdish Gediya <jvgediya@linux.ibm.com> >>>>> >>>>> In the current kernel, memory tiers are defined implicitly via a >>>>> demotion path relationship between NUMA nodes, which is created >>>>> during the kernel initialization and updated when a NUMA node is >>>>> hot-added or hot-removed. The current implementation puts all >>>>> nodes with CPU into the top tier, and builds the tier hierarchy >>>>> tier-by-tier by establishing the per-node demotion targets based >>>>> on the distances between nodes. >>>>> >>>>> This current memory tier kernel interface needs to be improved for >>>>> several important use cases, >>>>> >>>>> The current tier initialization code always initializes >>>>> each memory-only NUMA node into a lower tier. But a memory-only >>>>> NUMA node may have a high performance memory device (e.g. a DRAM >>>>> device attached via CXL.mem or a DRAM-backed memory-only node on >>>>> a virtual machine) and should be put into a higher tier. >>>>> >>>>> The current tier hierarchy always puts CPU nodes into the top >>>>> tier. But on a system with HBM or GPU devices, the >>>>> memory-only NUMA nodes mapping these devices should be in the >>>>> top tier, and DRAM nodes with CPUs are better to be placed into the >>>>> next lower tier. >>>>> >>>>> With current kernel higher tier node can only be demoted to selected nodes on the >>>>> next lower tier as defined by the demotion path, not any other >>>>> node from any lower tier. This strict, hard-coded demotion order >>>>> does not work in all use cases (e.g. some use cases may want to >>>>> allow cross-socket demotion to another node in the same demotion >>>>> tier as a fallback when the preferred demotion node is out of >>>>> space), This demotion order is also inconsistent with the page >>>>> allocation fallback order when all the nodes in a higher tier are >>>>> out of space: The page allocation can fall back to any node from >>>>> any lower tier, whereas the demotion order doesn't allow that. >>>>> >>>>> The current kernel also don't provide any interfaces for the >>>>> userspace to learn about the memory tier hierarchy in order to >>>>> optimize its memory allocations. >>>>> >>>>> This patch series address the above by defining memory tiers explicitly. >>>>> >>>>> This patch adds below sysfs interface which is read-only and >>>>> can be used to read nodes available in specific tier. >>>>> >>>>> /sys/devices/system/memtier/memtierN/nodelist >>>>> >>>>> Tier 0 is the highest tier, while tier MAX_MEMORY_TIERS - 1 is the >>>>> lowest tier. The absolute value of a tier id number has no specific >>>>> meaning. what matters is the relative order of the tier id numbers. >>>>> >>>>> All the tiered memory code is guarded by CONFIG_TIERED_MEMORY. >>>>> Default number of memory tiers are MAX_MEMORY_TIERS(3). All the >>>>> nodes are by default assigned to DEFAULT_MEMORY_TIER(1). >>>>> >>>>> Default memory tier can be read from, >>>>> /sys/devices/system/memtier/default_tier >>>>> >>>>> Max memory tier can be read from, >>>>> /sys/devices/system/memtier/max_tiers >>>>> >>>>> This patch implements the RFC spec sent by Wei Xu <weixugc@google.com> at [1]. >>>>> >>>>> [1] https://lore.kernel.org/linux-mm/CAAPL-u-DGLcKRVDnChN9ZhxPkfxQvz9Sb93kVoX_4J2oiJSkUw@mail.gmail.com/ >>>>> >>>>> Signed-off-by: Jagdish Gediya <jvgediya@linux.ibm.com> >>>>> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> >>>> >>>> IMHO, we should change the kernel internal implementation firstly, then >>>> implement the kerne/user space interface. That is, make memory tier >>>> explicit inside kernel, then expose it to user space. >>> >>> Why ignore this comment for v5? If you don't agree, please respond me. >>> >> >> I am not sure what benefit such a rearrange would bring in? Right now I >> am writing the series from the point of view of introducing all the >> plumbing and them switching the existing demotion logic to use the new >> infrastructure. Redoing the code to hide all the userspace sysfs till we >> switch the demotion logic to use the new infrastructure doesn't really >> bring any additional clarity to patch review and would require me to >> redo the series with a lot of conflicts across the patches in the patchset. > > IMHO, we shouldn't introduce regression even in the middle of a > patchset. Each step should only rely on previous patches in the series > to work correctly. In your current way of organization, after patch > [1/7], on a system with 2 memory tiers, the user space interface will > output wrong information (only 1 memory tier). So I think the correct > way is to make it right inside the kenrel firstly, then expose the right > information to user space. >
The patchset doesn't add additional tier until "mm/demotion/dax/kmem: Set node's memory tier to MEMORY_TIER_PMEM". ie, there is no additional tiers done till all the demotion logic is in place. So even if the system got dax/kmem, the support for adding dax/kmem as a memory tier comes later in the patch series.
-aneesh
| |