lkml.org 
[lkml]   [2021]   [Jun]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Date
    SubjectRe: [PATCH v1] proc: Implement /proc/self/meminfo
    Shakeel Butt <shakeelb@google.com> writes:

    > On Tue, Jun 15, 2021 at 5:47 AM Alexey Gladkov <legion@kernel.org> wrote:
    >>
    > [...]
    >>
    >> I made the second version of the patch [1], but then I had a conversation
    >> with Eric W. Biederman offlist. He convinced me that it is a bad idea to
    >> change all the values in meminfo to accommodate cgroups. But we agreed
    >> that MemAvailable in /proc/meminfo should respect cgroups limits. This
    >> field was created to hide implementation details when calculating
    >> available memory. You can see that it is quite widely used [2].
    >> So I want to try to move in that direction.
    >>
    >> [1] https://git.kernel.org/pub/scm/linux/kernel/git/legion/linux.git/log/?h=patchset/meminfo/v2.0
    >> [2] https://codesearch.debian.net/search?q=MemAvailable%3A
    >>
    >
    > Please see following two links on the previous discussion on having
    > per-memcg MemAvailable stat.
    >
    > [1] https://lore.kernel.org/linux-mm/alpine.DEB.2.22.394.2006281445210.855265@chino.kir.corp.google.com/
    > [2] https://lore.kernel.org/linux-mm/alpine.DEB.2.23.453.2007142018150.2667860@chino.kir.corp.google.com/
    >
    > MemAvailable itself is an imprecise metric and involving memcg makes
    > this metric even more weird. The difference of semantics of swap
    > accounting of v1 and v2 is one source of this weirdness (I have not
    > checked your patch if it is handling this weirdness). The lazyfree and
    > deferred split pages are another source.
    >
    > So, I am not sure if complicating an already imprecise metric will
    > make it more useful.

    Making a good guess at how much memory can be allocated without
    triggering swapping or otherwise stressing the system is something that
    requires understanding our mm internals.

    To be able to continue changing the mm or even mm policy without
    introducing regressions in userspace we need to export values that
    userspace can use.

    At a first approximation that seems to look like MemAvailable.

    MemAvailable seems to have a good definition. Roughly the amount of
    memory that can be allocated without triggering swapping. Updated
    to include not trigger memory cgroup based swapping and I sounds good.

    I don't know if it will work in practice but I think it is worth
    exploring.

    I do know that hiding the implementation details and providing userspace
    with information it can directly use seems like the programming model
    that needs to be explored. Most programs should not care if they are in
    a memory cgroup, etc. Programs, load management systems, and even
    balloon drivers have a legitimately interest in how much additional load
    can be placed on a systems memory.


    A version of this that I remember working fairly well is free space
    on compressed filesystems. As I recall compressed filesystems report
    the amount of uncompressed space that is available (an underestimate).
    This results in the amount of space consumed going up faster than the
    free space goes down.

    We can't do exactly the same thing with our memory usability estimate,
    but having our estimate be a reliable underestimate might be enough
    to avoid problems with reporting too much memory as available to
    userspace.

    I know that MemAvailable already does that /2 so maybe it is already
    aiming at being an underestimate. Perhaps we need some additional
    accounting to help create a useful metric for userspace as well.


    I don't know the final answer. I do know that not designing an
    interface that userspace can use to deal with it's legitimate concerns
    is sticking our collective heads in the sand and wishing the problem
    will go away.

    Eric

    \
     
     \ /
      Last update: 2021-06-16 18:18    [W:3.484 / U:0.556 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site