lkml.org 
[lkml]   [2017]   [Dec]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH] mm, meminit: Serially initialise deferred memory if trace_buf_size is specified
    On Wed, Nov 29, 2017 at 10:41:59PM -0500, Pavel Tatashin wrote:
    > Hi Mel,
    >
    > Thank you very much for your feedback, my replies below:
    >
    > > A lack of involvement from admins is indeed desirable. For example,
    > > while I might concede on using a disable-everything-switch, I would not
    > > be happy to introduce a switch that specified how much memory per node
    > > to initialise.
    > >
    > > For the forth approach, I really would be only thinking of a blunt
    > > "initialise everything instead of going OOM". I was wary of making things
    > > too complicated and I worried about some side-effects I'll cover later.
    >
    > I see, I misunderstood your suggestion. Switching to serial
    > initialization when OOM works, however, boot time becomes
    > unpredictable, with some configurations boot is fast with others it is
    > slow. All of that depends on whether predictions in
    > reset_deferred_meminit() were good or not which is not easy to debug
    > for users. Also, overtime predictions in reset_deferred_meminit() can
    > become very off, and I do not think that we want to continuously
    > adjust this function.
    >

    You could increase the probabilty of a report by doing a WARN_ON_ONCE if
    the serialised meminit is used.

    > >> With this approach we could always init a very small amount of struct
    > >> pages, and allow the rest to be initialized on demand as boot requires
    > >> until deferred struct pages are initialized. Since, having deferred
    > >> pages feature assumes that the machine is large, there is no drawback
    > >> of having some extra byte of dead code, especially that all the checks
    > >> can be permanently switched of via static branches once deferred init
    > >> is complete.
    > >>
    > >
    > > This is where I fear there may be dragons. If we minimse the number of
    > > struct pages and initialise serially as necessary, there is a danger that
    > > we'll allocate remote memory in cases where local memory would have done
    > > because a remote node had enough memory.
    >
    > True, but is not what we have now has the same issue as well? If one
    > node is gets out of memory we start using memory from another node,
    > before deferred pages are initialized?
    >

    It's possible but I'm not aware of it happening currently.

    > To offset that risk, it would be
    > > necessary at boot-time to force allocations from local node where possible
    > > and initialise more memory as necessary. That starts getting complicated
    > > because we'd need to adjust gfp-flags in the fast path with init-and-retry
    > > logic in the slow path and that could be a constant penalty. We could offset
    > > that in the fast path by using static branches
    >
    > I will try to implement this, and see how complicated the patch will
    > be, if it gets too complicated for the problem I am trying to solve we
    > can return to one of your suggestions.
    >
    > I was thinking to do something like this:
    >
    > Start with every small amount of initialized pages in every node.
    > If allocation fails, initialize enough struct pages to cover this
    > particular allocation with struct pages rounded up to section size but
    > in every single node.
    >

    Ok, just make sure it's all in the slow paths of the allocator when the
    alternative is to fail the allocation.

    > > but it's getting more and
    > > more complex for what is a minor optimisation -- shorter boot times on
    > > large machines where userspace itself could take a *long* time to get up
    > > and running (think database reading in 1TB of data from disk as it warms up).
    >
    > On M6-32 with 32T [1] of memory it saves over 4 minutes of boot time,
    > and this is on SPARC with 8K pages, on x86 it would be around of 8
    > minutes because of twice as many pages. This feature improves
    > availability for larger machines quite a bit. Overtime, systems are
    > growing, so I expect this feature to become a default configuration in
    > the next several years on server configs.
    >

    Ok, when developing the series originally, I had no machine even close
    to 32T of memory.

    --
    Mel Gorman
    SUSE Labs

    \
     
     \ /
      Last update: 2017-12-06 11:50    [W:4.061 / U:0.180 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site