lkml.org 
[lkml]   [2015]   [May]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH v2 00/10] evacuate struct page from the block layer, introduce __pfn_t

    * Dave Hansen <dave.hansen@linux.intel.com> wrote:

    > On 05/07/2015 10:42 AM, Dan Williams wrote:
    > > On Thu, May 7, 2015 at 10:36 AM, Ingo Molnar <mingo@kernel.org> wrote:
    > >> * Dan Williams <dan.j.williams@intel.com> wrote:
    > >>
    > >> So is there anything fundamentally wrong about creating struct
    > >> page backing at mmap() time (and making sure aliased mmaps share
    > >> struct page arrays)?
    > >
    > > Something like "get_user_pages() triggers memory hotplug for
    > > persistent memory", so they are actual real struct pages? Can we
    > > do memory hotplug at that granularity?
    >
    > We've traditionally limited them to SECTION_SIZE granularity, which
    > is 128MB IIRC. There are also assumptions in places that you can do
    > page++ within a MAX_ORDER block if !CONFIG_HOLES_IN_ZONE.

    I really don't think that's very practical: memory hotplug is slow,
    it's really not on the same abstraction level as mmap(), and the zone
    data structures are also fundamentally very coarse: not just because
    RAM ranges are huge, but also so that the pfn->page transformation
    stays relatively simple and fast.

    > But, in all practicality, a lot of those places are in code like the
    > buddy allocator. If your PTEs all have _PAGE_SPECIAL set and we're
    > not ever expecting these fake 'struct page's to hit these code
    > paths, it probably doesn't matter.
    >
    > You can probably get away with just allocating PAGE_SIZE worth of
    > 'struct page' (which is 64) and mapping it in to vmemmap[]. The
    > worst case is that you'll eat 1 page of space for each outstanding
    > page of I/O. That's a lot better than 2MB of temporary 'struct
    > page' space per page of I/O that it would take with a traditional
    > hotplug operation.

    So I think the main value of struct page is if everyone on the system
    sees the same struct page for the same pfn - not just the temporary IO
    instance.

    The idea of having very temporary struct page arrays misses the point
    I think: if struct page is used as essentially an IO sglist then most
    of the synchronization properties are lost: then we might as well use
    the real deal in that case and skip the dynamic allocation and use
    pfns directly and avoid the dynamic allocation overhead.

    Stable, global page-struct descriptors are a given for real RAM, where
    we allocate a struct page for every page in nice, large, mostly linear
    arrays.

    We'd really need that for pmem too, to get the full power of struct
    page: and that means allocating them in nice, large, predictable
    places - such as on the device itself ...

    It might even be 'scattered' across the device, with 64 byte struct
    page size we can pack 64 descriptors into a single page, so every 65
    pages we could have a page-struct page.

    Finding a pmem page's struct page would thus involve rounding it
    modulo 65 and reading that page.

    The problem with that is fourfold:

    - that we now turn a very kernel internal API and data structure into
    an ABI. If struct page grows beyond 64 bytes it's a problem.

    - on bootup (or device discovery time) we'd have to initialize all
    the page structs. We could probably do this in a hierarchical way,
    by dividing continuous pmem ranges into power-of-two groups of
    blocks, and organizing them like the buddy allocator does.

    - 1.5% of storage space lost.

    - will wear-leveling properly migrate these 'hot' pages around?

    The alternative would be some global interval-rbtree of struct page
    backed pmem ranges.

    Beyond the synchronization problems of such a data structure (which
    looks like a nightmare) I don't think it's even feasible: especially
    if there's a filesystem on the pmem device then the block allocations
    could be physically fragmented (and there's no fundamental reason why
    they couldn't be fragmented), so a continuous mmap() of a file on it
    will yield wildly fragmented device-pfn ranges, exploding the rbtree.
    Think 1 million node interval-rbtree with an average depth of 20:
    cachemiss country for even simple lookups - not to mention the
    freeing/recycling complexity of unused struct pages to not allow it to
    grow too large.

    I might be wrong though about all this :)

    Thanks,

    Ingo


    \
     
     \ /
      Last update: 2015-05-07 21:21    [W:7.887 / U:0.152 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site