lkml.org 
[lkml]   [2023]   [May]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    From
    Subject[PATCH net-next 06/11] net: page_pool: avoid calling no-op externals when possible
    Date
    Turned out page_pool_put{,_full}_page() can burn quite a bunch of cycles
    even when on DMA-coherent platforms (like x86) with no active IOMMU or
    swiotlb, just for the call ladder.
    Indeed, it's

    page_pool_put_page()
    page_pool_put_defragged_page() <- external
    __page_pool_put_page()
    page_pool_dma_sync_for_device() <- non-inline
    dma_sync_single_range_for_device()
    dma_sync_single_for_device() <- external
    dma_direct_sync_single_for_device()
    dev_is_dma_coherent() <- exit

    For the inline functions, no guarantees the compiler won't uninline them
    (they're clearly not one-liners and sometimes compilers uninline even
    2 + 2). The first external call is necessary, but the rest 2+ are done
    for nothing each time, plus a bunch of checks here and there.
    Since Page Pool mappings are long-term and for one "device + addr" pair
    dma_need_sync() will always return the same value (basically, whether it
    belongs to an swiotlb pool), addresses can be tested once right after
    they're obtained and the result can be reused until the page is unmapped.
    Define new PP flag, which will mean "do DMA syncs for device, but only
    when needed" and turn it on by default when the driver asks to sync
    pages. When a page is mapped, check whether it needs syncs and if so,
    replace that "sync when needed" back to "always do syncs" globally for
    the whole pool (better safe than sorry). As long as a pool has no pages
    requiring DMA syncs, this cuts off a good piece of calls and checks.
    On my x86_64, this gives from 2% to 5% performance benefit with no
    negative impact for cases when IOMMU is on and the shortcut can't be
    used.

    Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
    ---
    include/net/page_pool.h | 3 +++
    net/core/page_pool.c | 10 ++++++++++
    2 files changed, 13 insertions(+)

    diff --git a/include/net/page_pool.h b/include/net/page_pool.h
    index c8ec2f34722b..8435013de06e 100644
    --- a/include/net/page_pool.h
    +++ b/include/net/page_pool.h
    @@ -46,6 +46,9 @@
    * device driver responsibility
    */
    #define PP_FLAG_PAGE_FRAG BIT(2) /* for page frag feature */
    +#define PP_FLAG_DMA_MAYBE_SYNC BIT(3) /* Internal, should not be used in the
    + * drivers
    + */
    #define PP_FLAG_ALL (PP_FLAG_DMA_MAP |\
    PP_FLAG_DMA_SYNC_DEV |\
    PP_FLAG_PAGE_FRAG)
    diff --git a/net/core/page_pool.c b/net/core/page_pool.c
    index e212e9d7edcb..57f323dee6c4 100644
    --- a/net/core/page_pool.c
    +++ b/net/core/page_pool.c
    @@ -175,6 +175,10 @@ static int page_pool_init(struct page_pool *pool,
    /* pool->p.offset has to be set according to the address
    * offset used by the DMA engine to start copying rx data
    */
    +
    + /* Try to avoid calling no-op syncs */
    + pool->p.flags |= PP_FLAG_DMA_MAYBE_SYNC;
    + pool->p.flags &= ~PP_FLAG_DMA_SYNC_DEV;
    }

    if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT &&
    @@ -323,6 +327,12 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page)

    page_pool_set_dma_addr(page, dma);

    + if ((pool->p.flags & PP_FLAG_DMA_MAYBE_SYNC) &&
    + dma_need_sync(pool->p.dev, dma)) {
    + pool->p.flags |= PP_FLAG_DMA_SYNC_DEV;
    + pool->p.flags &= ~PP_FLAG_DMA_MAYBE_SYNC;
    + }
    +
    if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV)
    page_pool_dma_sync_for_device(pool, page, pool->p.max_len);

    --
    2.40.1
    \
     
     \ /
      Last update: 2023-05-16 18:21    [W:4.278 / U:0.000 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site