Messages in this thread | | | Subject | Re: [PATCH RFC net-next 2/4] net: page_pool: avoid calling no-op externals when possible | From | Alexander H Duyck <> | Date | Thu, 29 Jun 2023 09:45:26 -0700 |
| |
On Thu, 2023-06-29 at 17:23 +0200, Alexander Lobakin wrote: > Turned out page_pool_put{,_full}_page() can burn quite a bunch of cycles > even when on DMA-coherent platforms (like x86) with no active IOMMU or > swiotlb, just for the call ladder. > Indeed, it's > > page_pool_put_page() > page_pool_put_defragged_page() <- external > __page_pool_put_page() > page_pool_dma_sync_for_device() <- non-inline > dma_sync_single_range_for_device() > dma_sync_single_for_device() <- external > dma_direct_sync_single_for_device() > dev_is_dma_coherent() <- exit > > For the inline functions, no guarantees the compiler won't uninline them > (they're clearly not one-liners and sometimes compilers uninline even > 2 + 2). The first external call is necessary, but the rest 2+ are done > for nothing each time, plus a bunch of checks here and there. > Since Page Pool mappings are long-term and for one "device + addr" pair > dma_need_sync() will always return the same value (basically, whether it > belongs to an swiotlb pool), addresses can be tested once right after > they're obtained and the result can be reused until the page is unmapped. > Define new PP flag, which will mean "do DMA syncs for device, but only > when needed" and turn it on by default when the driver asks to sync > pages. When a page is mapped, check whether it needs syncs and if so, > replace that "sync when needed" back to "always do syncs" globally for > the whole pool (better safe than sorry). As long as a pool has no pages > requiring DMA syncs, this cuts off a good piece of calls and checks. > On my x86_64, this gives from 2% to 5% performance benefit with no > negative impact for cases when IOMMU is on and the shortcut can't be > used. > > Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com> > --- > include/net/page_pool.h | 3 +++ > net/core/page_pool.c | 10 ++++++++++ > 2 files changed, 13 insertions(+) > > diff --git a/include/net/page_pool.h b/include/net/page_pool.h > index 829dc1f8ba6b..ff3772fab707 100644 > --- a/include/net/page_pool.h > +++ b/include/net/page_pool.h > @@ -23,6 +23,9 @@ > * Please note DMA-sync-for-CPU is still > * device driver responsibility > */ > +#define PP_FLAG_DMA_MAYBE_SYNC BIT(2) /* Internal, should not be used in > + * drivers > + */ > #define PP_FLAG_ALL (PP_FLAG_DMA_MAP |\ > PP_FLAG_DMA_SYNC_DEV) > > diff --git a/net/core/page_pool.c b/net/core/page_pool.c > index dff0b4fa2316..498e058140b3 100644 > --- a/net/core/page_pool.c > +++ b/net/core/page_pool.c > @@ -197,6 +197,10 @@ static int page_pool_init(struct page_pool *pool, > /* pool->p.offset has to be set according to the address > * offset used by the DMA engine to start copying rx data > */ > + > + /* Try to avoid calling no-op syncs */ > + pool->p.flags |= PP_FLAG_DMA_MAYBE_SYNC; > + pool->p.flags &= ~PP_FLAG_DMA_SYNC_DEV; > } > > #ifdef CONFIG_PAGE_POOL_STATS > @@ -341,6 +345,12 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) > > page_pool_set_dma_addr(page, dma); > > + if ((pool->p.flags & PP_FLAG_DMA_MAYBE_SYNC) && > + dma_need_sync(pool->p.dev, dma)) { > + pool->p.flags |= PP_FLAG_DMA_SYNC_DEV; > + pool->p.flags &= ~PP_FLAG_DMA_MAYBE_SYNC; > + } > + > if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) > page_pool_dma_sync_for_device(pool, page, pool->p.max_len); >
I am pretty sure the logic is flawed here. The problem is dma_needs_sync depends on the DMA address being used. In the worst case scenario we could have a device that has something like a 32b DMA address space on a system with over 4GB of memory. In such a case the higher addresses would need to be synced because they will go off to a swiotlb bounce buffer while the lower addresses wouldn't.
If you were to store a flag like this it would have to be generated per page.
| |