Messages in this thread | | | Date | Wed, 20 Jul 2022 19:38:19 +0200 | Subject | Re: [PATCH net-next v2 0/6] net/smc: Introduce virtually contiguous buffers for SMC-R | From | Wenjia Zhang <> |
| |
On 14.07.22 11:43, Wen Gu wrote: > On long-running enterprise production servers, high-order contiguous > memory pages are usually very rare and in most cases we can only get > fragmented pages. > > When replacing TCP with SMC-R in such production scenarios, attempting > to allocate high-order physically contiguous sndbufs and RMBs may result > in frequent memory compaction, which will cause unexpected hung issue > and further stability risks. > > So this patch set is aimed to allow SMC-R link group to use virtually > contiguous sndbufs and RMBs to avoid potential issues mentioned above. > Whether to use physically or virtually contiguous buffers can be set > by sysctl smcr_buf_type. > > Note that using virtually contiguous buffers will bring an acceptable > performance regression, which can be mainly divided into two parts: > > 1) regression in data path, which is brought by additional address > translation of sndbuf by RNIC in Tx. But in general, translating > address through MTT is fast. According to qperf test, this part > regression is basically less than 10% in latency and bandwidth. > (see patch 5/6 for details) > > 2) regression in buffer initialization and destruction path, which is > brought by additional MR operations of sndbufs. But thanks to link > group buffer reuse mechanism, the impact of this kind of regression > decreases as times of buffer reuse increases. > > Patch set overview: > - Patch 1/6 and 2/6 mainly about simplifying and optimizing DMA sync > operation, which will reduce overhead on the data path, especially > when using virtually contiguous buffers; > - Patch 3/6 and 4/6 introduce a sysctl smcr_buf_type to set the type > of buffers in new created link group; > - Patch 5/6 allows SMC-R to use virtually contiguous sndbufs and RMBs, > including buffer creation, destruction, MR operation and access; > - patch 6/6 extends netlink attribute for buffer type of SMC-R link group; > > v1->v2: > - Patch 5/6 fixes build issue on 32bit; > - Patch 3/6 adds description of new sysctl in smc-sysctl.rst; > > Guangguan Wang (2): > net/smc: remove redundant dma sync ops > net/smc: optimize for smc_sndbuf_sync_sg_for_device and > smc_rmb_sync_sg_for_cpu > > Wen Gu (4): > net/smc: Introduce a sysctl for setting SMC-R buffer type > net/smc: Use sysctl-specified types of buffers in new link group > net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R > net/smc: Extend SMC-R link group netlink attribute > > Documentation/networking/smc-sysctl.rst | 13 ++ > include/net/netns/smc.h | 1 + > include/uapi/linux/smc.h | 1 + > net/smc/af_smc.c | 68 +++++++-- > net/smc/smc_clc.c | 8 +- > net/smc/smc_clc.h | 2 +- > net/smc/smc_core.c | 246 +++++++++++++++++++++----------- > net/smc/smc_core.h | 20 ++- > net/smc/smc_ib.c | 44 +++++- > net/smc/smc_ib.h | 2 + > net/smc/smc_llc.c | 33 +++-- > net/smc/smc_rx.c | 92 +++++++++--- > net/smc/smc_sysctl.c | 11 ++ > net/smc/smc_tx.c | 10 +- > 14 files changed, 404 insertions(+), 147 deletions(-) > It looks good for us. Thank you! Acked-by: Wenjia Zhang <wenjia@linux.ibm.com>
| |