Messages in this thread | | | Date | Mon, 30 Oct 2023 17:43:42 +0800 | Subject | Re: [PATCH RFC 1/2] RDMA/rxe: don't allow registering !PAGE_SIZE mr | From | Zhu Yanjun <> |
| |
在 2023/10/30 16:13, Zhijian Li (Fujitsu) 写道: > > > On 27/10/2023 16:17, Zhu Yanjun wrote: >> 在 2023/10/27 13:41, Li Zhijian 写道: >>> mr->page_list only encodes *page without page offset, when >>> page_size != PAGE_SIZE, we cannot restore the address with a wrong >>> page_offset. >>> >>> Note that this patch will break some ULPs that try to register 4K >>> MR when PAGE_SIZE is not 4K. >>> SRP and nvme over RXE is known to be impacted. >> >> When ULP uses folio or compound page, ULP can not work well with RXE after this commit is applied. >> >> Perhaps removing page_size set in RXE is a good solution because page_size is set twice, firstly page_size is set in infiniband/core, secondly it is set in RXE. > > Does The RXE one mean rxe_mr_init(), I think rxe_reg_user_mr() requires this.
Please read the discussions carefully. This problem has been discussed.
Best Regards, Zhu Yanjun
> > 48 static void rxe_mr_init(int access, struct rxe_mr *mr) > 49 { > 50 u32 key = mr->elem.index << 8 | rxe_get_next_key(-1); > 51 > 52 /* set ibmr->l/rkey and also copy into private l/rkey > 53 * for user MRs these will always be the same > 54 * for cases where caller 'owns' the key portion > 55 * they may be different until REG_MR WQE is executed. > 56 */ > 57 mr->lkey = mr->ibmr.lkey = key; > 58 mr->rkey = mr->ibmr.rkey = key; > 59 > 60 mr->access = access; > 61 mr->ibmr.page_size = PAGE_SIZE; > 62 mr->page_mask = PAGE_MASK; > 63 mr->page_shift = PAGE_SHIFT; > 64 mr->state = RXE_MR_STATE_INVALID; > 65 } > > > Thanks > Zhijian > >> >> When folio or compound page is used in ULP, it is very possible that page_size in infiniband/core is different from the page_size in RXE >> >> Not sure what problem this difference will cause. >> >> Zhu Yanjun >> >>> >>> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com> >>> --- >>> drivers/infiniband/sw/rxe/rxe_mr.c | 6 ++++++ >>> 1 file changed, 6 insertions(+) >>> >>> diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c >>> index f54042e9aeb2..61a136ea1d91 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe_mr.c >>> +++ b/drivers/infiniband/sw/rxe/rxe_mr.c >>> @@ -234,6 +234,12 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, >>> struct rxe_mr *mr = to_rmr(ibmr); >>> unsigned int page_size = mr_page_size(mr); >>> + if (page_size != PAGE_SIZE) { >>> + rxe_info_mr(mr, "Unsupported MR with page_size %u, expect %lu\n", >>> + page_size, PAGE_SIZE); >>> + return -EOPNOTSUPP; >>> + } >>> + >>> mr->nbuf = 0; >>> mr->page_shift = ilog2(page_size); >>> mr->page_mask = ~((u64)page_size - 1);
| |