Messages in this thread | | | Date | Mon, 19 Nov 2018 21:02:29 +0200 | From | Leon Romanovsky <> | Subject | Re: [RFCv3 PATCH 1/6] uacce: Add documents for WarpDrive/uacce |
| |
On Mon, Nov 19, 2018 at 01:42:16PM -0500, Jerome Glisse wrote: > On Mon, Nov 19, 2018 at 11:27:52AM -0700, Jason Gunthorpe wrote: > > On Mon, Nov 19, 2018 at 11:48:54AM -0500, Jerome Glisse wrote: > > > > > Just to comment on this, any infiniband driver which use umem and do > > > not have ODP (here ODP for me means listening to mmu notifier so all > > > infiniband driver except mlx5) will be affected by same issue AFAICT. > > > > > > AFAICT there is no special thing happening after fork() inside any of > > > those driver. So if parent create a umem mr before fork() and program > > > hardware with it then after fork() the parent might start using new > > > page for the umem range while the old memory is use by the child. The > > > reverse is also true (parent using old memory and child new memory) > > > bottom line you can not predict which memory the child or the parent > > > will use for the range after fork(). > > > > > > So no matter what you consider the child or the parent, what the hw > > > will use for the mr is unlikely to match what the CPU use for the > > > same virtual address. In other word: > > > > > > Before fork: > > > CPU parent: virtual addr ptr1 -> physical address = 0xCAFE > > > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE > > > > > > Case 1: > > > CPU parent: virtual addr ptr1 -> physical address = 0xCAFE > > > CPU child: virtual addr ptr1 -> physical address = 0xDEAD > > > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE > > > > > > Case 2: > > > CPU parent: virtual addr ptr1 -> physical address = 0xBEEF > > > CPU child: virtual addr ptr1 -> physical address = 0xCAFE > > > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE > > > > IIRC this is solved in IB by automatically calling > > madvise(MADV_DONTFORK) before creating the MR. > > > > MADV_DONTFORK > > .. This is useful to prevent copy-on-write semantics from changing the > > physical location of a page if the parent writes to it after a > > fork(2) .. > > This would work around the issue but this is not transparent ie > range marked with DONTFORK no longer behave as expected from the > application point of view. > > Also it relies on userspace doing the right thing (which is not > something i usualy trust :)).
The good thing that we didn't see anyone who succeeded to run IB stack without our user space, which does right thing under the hood :).
> > Cheers, > Jérôme [unhandled content-type:application/pgp-signature] | |