Messages in this thread | | | Subject | Re: [PATCH v8 00/10] Introduce VDUSE - vDPA Device in Userspace | From | Jason Wang <> | Date | Tue, 29 Jun 2021 12:10:51 +0800 |
| |
在 2021/6/28 下午1:54, Liu, Xiaodong 写道: >> Several issues: >> >> - VDUSE needs to limit the total size of the bounce buffers (64M if I was not >> wrong). Does it work for SPDK? > Yes, Jason. It is enough and works for SPDK. > Since it's a kind of bounce buffer mainly for in-flight IO, so limited size like > 64MB is enough.
Ok.
> >> - VDUSE can use hugepages but I'm not sure we can mandate hugepages (or we >> need introduce new flags for supporting this) > Same with your worry, I'm afraid too that it is a hard for a kernel module > to directly preallocate hugepage internal. > What I tried is that: > 1. A simple agent daemon (represents for one device) `preallocates` and maps > dozens of 2MB hugepages (like 64MB) for one device. > 2. The daemon passes its mapping addr&len and hugepage fd to kernel > module through created IOCTL. > 3. Kernel module remaps the hugepages inside kernel.
Such model should work, but the main "issue" is that it introduce overheads in the case of vhost-vDPA.
Note that in the case of vhost-vDPA, we don't use bounce buffer, the userspace pages were shared directly.
And since DMA is not done per page, it prevents us from using tricks like vm_insert_page() in those cases.
> 4. Vhost user target gets and maps hugepage fd from kernel module > in vhost-user msg through Unix Domain Socket cmsg. > Then kernel module and target map on the same hugepage based > bounce buffer for in-flight IO. > > If there is one option in VDUSE to map userspace preallocated memory, then > VDUSE should be able to mandate it even it is hugepage based. >
As above, this requires some kind of re-design since VDUSE depends on the model of mmap(MAP_SHARED) instead of umem registering.
Thanks
| |