Messages in this thread | | | Subject | Re: IIO, dmabuf, io_uring | From | Christian König <> | Date | Sun, 15 Aug 2021 20:02:02 +0200 |
| |
Hi Paul,
Am 13.08.21 um 13:41 schrieb Paul Cercueil: > Hi, > > A few months ago we (ADI) tried to upstream the interface we use with > our high-speed ADCs and DACs. It is a system with custom ioctls on the > iio device node to dequeue and enqueue buffers (allocated with > dma_alloc_coherent), that can then be mmap'd by userspace > applications. Anyway, it was ultimately denied entry [1]; this API was > okay in ~2014 when it was designed but it feels like re-inventing the > wheel in 2021. > > Back to the drawing table, and we'd like to design something that we > can actually upstream. This high-speed interface looks awfully similar > to DMABUF, so we may try to implement a DMABUF interface for IIO, > unless someone has a better idea.
Yeah, that sounds a lot like a DMABUF use case.
> > Our first usecase is, we want userspace applications to be able to > dequeue buffers of samples (from ADCs), and/or enqueue buffers of > samples (for DACs), and to be able to manipulate them (mmapped > buffers). With a DMABUF interface, I guess the userspace application > would dequeue a dma buffer from the driver, mmap it, read/write the > data, unmap it, then enqueue it to the IIO driver again so that it can > be disposed of. Does that sound sane?
Well it's pretty close. Doing the map/unmap dance all the time is usually a bad idea since flushing the CPU TLB all the time totally kills your performance.
What you do instead is to implement the CPU synchronize callbacks in your DMA-BUF implementation and flush caches as necessary.
> > Our second usecase is - and that's where things get tricky - to be > able to stream the samples to another computer for processing, over > Ethernet or USB. Our typical setup is a high-speed ADC/DAC on a dev > board with a FPGA and a weak soft-core or low-power CPU; processing > the data in-situ is not an option. Copying the data from one buffer to > another is not an option either (way too slow), so we absolutely want > zero-copy. > > Usual userspace zero-copy techniques (vmsplice+splice, MSG_ZEROCOPY > etc) don't really work with mmapped kernel buffers allocated for DMA > [2] and/or have a huge overhead, so the way I see it, we would also > need DMABUF support in both the Ethernet stack and USB (functionfs) > stack. However, as far as I understood, DMABUF is mostly a DRM/V4L2 > thing, so I am really not sure we have the right idea here.
Well two possibilities here: Either implement DMA-BUF support in the Ethernet/USB subsystem or get splice working efficiently with DMA-BUF mappings.
The first one is certainly a lot of work and no idea if the second is even doable and if yes also in a non-hacky way which you can get upstream.
> > And finally, there is the new kid in town, io_uring. I am not very > literate about the topic, but it does not seem to be able to handle > DMA buffers (yet?). The idea that we could dequeue a buffer of samples > from the IIO device and send it over the network in one single syscall > is appealing, though.
As far as I know this is orthogonal to DMA-BUF. Christoph's answer sounds like my understand is correct, but there are certainly people which know that better than I do.
Regards, Christian.
> > Any thoughts? Feedback would be greatly appreciated. > > Cheers, > -Paul > > [1]: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Flinux-iio%2F20210217073638.21681-1-alexandru.ardelean%40analog.com%2FT%2F%23m6b853addb77959c55e078fbb06828db33d4bf3d7&data=04%7C01%7Cchristian.koenig%40amd.com%7C2c62025e34b644b98e2508d95e4f4dcb%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637644516997743314%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vZfslxljjWcXi1RccZcsnKTD8x1CixRN%2Ftk4FMsWN3U%3D&reserved=0 > [2]: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnewbedev.com%2Fzero-copy-user-space-tcp-send-of-dma-mmap-coherent-mapped-memory&data=04%7C01%7Cchristian.koenig%40amd.com%7C2c62025e34b644b98e2508d95e4f4dcb%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637644516997753306%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Fn%2B3dO%2B%2F3r0ZpC5oKsQaPN2DREZKVWdVPahYgt2bsSw%3D&reserved=0 > >
| |