Messages in this thread | | | Date | Wed, 19 Apr 2006 14:49:50 -0700 (PDT) | From | Linus Torvalds <> | Subject | Re: Linux 2.6.17-rc2 |
| |
On Wed, 19 Apr 2006, Trond Myklebust wrote: > > Any chance this could be adapted to work with all those DMA (and RDMA) > engines that litter our motherboards? I'm thinking in particular of > stuff like the drm drivers, and userspace rdma.
Absolutely. Especially with "vmsplice()" (the not-yet-implemented "move these user pages into a kernel buffer") it should be entirely possible to set up an efficient zero-copy setup that does NOT have any of the problems with aio and TLB shootdown etc.
Note that a driver would have to support the splice_in() and splice_out() interfaces (which are basically just given the pipe buffers to do with as they wish), and perhaps more importantly: note that you need specialized apps that actually use splice() to do this.
That's the biggest downside by far, and is why I'm not 100% convinced splice() usage will be all that wide-spread. If you look at sendfile(), it's been available for a long time, and is actually even almost portable across different OS's _and_ it is easy to use. But almost nobody actually does. I suspect the only users are some apache mods, perhaps a ftp deamon or two, and probably samba. And that's probably largely it.
There's a _huge_ downside to specialized interfaces. Admittedly, splice() is a lot less specialized (ie it works in a much wider variety of loads), but it's still very much a "corner-case" thing. You can always do the same thing splice() does with a read/write pair instead, and be portable.
Also, the genericity of splice() does come at the cost of complexity. For example, to do a zero-copy from a user space buffer to some RDMA network interface, you'd have to basically keep track of _two_ buffers:
- keep track of how much of the user space buffer you have moved into kernel space with "vmsplice()" (or, for that matter, with any other source of data for the buffer - it might be a file, it might be another socket, whatever. I say "vmsplice()", but that's just an example for when you have the data in user space).
The kernel space buffer is - for obvious reasons - size limited in the way a user-space buffer is not. People are used to doing megabytes of buffers in user space. The splice buffer, in comparison, is maybe a few hundred kB at most. For some apps, that's "inifinity". For others, it's just a few tens of pages of data.
- keep track of how much of the kernel space buffer you have moved to the RDMA network interface with "splice()".
The splice buffer _is_ another buffer, and you have to feed the data from that buffer to the RDMA device manually.
In many usage schenarios, this means that you end up having the normal kind of poll/select loop. Now, that's nothing new: people are used to them, but people still hate them, and it just means that very few environments are going to spend the effort on another buffering setup.
So the upside of splice() is that it really can do some things very efficiently, by "copying" data with just a simple reference counted pointer. But the downside is that it makes for another level of buffering, and behind an interface that is in kernel space (for obvious reasons), which means that it's somewhat harder to wrap your hands and head around than just a regular user-space buffer.
So I'd expect this to be most useful for perhaps things like some HPC apps, where you can have specialized libraries for data communication. And servers, of course (but they might just continue to use the old "sendfile()" interface, without even knowing that it's not sendfile() any more, but just a wrapper around splice()).
Linus - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |