Messages in this thread | | | From | wzab@ise ... | Subject | Proposal of improvement for DMA - direct passing of hugepages to the SG list | Date | Wed, 14 Apr 2021 13:58:03 +0200 |
| |
Hi,
I'm working both on DMA engines implementations in FPGA and their Linux drivers. Now I need to create an engine that takes the hugepages-backed buffer allocated by the user-space application and passes it to the device. My current solution:
https://forums.xilinx.com/t5/Embedded-Linux/How-to-pass-efficiently-the-hugepages-backed-buffer-to-the-BM/m-p/1229340/highlight/true#M49777 or https://stackoverflow.com/a/67065962/1735409
uses the get_user_pages_fast function, to create the kernel mapping, and then uses sg_alloc_table_from_pages to build sg_table for it.
I have verified that the created sg_table has one entry for each individual hugepage (so I can efficiently map it for my SG-capable DMA device).
The disadvantage of that solution is that I need to create and keep a huge set of standard-size pages. Because the "struct page" occupies between 56 and 80 bytes, I get the overhead up to 80/4096 which is approx. 2%.
The open question is if I can free those pages as soon as the sg_table is created? (As far as I know, the hugepages are locked automatically). Of course it is unclear what happens if the application crashes and its mmaped hugepage-based buffer gets freed. Will the driver be notified about closing the file so that it can disable DMA before that memory can be taken for other purposes?
To be sure that it doesn't happen maybe it is good to keep those pages locked in the kernel as well. The big improvement would be if we could have the get_user_hugepages_fast function. The user should be allowed to create a smaller number of page structs. The function should check if the requested region really consists of hugepages and return an error if it doesn't.
Another important fact is that we don't need a kernel mapping for those pages at all. So it would be good to have yet another function: sg_alloc_table_from_user_pages which should take an additional "flag" argument enabling the user to decide if the area used consists of normal pages or of hugepages.
The function should return an error in case if the flag does not match the properties of the region. Of course the function should also lock the pages, and sg_free_table should unlock them (protecting against the danger of application crash, that I described above).
As a temporary workaround, is it possible to "manually" walk the pages creating the application-delivered buffer, verify that they are hugepages, lock them and create the sg_table? What functions/macros should be used for that to ensure that the implemented solution be portable and keeps working in a reasonable number of future versions of the Linux kernel?
I'll appreciate comments, suggestions and considering of the above proposal. With best regards, Wojtek
-- Wojciech M Zabolotny, PhD, DSc Institute of Electronic Systems Faculty of Electronics and Information Technology Warsaw University of Technology
| |