lkml.org 
[lkml]   [2023]   [Jul]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
SubjectRe: [PATCH] fuse: enable larger read buffers for readdir [v2].
From


On 7/27/23 17:35, Miklos Szeredi wrote:
> On Thu, 27 Jul 2023 at 10:13, Jaco Kroon <jaco@uls.co.za> wrote:
>>
>> This patch does not mess with the caching infrastructure like the
>> previous one, which we believe caused excessive CPU and broke directory
>> listings in some cases.
>>
>> This version only affects the uncached read, which then during parse adds an
>> entry at a time to the cached structures by way of copying, and as such,
>> we believe this should be sufficient.
>>
>> We're still seeing cases where getdents64 takes ~10s (this was the case
>> in any case without this patch, the difference now that we get ~500
>> entries for that time rather than the 14-18 previously). We believe
>> that that latency is introduced on glusterfs side and is under separate
>> discussion with the glusterfs developers.
>>
>> This is still a compile-time option, but a working one compared to
>> previous patch. For now this works, but it's not recommended for merge
>> (as per email discussion).
>>
>> This still uses alloc_pages rather than kvmalloc/kvfree.
>>
>> Signed-off-by: Jaco Kroon <jaco@uls.co.za>
>> ---
>> fs/fuse/Kconfig | 16 ++++++++++++++++
>> fs/fuse/readdir.c | 18 ++++++++++++------
>> 2 files changed, 28 insertions(+), 6 deletions(-)
>>
>> diff --git a/fs/fuse/Kconfig b/fs/fuse/Kconfig
>> index 038ed0b9aaa5..0783f9ee5cd3 100644
>> --- a/fs/fuse/Kconfig
>> +++ b/fs/fuse/Kconfig
>> @@ -18,6 +18,22 @@ config FUSE_FS
>> If you want to develop a userspace FS, or if you want to use
>> a filesystem based on FUSE, answer Y or M.
>>
>> +config FUSE_READDIR_ORDER
>> + int
>> + range 0 5
>> + default 5
>> + help
>> + readdir performance varies greatly depending on the size of the read.
>> + Larger buffers results in larger reads, thus fewer reads and higher
>> + performance in return.
>> +
>> + You may want to reduce this value on seriously constrained memory
>> + systems where 128KiB (assuming 4KiB pages) cache pages is not ideal.
>> +
>> + This value reprents the order of the number of pages to allocate (ie,
>> + the shift value). A value of 0 is thus 1 page (4KiB) where 5 is 32
>> + pages (128KiB).
>> +
>> config CUSE
>> tristate "Character device in Userspace support"
>> depends on FUSE_FS
>> diff --git a/fs/fuse/readdir.c b/fs/fuse/readdir.c
>> index dc603479b30e..47cea4d91228 100644
>> --- a/fs/fuse/readdir.c
>> +++ b/fs/fuse/readdir.c
>> @@ -13,6 +13,12 @@
>> #include <linux/pagemap.h>
>> #include <linux/highmem.h>
>>
>> +#define READDIR_PAGES_ORDER CONFIG_FUSE_READDIR_ORDER
>> +#define READDIR_PAGES (1 << READDIR_PAGES_ORDER)
>> +#define READDIR_PAGES_SIZE (PAGE_SIZE << READDIR_PAGES_ORDER)
>> +#define READDIR_PAGES_MASK (READDIR_PAGES_SIZE - 1)
>> +#define READDIR_PAGES_SHIFT (PAGE_SHIFT + READDIR_PAGES_ORDER)
>> +
>> static bool fuse_use_readdirplus(struct inode *dir, struct dir_context *ctx)
>> {
>> struct fuse_conn *fc = get_fuse_conn(dir);
>> @@ -328,25 +334,25 @@ static int fuse_readdir_uncached(struct file *file, struct dir_context *ctx)
>> struct fuse_mount *fm = get_fuse_mount(inode);
>> struct fuse_io_args ia = {};
>> struct fuse_args_pages *ap = &ia.ap;
>> - struct fuse_page_desc desc = { .length = PAGE_SIZE };
>> + struct fuse_page_desc desc = { .length = READDIR_PAGES_SIZE };
>
> Does this really work? I would've thought we are relying on single
> page lengths somewhere.
>
>> u64 attr_version = 0;
>> bool locked;
>>
>> - page = alloc_page(GFP_KERNEL);
>> + page = alloc_pages(GFP_KERNEL, READDIR_PAGES_ORDER);
>> if (!page)
>> return -ENOMEM;
>>
>> plus = fuse_use_readdirplus(inode, ctx);
>> ap->args.out_pages = true;
>> - ap->num_pages = 1;
>> + ap->num_pages = READDIR_PAGES;
>
> No. This is the array lenght, which is 1. This is the hack I guess,
> which makes the above trick work.
>
> Better use kvmalloc, which might have a slightly worse performance
> than a large page, but definitely not worse than the current single
> page.
>
> If we want to optimize the overhead of kvmalloc (and it's a big if)
> then the parse_dir*file() functions would need to be converted to
> using a page array instead of a plain kernel pointer, which would add
> some complexity for sure.

One simple possibility might be to do pos=0 with a small buffer size
single page and only if pos is set we switch to a larger buffer - that
way small directories don't get the overhead of the large allocation.
Although following your idea to to the getdents buffer size - this is
something libc could already start with.


Cheers,
Bernd

\
 
 \ /
  Last update: 2023-07-27 21:17    [W:0.202 / U:0.056 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site