lkml.org 
[lkml]   [2020]   [Mar]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: [PATCH 00/11] fs/dcache: Limit # of negative dentries
From
Date
On 15/03/2020 06.46, Matthew Wilcox wrote:
> On Wed, Feb 26, 2020 at 11:13:53AM -0500, Waiman Long wrote:
>> As there is no limit for negative dentries, it is possible that a sizeable
>> portion of system memory can be tied up in dentry cache slabs. Dentry slabs
>> are generally recalimable if the dentries are in the LRUs. Still having
>> too much memory used up by dentries can be problematic:
>>
>> 1) When a filesystem with too many negative dentries is being unmounted,
>> the process of draining the dentries associated with the filesystem
>> can take some time. To users, the system may seem to hang for
>> a while. The long wait may also cause unexpected timeout error or
>> other warnings. This can happen when a long running container with
>> many negative dentries is being destroyed, for instance.
>>
>> 2) Tying up too much memory in unused negative dentries means there
>> are less memory available for other use. Even though the kernel is
>> able to reclaim unused dentries when running out of free memory,
>> it will still introduce additional latency to the application
>> reducing its performance.
>
> There's a third problem, which is that having a lot of negative dentries
> can clog the hash chains. I tried to quantify this, and found a weird result:

Yep. I've seen this in the wild. Server hard too much unused memory.

https://lore.kernel.org/lkml/ff0993a2-9825-304c-6a5b-2e9d4b940032@yandex-team.ru/T/#u

---quote---

I've seen problem on large server where horde of negative dentries
slowed down all lookups significantly:

watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [atop:968884] at __d_lookup_rcu+0x6f/0x190

slabtop:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
85118166 85116916 0% 0.19K 2026623 42 16212984K dentry
16577106 16371723 0% 0.10K 425054 39 1700216K buffer_head
935850 934379 0% 1.05K 31195 30 998240K ext4_inode_cache
663740 654967 0% 0.57K 23705 28 379280K radix_tree_node
399987 380055 0% 0.65K 8163 49 261216K proc_inode_cache
226380 168813 0% 0.19K 5390 42 43120K cred_jar
70345 65721 0% 0.58K 1279 55 40928K inode_cache
105927 43314 0% 0.31K 2077 51 33232K filp
630972 601503 0% 0.04K 6186 102 24744K ext4_extent_status
5848 4269 0% 3.56K 731 8 23392K task_struct
16224 11531 0% 1.00K 507 32 16224K kmalloc-1024
6752 5833 0% 2.00K 422 16 13504K kmalloc-2048
199680 158086 0% 0.06K 3120 64 12480K anon_vma_chain
156128 154751 0% 0.07K 2788 56 11152K Acpi-Operand

Total RAM is 256 GB

These dentries came from temporary files created and deleted by postgres.
But this could be easily reproduced by lookup of non-existent files.

Of course, memory pressure easily washes them away.

Similar problem happened before around proc sysctl entries:
https://lkml.org/lkml/2017/2/10/47

This one does not concentrate in one bucket and needs much more memory.

Looks like dcache needs some kind of background shrinker started
when dcache size or fraction of negative dentries exceeds some threshold.

---end---

> > root@bobo-kvm:~# time for i in `seq 1 10000`; do cat /dev/null >/dev/zero; done
> real 0m5.402s
> user 0m4.361s
> sys 0m1.230s
> root@bobo-kvm:~# time for i in `seq 1 10000`; do cat /dev/null >/dev/zero; done
> real 0m5.572s
> user 0m4.337s
> sys 0m1.407s
> root@bobo-kvm:~# time for i in `seq 1 10000`; do cat /dev/null >/dev/zero; done
> real 0m5.607s
> user 0m4.522s
> sys 0m1.342s
> root@bobo-kvm:~# time for i in `seq 1 10000`; do cat /dev/null >/dev/zero; done
> real 0m5.599s
> user 0m4.472s
> sys 0m1.369s
> root@bobo-kvm:~# time for i in `seq 1 10000`; do cat /dev/null >/dev/zero; done
> real 0m5.574s
> user 0m4.498s
> sys 0m1.300s
>
> Pretty consistent system time, between about 1.3 and 1.4 seconds.
>
> root@bobo-kvm:~# grep dentry /proc/slabinfo
> dentry 20394 21735 192 21 1 : tunables 0 0 0 : slabdata 1035 1035 0
> root@bobo-kvm:~# time for i in `seq 1 10000`; do cat /dev/null >/dev/zero; done
> real 0m5.515s
> user 0m4.353s
> sys 0m1.359s
>
> At this point, we have 20k dentries allocated.
>
> Now, pollute the dcache with names that don't exist:
>
> root@bobo-kvm:~# for i in `seq 1 100000`; do cat /dev/null$i >/dev/zero; done 2>/dev/null
> root@bobo-kvm:~# grep dentry /proc/slabinfo
> dentry 20605 21735 192 21 1 : tunables 0 0 0 : slabdata 1035 1035 0
>
> Huh. We've kept the number of dentries pretty constant. Still, maybe the
> bad dentries have pushed out the good ones.
>
> root@bobo-kvm:~# time for i in `seq 1 10000`; do cat /dev/null >/dev/zero; done
> real 0m6.644s
> user 0m4.921s
> sys 0m1.946s
> root@bobo-kvm:~# time for i in `seq 1 10000`; do cat /dev/null >/dev/zero; done
> real 0m6.676s
> user 0m5.004s
> sys 0m1.909s
> root@bobo-kvm:~# time for i in `seq 1 10000`; do cat /dev/null >/dev/zero; done
> real 0m6.662s
> user 0m4.980s
> sys 0m1.916s
> root@bobo-kvm:~# time for i in `seq 1 10000`; do cat /dev/null >/dev/zero; done
> real 0m6.714s
> user 0m4.973s
> sys 0m1.986s
>
> Well, we certainly made it suck. Up to a pretty consistent 1.9-2.0 seconds
> of kernel time, or 50% worse. We've also made user time worse, somehow.
>
> Anyhow, I should write a proper C program to measure this. But I thought
> I'd share this raw data with you now to demonstrate that dcache pollution
> is a real problem today, even on a machine with 2GB.
>

\
 
 \ /
  Last update: 2020-03-21 11:19    [W:0.187 / U:0.048 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site