lkml.org 
[lkml]   [2022]   [Oct]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: sched/debug: CPU hotplug operation suffers in a large cpu systems
On Mon, Oct 17, 2022 at 04:54:11PM +0200, Greg Kroah-Hartman wrote:
> On Mon, Oct 17, 2022 at 04:19:31PM +0200, Peter Zijlstra wrote:
> >
> > +GregKH who actually knows about debugfs.
> >
> > On Mon, Oct 17, 2022 at 06:40:49PM +0530, Vishal Chourasia wrote:
> > > smt=off operation on system with 1920 CPUs is taking approx 59 mins on v5.14
> > > versus 29 mins on v5.11 measured using:
> > > # time ppc64_cpu --smt=off
> > >
> > >
> > > |--------------------------------+----------------+--------------|
> > > | method | sysctl | debugfs |
> > > |--------------------------------+----------------+--------------|
> > > | unregister_sysctl_table | 0.020050 s | NA |
> > > | build_sched_domains | 3.090563 s | 3.119130 s |
> > > | register_sched_domain_sysctl | 0.065487 s | NA |
> > > | update_sched_domain_debugfs | NA | 2.791232 s |
> > > | partition_sched_domains_locked | 3.195958 s | 5.933254 s |
> > > |--------------------------------+----------------+--------------|
> > >
> > > Note: partition_sched_domains_locked internally calls build_sched_domains
> > > and calls other functions respective to what's being currently used to
> > > export information i.e. sysctl or debugfs
> > >
> > > Above numbers are quoted from the case where we tried offlining 1 cpu in system
> > > with 1920 online cpus.
> > >
> > > From the above table, register_sched_domain_sysctl and
> > > unregister_sysctl_table collectively took ~0.085 secs, whereas
> > > update_sched_domain_debugfs took ~2.79 secs.
> > >
> > > Root cause:
> > >
> > > The observed regression stems from the way these two pseudo-filesystems handle
> > > creation and deletion of files and directories internally.
>
> Yes, debugfs is not optimized for speed or memory usage at all. This
> happens to be the first code path I have seen that cares about this for
> debugfs files.
>
> You can either work on not creating so many debugfs files (do you really
> really need all of them all the time?) Or you can work on moving
> debugfs to use kernfs as the backend logic, which will save you both
> speed and memory usage overall as kernfs is used to being used on
> semi-fast paths.
>
> Maybe do both?
>
> hope this helps,
>
> greg k-h

Yes, we need to create 7-8 files per domain per CPU, eventually ending up
creating a lot of files.

Peter,

Is there a possibility of reverting back to /proc/sys/kernel/sched_domain/?

-- vishal.c[unhandled content-type:application/pgp-signature]
\
 
 \ /
  Last update: 2022-10-18 12:38    [W:0.080 / U:24.368 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site