lkml.org 
[lkml]   [2022]   [Jul]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: [PATCH] sched/schedutil: Fix deadlock between cpuset and cpu hotplug when using schedutil
On Tue, Jul 12, 2022 at 5:34 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Mon, 11 Jul 2022 10:58:28 -1000
> Tejun Heo <tj@kernel.org> wrote:
>
> > I don't think lockdep would be able to track CPU1 -> CPU2 dependency here
> > unfortunately.
> >
> > > AFAIU:
> > >
> > >
> > > CPU0 CPU1 CPU2
> > >
> > > // attach task to a different
> > > // cpuset cgroup via sysfs
> > > __acquire(cgroup_threadgroup_rwsem)
> > >
> > > // pring up CPU2 online
> > > __acquire(cpu_hotplug_lock)
> > > // wait for CPU2 to come online
>
> Should there be some annotation here that tells lockdep that CPU1 is now
> blocked on CPU2?
>
> Then this case would be caught by lockdep.
>
> -- Steve
>
>
> > > // bringup cpu online
> > > // call cpufreq_online() which tries to create sugov kthread
> > > __acquire(cpu_hotplug_lock) copy_process()
> > > cgroup_can_fork()
> > > cgroup_css_set_fork()
> > > __acquire(cgroup_threadgroup_rwsem)
> > > // blocks forever // blocks forever // blocks forever
> > >

Indeed, It's caused by threads instead of cpus.

Our soc contains two cpufreq policy.0-5 belongs to policy0, 6-7
belongs to policy1.

when cpu6/7 online

Thread-A Thread-B
cpuhp/6
kthreadd

cgroup_file_write device_online
cgroup1_tasks_write ...
__cgroup1_procs_write _cpu_up
write(&cgroup_threadgroup_rwsem); << cpus_write_lock();<<
cgroup_attach_task ......
cgroup_migrate_execute cpuhp_kick_ap
---------> cpuhp_thread_fun
cpuset_attach //waiting
for cpuhp cpuhp_invoke_callback
cpus_read_lock()
cpuhp_cpufreq_online

cpufreq_online
//blocked
sugov_init

__kthread_create_on_node
kthreadd

//blocked, waiting for kthreadd
copy_process


cgroup_can_fork


cgroup_css_set_fork


__acquires(&cgroup_threadgroup_rwsem)

So it's logic is:
Thread-A ----->Thread-B------>cpuhp----->kthreadd-----
^
|
|<---------------------------------------------------------------<-

When cpu offline, the sugov thread would stop, so it would waiting
cgroup_threadgroup_rwsem when kthread_stop();
It's logic is:
Thread-A ----->Thread-B------>cpuhp-------->sugov-----
^
|
|<---------------------------------------------------------------<-


As Qais said:
> if there's anything else that creates a kthread when a cpu goes online/offline
> then we'll hit the same problem again.

Indeed, only the cpuhp thread create/destroy kthread would cause the case.


I have put the test script in the mail, and I have tested it without
monkey test, the deadlock still occurs..


Thanks!
xuewen.yan
#!/system/bin/sh

interval=0.5
path="/sys/devices/system/cpu/cpu"
n_cpu=`ls /sys/devices/system/cpu | grep "^cpu[0-9]" | wc -l`
n_policy=`ls /sys/devices/system/cpu/cpufreq | grep "^policy[0-9]" | wc -l`

function random_hotplug(){
let n_cpu-=1
local ret=0
while [ 1 ]
do
let ret=$RANDOM%$n_cpu
let ret+=1
local online=`cat $path$ret"/online"`
[ "$online" = "0" ] && cm=1 || cm=0
online_cpus
get_cpus=$?
if [ "$cm" = "1" ];then
echo $cm > $path$ret"/online"
echo "online_cpu_count=$get_cpus n_policy=$n_policy $path$ret/online $cm $online `cat $path$ret/online` `cat /sys/devices/system/cpu/online`"
sleep $interval
else
if [ "$get_cpus" -gt "2" ];then
echo $cm > $path$ret"/online"
echo "online_cpu_count=$get_cpus n_policy=$n_policy $path$ret/online $cm $online `cat $path$ret/online` `cat /sys/devices/system/cpu/online`"
sleep $interval
else
echo "online_cpu_count=[ $get_cpus -lt 2 ], n_policy=$n_policy , next loop, $path$ret/online $cm $online `cat $path$ret/online` `cat /sys/devices/system/cpu/online`"
fi
fi

done
}

function online_cpus(){
local cpu_count=1
for i in `seq 1 $n_cpu`
do
[ "`cat $path${i}/online`" -eq "1" ] && let cpu_count=cpu_count+1
done
return $cpu_count
}


random_hotplug


\
 
 \ /
  Last update: 2022-07-12 07:43    [W:0.281 / U:0.152 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site