Messages in this thread | | | From | Xuewen Yan <> | Date | Tue, 12 Jul 2022 13:42:37 +0800 | Subject | Re: [PATCH] sched/schedutil: Fix deadlock between cpuset and cpu hotplug when using schedutil |
| |
On Tue, Jul 12, 2022 at 5:34 AM Steven Rostedt <rostedt@goodmis.org> wrote: > > On Mon, 11 Jul 2022 10:58:28 -1000 > Tejun Heo <tj@kernel.org> wrote: > > > I don't think lockdep would be able to track CPU1 -> CPU2 dependency here > > unfortunately. > > > > > AFAIU: > > > > > > > > > CPU0 CPU1 CPU2 > > > > > > // attach task to a different > > > // cpuset cgroup via sysfs > > > __acquire(cgroup_threadgroup_rwsem) > > > > > > // pring up CPU2 online > > > __acquire(cpu_hotplug_lock) > > > // wait for CPU2 to come online > > Should there be some annotation here that tells lockdep that CPU1 is now > blocked on CPU2? > > Then this case would be caught by lockdep. > > -- Steve > > > > > // bringup cpu online > > > // call cpufreq_online() which tries to create sugov kthread > > > __acquire(cpu_hotplug_lock) copy_process() > > > cgroup_can_fork() > > > cgroup_css_set_fork() > > > __acquire(cgroup_threadgroup_rwsem) > > > // blocks forever // blocks forever // blocks forever > > >
Indeed, It's caused by threads instead of cpus.
Our soc contains two cpufreq policy.0-5 belongs to policy0, 6-7 belongs to policy1.
when cpu6/7 online
Thread-A Thread-B cpuhp/6 kthreadd
cgroup_file_write device_online cgroup1_tasks_write ... __cgroup1_procs_write _cpu_up write(&cgroup_threadgroup_rwsem); << cpus_write_lock();<< cgroup_attach_task ...... cgroup_migrate_execute cpuhp_kick_ap ---------> cpuhp_thread_fun cpuset_attach //waiting for cpuhp cpuhp_invoke_callback cpus_read_lock() cpuhp_cpufreq_online
cpufreq_online //blocked sugov_init
__kthread_create_on_node kthreadd
//blocked, waiting for kthreadd copy_process
cgroup_can_fork
cgroup_css_set_fork
__acquires(&cgroup_threadgroup_rwsem)
So it's logic is: Thread-A ----->Thread-B------>cpuhp----->kthreadd----- ^ | |<---------------------------------------------------------------<-
When cpu offline, the sugov thread would stop, so it would waiting cgroup_threadgroup_rwsem when kthread_stop(); It's logic is: Thread-A ----->Thread-B------>cpuhp-------->sugov----- ^ | |<---------------------------------------------------------------<-
As Qais said: > if there's anything else that creates a kthread when a cpu goes online/offline > then we'll hit the same problem again.
Indeed, only the cpuhp thread create/destroy kthread would cause the case.
I have put the test script in the mail, and I have tested it without monkey test, the deadlock still occurs..
Thanks! xuewen.yan #!/system/bin/sh
interval=0.5 path="/sys/devices/system/cpu/cpu" n_cpu=`ls /sys/devices/system/cpu | grep "^cpu[0-9]" | wc -l` n_policy=`ls /sys/devices/system/cpu/cpufreq | grep "^policy[0-9]" | wc -l`
function random_hotplug(){ let n_cpu-=1 local ret=0 while [ 1 ] do let ret=$RANDOM%$n_cpu let ret+=1 local online=`cat $path$ret"/online"` [ "$online" = "0" ] && cm=1 || cm=0 online_cpus get_cpus=$? if [ "$cm" = "1" ];then echo $cm > $path$ret"/online" echo "online_cpu_count=$get_cpus n_policy=$n_policy $path$ret/online $cm $online `cat $path$ret/online` `cat /sys/devices/system/cpu/online`" sleep $interval else if [ "$get_cpus" -gt "2" ];then echo $cm > $path$ret"/online" echo "online_cpu_count=$get_cpus n_policy=$n_policy $path$ret/online $cm $online `cat $path$ret/online` `cat /sys/devices/system/cpu/online`" sleep $interval else echo "online_cpu_count=[ $get_cpus -lt 2 ], n_policy=$n_policy , next loop, $path$ret/online $cm $online `cat $path$ret/online` `cat /sys/devices/system/cpu/online`" fi fi
done }
function online_cpus(){ local cpu_count=1 for i in `seq 1 $n_cpu` do [ "`cat $path${i}/online`" -eq "1" ] && let cpu_count=cpu_count+1 done return $cpu_count }
random_hotplug
| |