lkml.org 
[lkml]   [2013]   [Jan]   [29]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectFollow up: multiple heavy-loaded KVM guests cause NULL pointer in CFS (kernel v 3.7.4)
Date
Hello,
A bit analysis in addition to my previous message.
The problem is that we receive NULL pointer dereference errors from inside of CFS (upstream kernel version 3.2-3.7) while running heavy loaded kvm-virtualized guests on 2-numa node server.

Null pointers are always met in pick_next_task_fair() method, with the following call traces:

1st case (confirmed in 3 tests):
http://xdel.ru/downloads/oops-default-kvmintel.txt (1 trace)
http://xdel.ru/downloads/oops-old-qemu.txt (2nd and 3rd traces)

2nd case (confirmed in 1 test):
http://imgur.com/QUmszYj
http://imgur.com/zhqLrCy
http://imgur.com/TZipg7F (4nd trace, sorry for images)


I'm not familiar with kernel internal arch, but seems that in 1st case
cfs_rq->nr_running != 0 && __pick_first_entity(struct cfs_rq *cfs_rq) == null
OR se->run_node is null.

and in 2nd case it seems that cfs_rq->tasks_timeline seems to be null.

I tried to mind map calls, here is a scheme http://imgur.com/bvEFX5h
Bug is exposed randomly while running cpu-consuming operations (like installation or simultaneous start of multiple virtual machines) on multi-numa node server with cpu cgroups enabled.
Tested and confirmed on 3.2, 3.4, 3.7 kernels.


I see 3 possible sources of described problem:
1. External code (qemu-kvm or cgroups) that breaks internal state of scheduler.
Have not idea whether it's possible or not for 3rd party kernel module (like kvm or cgroups) to break internal state of scheduler.

2. Scheduler
The bug is very rare and exposes only on heavy-loaded multi-numa server.
So it's virtually possible that bug exists and was not detected earlier just due to it's rarity.

3. Hardware
Very unlikely, as the bug is stably detected with same call trace and no other symptoms of hardware problem are exposed.

Thank you for help.
--
wbr, Igor Lukyanov

\
 
 \ /
  Last update: 2013-01-29 13:01    [W:0.029 / U:0.736 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site