Messages in this thread | | | Date | Sun, 27 Apr 2003 12:52:49 +0200 | From | Mike Galbraith <> | Subject | Houston, I think we have a problem |
| |
<SQUEAK! SQUEAK! SQUEAK!>
Hi Folks,
I don't generally squeak unless I'm pretty darn sure I see a genuine problem. I think I see one right now, so here I am squeaking my little lungs out ;-) Perhaps I'm being stupid, and if that's the case, someone please apply a size 15EE boot vigorously to my tail-feathers (jump-start brain), and I'll shut up.
The problem I see is terrible terrible semaphore starvation. It comes in two varieties, and might apply to other locks as well [1]. Variety 1 is owners of semaphores being sent off to the expired array, which happens with remarkable frequency. This variant is the lesser of the two evils, because here at least you have _some_ protection via EXPIRED_STARVING(), even if you have interactive tasks doing round robin. The worst variant is when you have a steady stream of tasks being upgraded to TASK_INTERACTIVE() while someone of low/modest priority has a semaphore downed... the poor guy can (seemingly) wait for _ages_ to get a chance to release it, and will starve all comers in the meantime. I regularly see a SCHED_RR and mlockall() vmstat stall for several seconds, and _sometimes_ my poor little box goes utterly insane and stalls vmstat for over a MINUTE [2].
To reproduce this 100% of the time, simply compile virgin 2.5.68 up/preempt, reduce your ram to 128mb, and using gcc-2.95.3 as to not overload the vm, run a make -j30 bzImage in an ext3 partition on a P3/500 single ide disk box. No, you don't really need to meet all of those restrictions... you'll see the problem on a big hairy chested box as well, just not as bad as I see it on my little box. The first symptom of the problem you will notice is a complete lack of swap activity along with highly improbable quantities of unused ram were all those hungry cc1's getting regular CPU feedings.
If the huge increase in hold time (induced by a stream of elevated priority tasks who may even achieve their elevated status via _one_ wakeup), is the desired behavior now, so be it. If that's the case, someone please say so, that I may cease and desist fighting with the dang thing. I'm having lots of fun mind you, but testing is supposed to be mind-numbingly boring ;-)
Anyway, grep for pid:prio pair 301:-2 in the attached log to see vmstat being nailed for over 8 seconds. Then, grep for pid:prio pair 1119:23 to see a task holding up a parade for 7 seconds. The patch I used to generate this log is also attached for idiot-reproachment purposes.
(um, don't anyone try running it on an SMP or NUMA beast [those folks would surely know better, but...] as it's highly likely to explode violently)
halbaderi,
-Mike
1. I'm pretty sure it does... might really be that Heisenberg fellow messing with me again.
2. The 100% simple and effective way to "fix" this problem for this work load is to "just say no" to coughing up more than HZ worth of cpu time in activate_task(). This seems perfectly obvious and correct to me... though I'll admit it would seem much _more_ perfectly obvious and correct if MAX_SLEEP_AVG were 11000 instead of 10000... or maybe even 40000. Whatever. I posted one X-patch that worked pretty darn well, but nobody tried it. Not even the folks who were _griping_ about interactivity, fairness and whatnot. How boring.
btw, what happens when kjournald yields and goes off to expired land? see log2.txt1131:18 starved 5 secs by 1119:23 who last ran 5152 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1132:18 starved 5 secs by 1119:23 who last ran 5178 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1093:18 starved 5 secs by 1119:23 who last ran 5204 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1067:18 starved 5 secs by 1119:23 who last ran 5230 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1142:22 starved 1 secs by 1133:23 who last ran 5613 ticks ago. semaphore downed at fs/namei.c:1249.
396:15 starved 2 secs by 1119:23 who last ran 5767 ticks ago. semaphore downed at fs/namei.c:1249.
1139:21 starved 2 secs by 1119:23 who last ran 5767 ticks ago. semaphore downed at fs/namei.c:1249.
301:-2 starved 3 secs by 1137:21 who last ran 3667 ticks ago. semaphore downed at arch/i386/kernel/sys_i386.c:58.
1079:16 starved 6 secs by 1119:23 who last ran 6155 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1131:17 starved 6 secs by 1119:23 who last ran 6181 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] <7>1132:17 starved 6 secs by 1119:23 who last ran 6205 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
[<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1093:17 starved 6 secs by 1119:23 who last ran 6233 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1067:17 starved 6 secs by 1119:23 who last ran 6259 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1142:21 starved 2 secs by 1133:23 who last ran 6708 ticks ago. semaphore downed at fs/namei.c:1249.
396:15 starved 3 secs by 1119:23 who last ran 6773 ticks ago. semaphore downed at fs/namei.c:1249.
1139:20 starved 3 secs by 1119:23 who last ran 6773 ticks ago. semaphore downed at fs/namei.c:1249.
301:-2 starved 4 secs by 1137:21 who last ran 4667 ticks ago. semaphore downed at arch/i386/kernel/sys_i386.c:58.
1079:15 starved 7 secs by 1119:23 who last ran 7181 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1131:16 starved 7 secs by 1119:23 who last ran 7228 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1132:16 starved 7 secs by 1119:23 who last ran 7254 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1093:16 starved 7 secs by 1119:23 who last ran 7281 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1067:16 starved 7 secs by 1119:23 who last ran 7307 ticks ago. sh R C5D95E10 92758188 1119 1118 (NOTLB) Call Trace: [<c011677a>] io_schedule+0xe/0x18 [<c0143bee>] __wait_on_buffer+0xa2/0xbc [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0116e34>] autoremove_wake_function+0x0/0x38 [<c0144c83>] __bread_slow+0x73/0x94 [<c0144ebe>] __bread+0x2a/0x34 [<c0177c31>] ext3_get_inode_loc+0xe1/0x140 [<c0178619>] ext3_reserve_inode_write+0x1d/0x94 [<c01786aa>] ext3_mark_inode_dirty+0x1a/0x34 [<c0174f74>] ext3_new_inode+0x714/0x7f0 [<c017ac95>] ext3_create+0x8d/0x180 [<c014f46a>] vfs_create+0xae/0xd4 [<c014f82d>] open_namei+0x1ad/0x4a8 [<c0142073>] filp_open+0x3b/0x5c [<c014249b>] sys_open+0x37/0x70 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at fs/namei.c:1249.
1142:20 starved 3 secs by 1133:23 who last ran 7716 ticks ago. semaphore downed at fs/namei.c:1249.
396:15 starved 4 secs by 1119:23 who last ran 28 ticks ago. semaphore downed at fs/namei.c:1249.
1139:19 starved 4 secs by 1119:23 who last ran 42 ticks ago. semaphore downed at fs/namei.c:1249.
301:-2 starved 5 secs by 1137:21 who last ran 5667 ticks ago. sh R 00000073 1546400 1137 396 1090 (NOTLB) Call Trace: [<c0108c09>] need_resched+0x27/0x32 [<c013007b>] __set_page_dirty_buffers+0xd3/0x114 [<c0138314>] do_mmap_pgoff+0x14c/0x608 [<c0139735>] mprotect_fixup+0x119/0x134 [<c010e011>] old_mmap+0x115/0x164 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at arch/i386/kernel/sys_i386.c:58.
301:-2 starved 6 secs by 1137:21 who last ran 6672 ticks ago. sh R 00000073 1546400 1137 396 1090 (NOTLB) Call Trace: [<c0108c09>] need_resched+0x27/0x32 [<c013007b>] __set_page_dirty_buffers+0xd3/0x114 [<c0138314>] do_mmap_pgoff+0x14c/0x608 [<c0139735>] mprotect_fixup+0x119/0x134 [<c010e011>] old_mmap+0x115/0x164 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at arch/i386/kernel/sys_i386.c:58.
301:-2 starved 7 secs by 1137:21 who last ran 7677 ticks ago. sh R 00000073 1546400 1137 396 1090 (NOTLB) Call Trace: [<c0108c09>] need_resched+0x27/0x32 [<c013007b>] __set_page_dirty_buffers+0xd3/0x114 [<c0138314>] do_mmap_pgoff+0x14c/0x608 [<c0139735>] mprotect_fixup+0x119/0x134 [<c010e011>] old_mmap+0x115/0x164 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at arch/i386/kernel/sys_i386.c:58.
301:-2 starved 8 secs by 1137:21 who last ran 8682 ticks ago. sh R 00000073 1546400 1137 396 1090 (NOTLB) Call Trace: [<c0108c09>] need_resched+0x27/0x32 [<c013007b>] __set_page_dirty_buffers+0xd3/0x114 [<c0138314>] do_mmap_pgoff+0x14c/0x608 [<c0139735>] mprotect_fixup+0x119/0x134 [<c010e011>] old_mmap+0x115/0x164 [<c0108cbf>] syscall_call+0x7/0xb
semaphore downed at arch/i386/kernel/sys_i386.c:58.
[unhandled content-type:application/octet-stream]29 yielded holding sem. kjournald R current 4182741476 29 1 123 28 (L-TLB) Call Trace: [<c01348e9>] shrink_cache+0x229/0x284 [<c011985a>] release_console_sem+0x4e/0xa0 [<c011977f>] printk+0x147/0x178 [<c011977f>] printk+0x147/0x178 [<c012bb9e>] __print_symbol+0x106/0x113 [<c012bb9e>] __print_symbol+0x106/0x113 [<c012bb9e>] __print_symbol+0x106/0x113 [<c012728b>] kernel_text_address+0x2f/0x3b [<c01092ac>] show_trace+0x3c/0x8c [<c010931a>] show_trace_task+0x1e/0x24 [<c0116f23>] show_task+0x1cb/0x1d4 [<c0116b45>] sys_sched_yield+0x95/0xe4 [<c0116bc7>] yield+0x17/0x1c [<c012fb08>] __alloc_pages+0x218/0x254 [<c012c810>] find_or_create_page+0x3c/0xac [<c0145699>] grow_dev_page+0x21/0x128 [<c0145831>] __getblk_slow+0x91/0xfc [<c0145be2>] __getblk+0x2e/0x38 [<c0187bb9>] journal_get_descriptor_buffer+0x35/0x54 [<c0184cb9>] journal_commit_transaction+0x675/0x1253 [<c0115ea2>] schedule+0x28e/0x338 [<c0187547>] kjournald+0xff/0x1e8 [<c0187448>] kjournald+0x0/0x1e8 [<c0187430>] commit_timeout+0x0/0x10 [<c0107111>] kernel_thread_helper+0x5/0xc
29 yielded holding sem. kjournald R current 4182741476 29 1 123 28 (L-TLB) Call Trace: [<c010931a>] show_trace_task+0x1e/0x24 [<c0116f23>] show_task+0x1cb/0x1d4 [<c0116b45>] sys_sched_yield+0x95/0xe4 [<c0116bc7>] yield+0x17/0x1c [<c012fb08>] __alloc_pages+0x218/0x254 [<c012c810>] find_or_create_page+0x3c/0xac [<c0145699>] grow_dev_page+0x21/0x128 [<c0145831>] __getblk_slow+0x91/0xfc [<c0145be2>] __getblk+0x2e/0x38 [<c0187bb9>] journal_get_descriptor_buffer+0x35/0x54 [<c0184cb9>] journal_commit_transaction+0x675/0x1253 [<c0115ea2>] schedule+0x28e/0x338 [<c0187547>] kjournald+0xff/0x1e8 [<c0187448>] kjournald+0x0/0x1e8 [<c0187430>] commit_timeout+0x0/0x10 [<c0107111>] kernel_thread_helper+0x5/0xc
| |