lkml.org 
[lkml]   [2012]   [Jun]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patches in this message
/
Date
From
SubjectRe: [bisected commit 0fc9d10] NFS-server corruption with 3.4
Ondrej Zary wrote:
> Hello,
> I use NFS for deploying HDD images on new machines. My machine has 2nd network
> card just for this, running DHCPD, TFTPD and kernel NFS server. The target
> machine is set to boot from LAN and boots SystemRescueCD from my machine with
> an autorun script that launches Partimage and deploys the HDD image (400 to
> 900 MB compressed).
>
> It worked fine for years, until now. With kernel 3.4, everyting
> works only for the first time after boot (and not always). Next time (next
> machine), partimage aborts almost immediately as it's probably unable to
> decompress the image file. md5sum is different on my machine vs. on the
> target (through NFS). Also SystemRescueCD boot aborts with md5 error
> sometimes. Everything works fine after rebooting back to 3.3.
>
> Bisection found this:
>
> 0fc9d1040313047edf6a39fd4d7c7defdca97c62 is the first bad commit
> commit 0fc9d1040313047edf6a39fd4d7c7defdca97c62
> Author: Konstantin Khlebnikov<khlebnikov@openvz.org>
> Date: Wed Mar 28 14:42:54 2012 -0700
>
> radix-tree: use iterators in find_get_pages* functions
>
> Reverting this commit in 3.4 fixes the problem.
>
>

[all reporters added to CC] let's keep all in one thread

In attachment two patches which might help to debug this regression:

"mm: recheck page index in find_get_pages_contig" adds paranoid check into find_get_pages_contig().
It can explain everything, but currently I don't see how this can hapens.

"mm: debug fing_get_pages speculative restart" shows lookup restarting condition
which was removed by bisected commit. It was checked by Hans, but unsuccessfully:

Hans de Bruin wrote:
> On 06/04/2012 12:31 PM, Konstantin Khlebnikov wrote:
>> Hans de Bruin wrote:
>>> On 06/01/2012 09:11 PM, Hans de Bruin wrote:
>>>> On 05/29/2012 12:19 AM, Hans de Bruin wrote:
>>>>> I just upgraded my home server from kernel 3.3.5 to 3.4.0 and ran into
>>>>> some trouble. My laptop, a nfsroot client, will not run firefox and
>>>>> thunderbird anymore. When I start these programs from an xterm, the
>>>>> cursor goes to the next line and waits indefinitely.
>>>>>
>>>>> I do not know if there is any order is lsof's output. A lsof | grep
>>>>> firefox or thunderbird shows ......./.parentlock as the last line.
>>>>>
>>>>> It does not matter whether the client is running a 3.4.0 or a 3.3.0
>>>>> kernel, or if the server is running on top of xen or not.
>>>>>
>>>>> There is some noise in the servers dmesg:
>>>>>
>>>>> [ 241.256684] INFO: task kworker/u:2:801 blocked for more than 120
>>>>> seconds.
>>>>> [ 241.256691] "echo 0> /proc/sys/kernel/hung_task_timeout_secs"
>>>>
>>>> ...
>>>>
>>>> On a almost identical testsystem firefox en thunderbird segfault after
>>>> upgrading to 3.4.0. I would have been nice if it would behave exaclty
>>>> like my home server. I bisected the segfault to:
>>>>
>>>> commit 0fc9d1040313047edf6a39fd4d7c7defdca97c62
>>>> Author: Konstantin Khlebnikov<khlebnikov@openvz.org>
>>>> Date: Wed Mar 28 14:42:54 2012 -0700
>>>>
>>>> radix-tree: use iterators in find_get_pages* functions
>>>>
>>>>
>>>> When I revert that on top of 3.4.0 the segfaults are gone but both
>>>> firefox en thunderbird go in the lets wait indefinitely mode like the
>>>> homeserver.
>>>>
>>>> I am going to make a bit-wise copy from from my homeserver to my
>>>> testserver and try again.
>>>>
>>>
>>> The bit-wise copy also segfaults firefox and thunderbird at the same
>>> commit.
>>>
>>
>> I think bug somewhere in NFS, that patch only highlighted it.
>> Please, try to run it with debug patch from attachment.
>
> Before I can start firefox from an xterm the lines below are shown on
> the server:
>
> [ 241.260076] INFO: task kworker/u:2:791 blocked for more than 120 seconds.
> [ 241.260084] "echo 0> /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 241.260090] kworker/u:2 D 000000000000000c 0 791 2
> 0x00000000
> [ 241.260102] ffff8801390b1cf0 0000000000000046 0000000000012d00
> 0000000000012d00
> [ 241.260113] 0000000000012d00 ffff880139141470 0000000000012d00
> ffff8801390b1fd8
> [ 241.260124] ffff8801390b1fd8 0000000000012d00 ffff880139cdc420
> ffff880139141470
> [ 241.260135] Call Trace:
> [ 241.260152] [<ffffffff81513116>] schedule+0x64/0x66
> [ 241.260162] [<ffffffff812005a6>] cld_pipe_upcall+0x95/0xd1
> [ 241.260173] [<ffffffff811faae5>] ? nfsd4_exchange_id+0x23e/0x23e
> [ 241.260182] [<ffffffff81200a5e>] nfsd4_cld_grace_done+0x50/0x8a
> [ 241.260191] [<ffffffff81200f8b>] nfsd4_record_grace_done+0x18/0x1a
> [ 241.260200] [<ffffffff811fab2f>] laundromat_main+0x4a/0x213
> [ 241.260210] [<ffffffff81069aeb>] ? need_resched+0x1e/0x28
> [ 241.260218] [<ffffffff81513035>] ? __schedule+0x49d/0x4b5
> [ 241.260227] [<ffffffff811faae5>] ? nfsd4_exchange_id+0x23e/0x23e
> [ 241.260237] [<ffffffff8105b8ad>] process_one_work+0x190/0x28d
> [ 241.260248] [<ffffffff8105c4e7>] worker_thread+0x105/0x189
> [ 241.260260] [<ffffffff81513a75>] ? _raw_spin_unlock_irqrestore+0x1a/0x1d
> [ 241.260274] [<ffffffff8105c3e2>] ? manage_workers.clone.17+0x173/0x173
> [ 241.260287] [<ffffffff8105ff30>] kthread+0x8a/0x92
> [ 241.260325] [<ffffffff815158a4>] kernel_thread_helper+0x4/0x10
> [ 241.260335] [<ffffffff8105fea6>] ?
> kthread_freezable_should_stop+0x47/0x47
> [ 241.260343] [<ffffffff815158a0>] ? gs_change+0x13/0x13
> [ 361.260025] INFO: task kworker/u:2:791 blocked for more than 120 seconds.
> [ 361.260032] "echo 0> /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 361.260039] kworker/u:2 D 000000000000000c 0 791 2
> 0x00000000
> [ 361.260051] ffff8801390b1cf0 0000000000000046 0000000000012d00
> 0000000000012d00
> [ 361.260062] 0000000000012d00 ffff880139141470 0000000000012d00
> ffff8801390b1fd8
> [ 361.260072] ffff8801390b1fd8 0000000000012d00 ffff880139cdc420
> ffff880139141470
> [ 361.260084] Call Trace:
> [ 361.260099] [<ffffffff81513116>] schedule+0x64/0x66
> [ 361.260110] [<ffffffff812005a6>] cld_pipe_upcall+0x95/0xd1
> [ 361.260121] [<ffffffff811faae5>] ? nfsd4_exchange_id+0x23e/0x23e
> [ 361.260130] [<ffffffff81200a5e>] nfsd4_cld_grace_done+0x50/0x8a
> [ 361.260139] [<ffffffff81200f8b>] nfsd4_record_grace_done+0x18/0x1a
> [ 361.260148] [<ffffffff811fab2f>] laundromat_main+0x4a/0x213
> [ 361.260158] [<ffffffff81069aeb>] ? need_resched+0x1e/0x28
> [ 361.260166] [<ffffffff81513035>] ? __schedule+0x49d/0x4b5
> [ 361.260175] [<ffffffff811faae5>] ? nfsd4_exchange_id+0x23e/0x23e
> [ 361.260185] [<ffffffff8105b8ad>] process_one_work+0x190/0x28d
> [ 361.260194] [<ffffffff8105c4e7>] worker_thread+0x105/0x189
> [ 361.260203] [<ffffffff81513a75>] ? _raw_spin_unlock_irqrestore+0x1a/0x1d
> [ 361.260213] [<ffffffff8105c3e2>] ? manage_workers.clone.17+0x173/0x173
> [ 361.260222] [<ffffffff8105ff30>] kthread+0x8a/0x92
> [ 361.260231] [<ffffffff815158a4>] kernel_thread_helper+0x4/0x10
> [ 361.260241] [<ffffffff8105fea6>] ?
> kthread_freezable_should_stop+0x47/0x47
> [ 361.260249] [<ffffffff815158a0>] ? gs_change+0x13/0x13
> [ 481.260010] INFO: task kworker/u:2:791 blocked for more than 120 seconds.
> [ 481.260019] "echo 0> /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 481.260028] kworker/u:2 D 000000000000000c 0 791 2
> 0x00000000
> [ 481.260043] ffff8801390b1cf0 0000000000000046 0000000000012d00
> 0000000000012d00
> [ 481.260058] 0000000000012d00 ffff880139141470 0000000000012d00
> ffff8801390b1fd8
> [ 481.260073] ffff8801390b1fd8 0000000000012d00 ffff880139cdc420
> ffff880139141470
> [ 481.260088] Call Trace:
> [ 481.260107] [<ffffffff81513116>] schedule+0x64/0x66
> [ 481.260120] [<ffffffff812005a6>] cld_pipe_upcall+0x95/0xd1
> [ 481.260135] [<ffffffff811faae5>] ? nfsd4_exchange_id+0x23e/0x23e
> [ 481.260147] [<ffffffff81200a5e>] nfsd4_cld_grace_done+0x50/0x8a
> [ 481.260159] [<ffffffff81200f8b>] nfsd4_record_grace_done+0x18/0x1a
> [ 481.260172] [<ffffffff811fab2f>] laundromat_main+0x4a/0x213
> [ 481.260185] [<ffffffff81069aeb>] ? need_resched+0x1e/0x28
> [ 481.260196] [<ffffffff81513035>] ? __schedule+0x49d/0x4b5
> [ 481.260206] [<ffffffff811faae5>] ? nfsd4_exchange_id+0x23e/0x23e
> [ 481.260215] [<ffffffff8105b8ad>] process_one_work+0x190/0x28d
> [ 481.260225] [<ffffffff8105c4e7>] worker_thread+0x105/0x189
> [ 481.260234] [<ffffffff81513a75>] ? _raw_spin_unlock_irqrestore+0x1a/0x1d
> [ 481.260243] [<ffffffff8105c3e2>] ? manage_workers.clone.17+0x173/0x173
> [ 481.260252] [<ffffffff8105ff30>] kthread+0x8a/0x92
> [ 481.260262] [<ffffffff815158a4>] kernel_thread_helper+0x4/0x10
> [ 481.260271] [<ffffffff8105fea6>] ?
> kthread_freezable_should_stop+0x47/0x47
> [ 481.260279] [<ffffffff815158a0>] ? gs_change+0x13/0x13
>
>
> dmesg on the client side:
>
> [ 27.607606] gtk-query-immod[1976]: segfault at 1d2d1f30 ip b7734391
> sp bfe3e984 error 4 in ld-2.13.so[b772b000+1d000]
> [ 48.136763] start_kdeinit (2086): /proc/2086/oom_adj is deprecated,
> please use /proc/2086/oom_score_adj instead.
> [ 75.801804] blueman-applet[2150]: segfault at 1cf2cf30 ip b7741391 sp
> bfb456b8 error 4 in ld-2.13.so[b7738000+1d000]
> [ 140.226371] firefox[2175]: segfault at 1b065f30 ip b76f6391 sp
> bfb15db8 error 4 in ld-2.13.so[b76ed000+1d000]
>
>
> The firefox dump on client side produces no messages on server side.
>
> md5sum's of ld-2.13.so are equal on server and client and across
> kernlversions.
>
>
>
> Did I miss the output off the debug patch?
>
>
mm: debug fing_get_pages speculative restart

From: Konstantin Khlebnikov <khlebnikov@openvz.org>

Signed-off-by: Konstantin Khlebnikov <khlebnikov@openvz.org>
---
mm/filemap.c | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/mm/filemap.c b/mm/filemap.c
index b3a698e..5e3b3c2 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -816,6 +816,7 @@ unsigned find_get_pages(struct address_space *mapping, pgoff_t start,
struct radix_tree_iter iter;
void **slot;
unsigned ret = 0;
+ int nr_found = 0;

if (unlikely(!nr_pages))
return 0;
@@ -847,6 +848,7 @@ repeat:
continue;
}

+ nr_found++;
if (!page_cache_get_speculative(page))
goto repeat;

@@ -862,6 +864,7 @@ repeat:
}

rcu_read_unlock();
+ WARN_ON(!ret && nr_found);
return ret;
}

@@ -883,6 +886,7 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index,
struct radix_tree_iter iter;
void **slot;
unsigned int ret = 0;
+ int nr_found = 0;

if (unlikely(!nr_pages))
return 0;
@@ -914,6 +918,7 @@ repeat:
break;
}

+ nr_found++;
if (!page_cache_get_speculative(page))
goto repeat;

@@ -939,6 +944,7 @@ repeat:
break;
}
rcu_read_unlock();
+ WARN_ON(!ret && nr_found);
return ret;
}
EXPORT_SYMBOL(find_get_pages_contig);
@@ -960,6 +966,7 @@ unsigned find_get_pages_tag(struct address_space *mapping, pgoff_t *index,
struct radix_tree_iter iter;
void **slot;
unsigned ret = 0;
+ int nr_found = 0;

if (unlikely(!nr_pages))
return 0;
@@ -990,6 +997,7 @@ repeat:
BUG();
}

+ nr_found++;
if (!page_cache_get_speculative(page))
goto repeat;

@@ -1009,6 +1017,7 @@ repeat:
if (ret)
*index = pages[ret - 1]->index + 1;

+ WARN_ON(!ret && nr_found);
return ret;
}
EXPORT_SYMBOL(find_get_pages_tag);mm: recheck page index in find_get_pages_contig

From: Konstantin Khlebnikov <khlebnikov@openvz.org>

Signed-off-by: Konstantin Khlebnikov <khlebnikov@openvz.org>
---
mm/filemap.c | 1 +
1 file changed, 1 insertion(+)

diff --git a/mm/filemap.c b/mm/filemap.c
index 79c4b2b..b3a698e 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -933,6 +933,7 @@ repeat:
break;
}

+ WARN_ON(page->index != index + ret);
pages[ret] = page;
if (++ret == nr_pages)
break;
\
 
 \ /
  Last update: 2012-06-05 16:21    [W:0.073 / U:0.160 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site