Messages in this thread | | | Date | Wed, 13 Nov 2013 05:59:11 +0200 | From | Denys Fedoryshchenko <> | Subject | Re: 3.12 BUG() on ext4, kernel crash on nbd-client when nbd server rebooting |
| |
Hi
On 2013-11-12 23:46, Jan Kara wrote: > Hello, > > On Tue 12-11-13 16:34:07, Denys Fedoryshchenko wrote: >> I just did some fault testing for test nbd setup, and found that if >> i reboot nbd server i will get immediately BUG() message on nbd >> client and filesystem that i cannot unmount, and any operations on >> it will freeze and lock processes trying to access it. > So how exactly did you do the fault testing? Because it seems > something > has discarded the block device under filesystem's toes and the > superblock > buffer_head got unmapped. Didn't something call NBD_CLEAR_SOCK ioctl? > Because that calls kill_bdev() which would do exactly that...
Client side: modprobe nbd nbd-client 2.2.2.29 /dev/nbd0 -name export1 nbd-client 2.2.2.29 /dev/nbd1 -name export2 nbd-client 2.2.2.29 /dev/nbd2 -name export3 mount /dev/nbd0 /mnt/disk1 mount /dev/nbd1 /mnt/disk2 mount /dev/nbd2 /mnt/disk3
On server i have config: [generic] [export1] exportname = /dev/sda1 [export2] exportname = /dev/sdb1 [export3] exportname = /dev/sdc1
Steps to reproduce: 1)Start some large file copy on client side to /mnt/disk1/ 2)Reboot server. It reboots quite fast, just few seconds, server system will get ip before nbd-server process started listening, so probably nbd-client will see connection refused. 3)seems when client gets connection refused - it is going mad
I can try to capture traffic dump, or do any other debug operation, please let me know, what i should run :) P.S. I noticed maybe i should run persist mode, but anyway it should not crash like this i think.
> > Honza >> Kernel 3.12, x86_64 >> >> Please let me know if you need more information >> >> Here is dmesg contents i got: >> [ 102.269270] block nbd1: Receive control failed (result -32) >> [ 102.269443] block nbd1: shutting down socket >> [ 102.269461] block nbd1: queue cleared >> [ 102.269859] block nbd2: Receive control failed (result -32) >> [ 102.269873] block nbd2: shutting down socket >> [ 102.269883] block nbd2: queue cleared >> [ 102.271353] block nbd0: Receive control failed (result -32) >> [ 102.271518] block nbd0: shutting down socket >> [ 102.271536] block nbd0: queue cleared >> [ 106.297217] block nbd0: Attempted send on closed socket >> [ 106.297219] end_request: I/O error, dev nbd0, sector 73992 >> [ 106.297226] EXT4-fs warning (device nbd0): >> __ext4_read_dirblock:908: error reading directory block (ino 2, >> block 0) >> [ 106.297233] block nbd0: Attempted send on closed socket >> [ 106.297235] end_request: I/O error, dev nbd0, sector 8456 >> [ 106.297245] ------------[ cut here ]------------ >> [ 106.297343] kernel BUG at fs/buffer.c:3015! >> [ 106.297438] invalid opcode: 0000 [#1] SMP >> [ 106.297716] Modules linked in: nbd act_mirred cls_u32 sch_ingress >> sch_htb iptable_filter i2c_i801 >> [ 106.298568] CPU: 0 PID: 2587 Comm: ls Not tainted 3.12.0noc-02 #1 >> [ 106.298665] Hardware name: /DH55TC, BIOS >> TCIBX10H.86A.0037.2010.0614.1712 06/14/2010 >> [ 106.298772] task: ffff880231da9770 ti: ffff880231cd4000 task.ti: >> ffff880231cd4000 >> [ 106.298879] RIP: 0010:[<ffffffff811253e0>] [<ffffffff811253e0>] >> _submit_bh+0x26/0x1d3 >> [ 106.299078] RSP: 0018:ffff880231cd5b48 EFLAGS: 00010246 >> [ 106.299182] RAX: 0000000000000005 RBX: ffff8800b7456b60 RCX: >> 0000000000000008 >> [ 106.299285] RDX: 0000000000000000 RSI: ffff8800b7456b60 RDI: >> 0000000000000411 >> [ 106.299388] RBP: ffff880231cd5b68 R08: 0000000000000040 R09: >> ffffffff81a9a370 >> [ 106.299487] R10: ffffffff810c0d61 R11: 0000000000000000 R12: >> 0000000000000411 >> [ 106.299590] R13: ffff880231b21400 R14: 0000000000000000 R15: >> 000000000aea9ff5 >> [ 106.299697] FS: 00007f4f0d755700(0000) GS:ffff88023fc00000(0000) >> knlGS:0000000000000000 >> [ 106.299800] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >> [ 106.300114] CR2: 00000000022275c8 CR3: 0000000235538000 CR4: >> 00000000000007f0 >> [ 106.300438] Stack: >> [ 106.300750] ffff8800b7456b60 0000000000000411 ffff880231b21400 >> 0000000000000001 >> [ 106.301652] ffff880231cd5b78 ffffffff81125598 ffff880231cd5ba8 >> ffffffff8112761a >> [ 106.307886] ffff880231cd5bb8 ffffffff81293a72 ffff8800b7456b60 >> ffff8802358d4800 >> [ 106.308794] Call Trace: >> [ 106.309105] [<ffffffff81125598>] submit_bh+0xb/0xd >> [ 106.309419] [<ffffffff8112761a>] __sync_dirty_buffer+0x53/0x86 >> [ 106.309736] [<ffffffff81293a72>] ? __percpu_counter_sum+0x4d/0x63 >> [ 106.310058] [<ffffffff8112765b>] sync_dirty_buffer+0xe/0x10 >> [ 106.310368] [<ffffffff81191327>] ext4_commit_super+0x19e/0x1e7 >> [ 106.310687] [<ffffffff8119230f>] save_error_info+0x1e/0x22 >> [ 106.311002] [<ffffffff811925a4>] __ext4_error_inode+0x52/0x10b >> [ 106.311326] [<ffffffff81060bbd>] ? __cond_resched+0x25/0x30 >> [ 106.311634] [<ffffffff8117b4ee>] __ext4_get_inode_loc+0x310/0x336 >> [ 106.311954] [<ffffffff8117cc0f>] ? ext4_dirty_inode+0x3b/0x54 >> [ 106.312277] [<ffffffff8117b60a>] ext4_get_inode_loc+0x17/0x19 >> [ 106.312596] [<ffffffff8117b62d>] >> ext4_reserve_inode_write+0x21/0x7e >> [ 106.312916] [<ffffffff811b3288>] ? jbd2__journal_start+0xe0/0x199 >> [ 106.313229] [<ffffffff8117ca57>] ext4_mark_inode_dirty+0x67/0x1e4 >> [ 106.313549] [<ffffffff8117cbf9>] ? ext4_dirty_inode+0x25/0x54 >> [ 106.313861] [<ffffffff8117cc0f>] ext4_dirty_inode+0x3b/0x54 >> [ 106.314177] [<ffffffff81121010>] __mark_inode_dirty+0x60/0x224 >> [ 106.314493] [<ffffffff811130be>] update_time+0x99/0xa2 >> [ 106.314807] [<ffffffff81113381>] touch_atime+0xf1/0x126 >> [ 106.315117] [<ffffffff8110db84>] iterate_dir+0x87/0xaa >> [ 106.315439] [<ffffffff8110dd01>] SyS_getdents+0x85/0xd0 >> [ 106.315757] [<ffffffff8110da28>] ? SyS_ioctl+0x80/0x80 >> [ 106.316081] [<ffffffff815a2f62>] system_call_fastpath+0x16/0x1b >> [ 106.316399] Code: 00 5f 5b c9 c3 55 48 89 e5 41 56 49 89 d6 41 55 >> 41 54 41 89 fc 53 48 89 f3 48 8b 06 a8 04 75 04 0f 0b eb fe 48 8b 06 >> a8 20 75 04 <0f> 0b eb fe 48 83 7e 38 00 75 04 0f 0b eb fe 48 8b 06 >> f6 c4 02 >> [ 106.323170] RIP [<ffffffff811253e0>] _submit_bh+0x26/0x1d3 >> [ 106.323579] RSP <ffff880231cd5b48> >> [ 106.323983] ---[ end trace 205f692f3e0cfed7 ]--- >> [ 111.834648] ------------[ cut here ]------------ >> [ 111.834975] kernel BUG at fs/buffer.c:3015! >> [ 111.835283] invalid opcode: 0000 [#2] SMP >> [ 111.835837] Modules linked in: nbd act_mirred cls_u32 sch_ingress >> sch_htb iptable_filter i2c_i801 >> [ 111.837121] CPU: 3 PID: 2578 Comm: jbd2/nbd0-8 Tainted: G D >> 3.12.0noc-02 #1 >> [ 111.837656] Hardware name: /DH55TC, BIOS >> TCIBX10H.86A.0037.2010.0614.1712 06/14/2010 >> [ 111.838193] task: ffff88023574f530 ti: ffff8800b727a000 task.ti: >> ffff8800b727a000 >> [ 111.838741] RIP: 0010:[<ffffffff811253e0>] [<ffffffff811253e0>] >> _submit_bh+0x26/0x1d3 >> [ 111.839372] RSP: 0018:ffff8800b727bbb8 EFLAGS: 00010246 >> [ 111.839688] RAX: 0000000004000005 RBX: ffff8800b7452750 RCX: >> 0000000000000411 >> [ 111.840005] RDX: 0000000000000000 RSI: ffff8800b7452750 RDI: >> 0000000000000411 >> [ 111.840325] RBP: ffff8800b727bbd8 R08: ffff8800b727a000 R09: >> ffffffff81a58ea0 >> [ 111.840634] R10: ffff88023fc6d500 R11: ffff8802360dfed8 R12: >> 0000000000000411 >> [ 111.840954] R13: 0000000000000411 R14: 0000000000000000 R15: >> ffff88023574f530 >> [ 111.841271] FS: 0000000000000000(0000) GS:ffff88023fc60000(0000) >> knlGS:0000000000000000 >> [ 111.841816] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >> [ 111.842130] CR2: 00007ff5bae1b0c0 CR3: 0000000001a0b000 CR4: >> 00000000000007e0 >> [ 111.842441] Stack: >> [ 111.842768] ffff8800b7452750 ffff8802358d5800 0000000000000411 >> ffff8800b4c1b000 >> [ 111.843684] ffff8800b727bbe8 ffffffff81125598 ffff8800b727bc28 >> ffffffff811bb44a >> [ 111.844594] ffff880237004a60 ffff8802358d5800 ffff8800b4c1b000 >> ffff8800b727be58 >> [ 111.845540] Call Trace: >> [ 111.845860] [<ffffffff81125598>] submit_bh+0xb/0xd >> [ 111.846171] [<ffffffff811bb44a>] jbd2_write_superblock+0xac/0xfa >> [ 111.846487] [<ffffffff811bbc0b>] >> jbd2_journal_update_sb_log_tail+0x31/0x64 >> [ 111.846811] [<ffffffff811b4a56>] >> jbd2_journal_commit_transaction+0xa2/0x14d0 >> [ 111.847129] [<ffffffff81064f92>] ? idle_balance+0xdc/0x109 >> [ 111.847453] [<ffffffff8159b5b7>] ? __schedule+0x55f/0x591 >> [ 111.847834] [<ffffffff81043162>] ? lock_timer_base+0x27/0x4d >> [ 111.848145] [<ffffffff811ba10e>] kjournald2+0xc8/0x23a >> [ 111.848469] [<ffffffff8105493e>] ? wake_up_bit+0x25/0x25 >> [ 111.848775] [<ffffffff811ba046>] ? commit_timeout+0xb/0xb >> [ 111.849155] [<ffffffff81054395>] kthread+0xb0/0xb8 >> [ 111.849473] [<ffffffff810542e5>] ? >> kthread_freezable_should_stop+0x60/0x60 >> [ 111.849780] [<ffffffff815a2ebc>] ret_from_fork+0x7c/0xb0 >> [ 111.850099] [<ffffffff810542e5>] ? >> kthread_freezable_should_stop+0x60/0x60 >> [ 111.850418] Code: 00 5f 5b c9 c3 55 48 89 e5 41 56 49 89 d6 41 55 >> 41 54 41 89 fc 53 48 89 f3 48 8b 06 a8 04 75 04 0f 0b eb fe 48 8b 06 >> a8 20 75 04 <0f> 0b eb fe 48 83 7e 38 00 75 04 0f 0b eb fe 48 8b 06 >> f6 c4 02 >> [ 111.857588] RIP [<ffffffff811253e0>] _submit_bh+0x26/0x1d3 >> [ 111.858064] RSP <ffff8800b727bbb8> >> [ 111.858448] ---[ end trace 205f692f3e0cfed8 ]--- >> >> -- >> To unsubscribe from this list: send the line "unsubscribe >> linux-fsdevel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html
| |