Messages in this thread | | | Date | Fri, 24 May 2013 02:57:41 +0200 | From | laurent.jourden@free ... | Subject | Kernel Oops on xfrm_output_resume |
| |
Hello,
I don't know how to handle this, not a lkml subscriber, just reporting this bug to the devs. All 3.9.y kernels crash after about 6 to 9 hours running without error messages before the Oops. I'm running 3.8.13 kernel so far without problem.
Even tried 3.10-rc2, same issue.
Context : two machines running with drbd block devices mirrored over ipsec. No Oops with kernels < 3.9.y
[29545.764401] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020 [29545.764419] IP: [<ffffffff815cb873>] xfrm_output_resume+0x293/0x380 [29545.764433] PGD 0 [29545.764439] Oops: 0000 [#1] SMP [29545.764446] Modules linked in: authenc esp4 xfrm4_mode_transport xt_sctp ip6t_REJECT nf_conntrack_ipv6 ip6table_raw ip6table_mangle ip6table_filter ip6_tables xt_TCPMSS act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq bridge stp llc xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm xt_addrtype iptable_raw xt_comment xt_recent xt_policy xt_nat ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_defrag_ipv6 nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NF QUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables deflate ctr twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic camellia_aesni_avx_x86_64 camellia_x86_64 cast5_avx_x86_64 cast5_generic cast_common des_generic xcbc rmd160 crypto_null af_key xfrm_algo ipv6 drbd lru_cache btrfs xor zlib_deflate raid6_pq libcrc32c crc32_pclmul crc32c_intel ghash_clmulni_intel [29545.765130] CPU 3 [29545.765135] Pid: 3245, comm: drbd_r_bdata Not tainted 3.9.3-051922 #14 /DH67BL [29545.765182] RIP: 0010:[<ffffffff815cb873>] [<ffffffff815cb873>] xfrm_output_resume+0x293/0x380 [29545.765220] RSP: 0018:ffff8803faa21af0 EFLAGS: 00010246 [29545.765240] RAX: 0000000000000000 RBX: ffff8804071258f8 RCX: 0000000000000770 [29545.765263] RDX: 000000000000076f RSI: ffff8803fe126000 RDI: ffff8804071258f8 [29545.765285] RBP: ffff8803faa21b18 R08: 000000000001a8f0 R09: ffffffffa0298313 [29545.765308] R10: ffffea000ff84980 R11: ffff88040e801300 R12: 0000000000000000 [29545.765330] R13: ffff88040728b400 R14: ffff88040728b43c R15: 0000000000000000 [29545.765353] FS: 0000000000000000(0000) GS:ffff88041f380000(0000) knlGS:0000000000000000 [29545.765388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29545.765409] CR2: 0000000000000020 CR3: 00000000017d2000 CR4: 00000000000407e0 [29545.765432] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29545.765454] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [29545.765477] Process drbd_r_bdata (pid: 3245, threadinfo ffff8803faa20000, task ffff880403791000) [29545.765513] Stack: [29545.765527] 0000000000000000 ffff8804071258f8 ffff8804071258f8 ffff8804077cc89c [29545.765563] ffff88040751ec00 ffff8803faa21b28 ffffffff815cb96e ffff8803faa21b50 [29545.765599] ffffffff815cb9a6 ffff8804071258f8 ffff8804071258f8 ffffffff815c0aa0 [29545.765635] Call Trace: [29545.765652] [<ffffffff815cb96e>] xfrm_output2+0xe/0x10 [29545.765673] [<ffffffff815cb9a6>] xfrm_output+0x36/0xe0 [29545.765694] [<ffffffff815c0aa0>] ? xfrm4_extract_output+0xd0/0xd0 [29545.765717] [<ffffffff815c0ac2>] xfrm4_output_finish+0x22/0x40 [29545.765738] [<ffffffff815c0b26>] xfrm4_output+0x46/0x80 [29545.765760] [<ffffffff81575100>] ip_local_out+0x20/0x30 [29545.765780] [<ffffffff8157526a>] ip_queue_xmit+0x15a/0x3d0 [29545.765803] [<ffffffff8158d636>] tcp_transmit_skb+0x406/0x920 [29545.765824] [<ffffffff8158dd89>] tcp_write_xmit+0x189/0xaa0 [29545.765847] [<ffffffff81534ae7>] ? __kmalloc_reserve.isra.53+0x37/0xa0 [29545.765870] [<ffffffff8158e6f9>] __tcp_push_pending_frames+0x29/0x90 [29545.765892] [<ffffffff8158f36a>] tcp_send_fin+0x6a/0x190 [29545.765913] [<ffffffff8157de2e>] tcp_shutdown+0x5e/0x70 [29545.765934] [<ffffffff815a7602>] inet_shutdown+0x92/0x130 [29545.765955] [<ffffffff81529f6b>] kernel_sock_shutdown+0xb/0x10 [29545.765983] [<ffffffffa015c89d>] drbd_free_sock+0x7d/0xb0 [drbd] [29545.766008] [<ffffffffa016f2b3>] conn_disconnect.part.48+0x49/0x444 [drbd] [29545.766035] [<ffffffffa016f0f3>] ? conn_request_state+0x53/0x80 [drbd] [29545.766061] [<ffffffffa0151705>] drbdd_init+0x225/0x230 [drbd] [29545.766087] [<ffffffffa0158ae8>] drbd_thread_setup+0x58/0x130 [drbd] [29545.766112] [<ffffffffa0158a90>] ? drbd_open+0xa0/0xa0 [drbd] [29545.766135] [<ffffffff810a2f5b>] kthread+0xbb/0xc0 [29545.766155] [<ffffffff810a2ea0>] ? kthread_create_on_node+0x120/0x120 [29545.766178] [<ffffffff81606f3c>] ret_from_fork+0x7c/0xb0 [29545.766199] [<ffffffff810a2ea0>] ? kthread_create_on_node+0x120/0x120 [29545.766221] Code: 00 48 85 ff 74 0f f0 ff 0f 0f 94 c0 84 c0 74 05 e8 43 f7 b7 ff 48 8b 43 58 48 89 df 48 c7 83 98 00 00 00 00 00 00 00 48 83 e0 fe <48> 8b 40 20 ff 50 60 83 f8 01 41 89 c4 0f 85 13 fe ff ff 48 8b [29545.766324] RIP [<ffffffff815cb873>] xfrm_output_resume+0x293/0x380 [29545.766346] RSP <ffff8803faa21af0> [29545.766363] CR2: 0000000000000020 [29545.766407] ---[ end trace 75eb563333dd96e1 ]--- [29545.991903] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020 [29545.991964] IP: [<ffffffff815cb873>] xfrm_output_resume+0x293/0x380 [29545.991993] PGD 0 [29545.992014] Oops: 0000 [#2] SMP [29545.992040] Modules linked in: authenc esp4 xfrm4_mode_transport xt_sctp ip6t_REJECT nf_conntrack_ipv6 ip6table_raw ip6table_mangle ip6table_filter ip6_tables xt_TCPMSS act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq bridge stp llc xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm xt_addrtype iptable_raw xt_comment xt_recent xt_policy xt_nat ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_defrag_ipv6 nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NF QUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables deflate ctr twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic camellia_aesni_avx_x86_64 camellia_x86_64 cast5_avx_x86_64 cast5_generic cast_common des_generic xcbc rmd160 crypto_null af_key xfrm_algo ipv6 drbd lru_cache btrfs xor zlib_deflate raid6_pq libcrc32c crc32_pclmul crc32c_intel ghash_clmulni_intel [29545.992974] CPU 0 [29545.992981] Pid: 3290, comm: drbd_a_build Tainted: G D 3.9.3-051922 #14 /DH67BL [29545.993035] RIP: 0010:[<ffffffff815cb873>] [<ffffffff815cb873>] xfrm_output_resume+0x293/0x380 [29545.993078] RSP: 0018:ffff8803faa0b810 EFLAGS: 00010246 [29545.993101] RAX: 0000000000000000 RBX: ffff8803faa4d8f8 RCX: 00000001801c0006 [29545.993126] RDX: 00000001801c0007 RSI: 00000000801c0006 RDI: ffff8803faa4d8f8 [29545.993152] RBP: ffff8803faa0b838 R08: 0000000000000000 R09: 0000000000000001 [29545.993177] R10: ffffea000ff61300 R11: ffff88040e801300 R12: 0000000000000000 [29545.993202] R13: ffff88040728b400 R14: ffff88040728b43c R15: 0000000000000000 [29545.993228] FS: 0000000000000000(0000) GS:ffff88041f200000(0000) knlGS:0000000000000000 [29545.993266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29545.993289] CR2: 0000000000000020 CR3: 00000000017d2000 CR4: 00000000000407f0 [29545.993315] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29545.993340] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [29545.993366] Process drbd_a_build (pid: 3290, threadinfo ffff8803faa0a000, task ffff8803fab02000) [29545.993404] Stack: [29545.993421] 0000000000000000 ffff8803faa4d8f8 ffff8803faa4d8f8 ffff88040410709c [29545.993470] ffff8804074a6800 ffff8803faa0b848 ffffffff815cb96e ffff8803faa0b870 [29545.993519] ffffffff815cb9a6 ffff8803faa4d8f8 ffff8803faa4d8f8 ffffffff815c0aa0 [29545.993569] Call Trace: [29545.993588] [<ffffffff815cb96e>] xfrm_output2+0xe/0x10 [29545.993611] [<ffffffff815cb9a6>] xfrm_output+0x36/0xe0 [29545.993635] [<ffffffff815c0aa0>] ? xfrm4_extract_output+0xd0/0xd0 [29545.993660] [<ffffffff815c0ac2>] xfrm4_output_finish+0x22/0x40 [29545.993684] [<ffffffff815c0b26>] xfrm4_output+0x46/0x80 [29545.993708] [<ffffffff81575100>] ip_local_out+0x20/0x30 [29545.993731] [<ffffffff8157526a>] ip_queue_xmit+0x15a/0x3d0 [29545.993755] [<ffffffff8158d636>] tcp_transmit_skb+0x406/0x920 [29545.993779] [<ffffffff8158dd89>] tcp_write_xmit+0x189/0xaa0 [29545.993804] [<ffffffff8158e6f9>] __tcp_push_pending_frames+0x29/0x90 [29545.993829] [<ffffffff8157f3f6>] tcp_sendmsg+0x116/0xdc0 [29545.993853] [<ffffffff8160358e>] ? schedule_timeout+0x12e/0x1b0 [29545.993877] [<ffffffff815a7dde>] inet_sendmsg+0x5e/0xa0 [29545.993901] [<ffffffff81530259>] ? release_sock+0x119/0x170 [29545.993925] [<ffffffff8152bc6b>] sock_sendmsg+0xab/0xe0 [29545.993948] [<ffffffff8158144e>] ? tcp_recvmsg+0x75e/0xcd0 [29545.993973] [<ffffffff8152c06b>] kernel_sendmsg+0x3b/0x50 [29545.994001] [<ffffffffa0159de5>] drbd_send+0x85/0x1b0 [drbd] [29545.994029] [<ffffffffa0159f1d>] drbd_send_all+0xd/0x30 [drbd] [29545.994057] [<ffffffffa0159fee>] __send_command.isra.32+0xae/0x170 [drbd] [29545.994086] [<ffffffffa015a0e6>] conn_send_command+0x36/0x50 [drbd] [29545.994115] [<ffffffffa015a33f>] drbd_send_ping+0x3f/0x60 [drbd] [29545.994143] [<ffffffffa0151ac8>] drbd_asender+0x3b8/0x6f0 [drbd] [29545.994168] [<ffffffff813c8729>] ? snprintf+0x39/0x40 [29545.994195] [<ffffffffa0158a90>] ? drbd_open+0xa0/0xa0 [drbd] [29545.994223] [<ffffffffa0158ae8>] drbd_thread_setup+0x58/0x130 [drbd] [29545.994251] [<ffffffffa0158a90>] ? drbd_open+0xa0/0xa0 [drbd] [29545.994275] [<ffffffff810a2f5b>] kthread+0xbb/0xc0 [29545.994298] [<ffffffff810a2ea0>] ? kthread_create_on_node+0x120/0x120 [29545.994324] [<ffffffff81606f3c>] ret_from_fork+0x7c/0xb0 [29545.994347] [<ffffffff810a2ea0>] ? kthread_create_on_node+0x120/0x120 [29545.994372] Code: 00 48 85 ff 74 0f f0 ff 0f 0f 94 c0 84 c0 74 05 e8 43 f7 b7 ff 48 8b 43 58 48 89 df 48 c7 83 98 00 00 00 00 00 00 00 48 83 e0 fe <48> 8b 40 20 ff 50 60 83 f8 01 41 89 c4 0f 85 13 fe ff ff 48 8b [29545.994653] RIP [<ffffffff815cb873>] xfrm_output_resume+0x293/0x380 [29545.994681] RSP <ffff8803faa0b810> [29545.994701] CR2: 0000000000000020 [29545.994721] ---[ end trace 75eb563333dd96e2 ]--- [29546.502319] BUG: unable to handle kernel NULL pointer dereference at 0000000000000808 [29546.502364] IP: [<ffffffff8105ed79>] __ticket_spin_lock+0x9/0x30 [29546.502393] PGD 0 [29546.502416] Oops: 0002 [#3] SMP [29546.502442] Modules linked in: authenc esp4 xfrm4_mode_transport xt_sctp ip6t_REJECT nf_conntrack_ipv6 ip6table_raw ip6table_mangle ip6table_filter ip6_tables xt_TCPMSS act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq bridge stp llc xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm xt_addrtype iptable_raw xt_comment xt_recent xt_policy xt_nat ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_defrag_ipv6 nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NF QUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables deflate ctr twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic camellia_aesni_avx_x86_64 camellia_x86_64 cast5_avx_x86_64 cast5_generic cast_common des_generic xcbc rmd160 crypto_null af_key xfrm_algo ipv6 drbd lru_cache btrfs xor zlib_deflate raid6_pq libcrc32c crc32_pclmul crc32c_intel ghash_clmulni_intel [29546.503389] CPU 3 [29546.503397] Pid: 3251, comm: drbd_r_build Tainted: G D 3.9.3-051922 #14 /DH67BL [29546.503450] RIP: 0010:[<ffffffff8105ed79>] [<ffffffff8105ed79>] __ticket_spin_lock+0x9/0x30 [29546.503493] RSP: 0018:ffff8803faa4fcf0 EFLAGS: 00010082 [29546.503516] RAX: 0000000000000100 RBX: 0000000000000082 RCX: 0000000000000001 [29546.503541] RDX: ffff8803fab02000 RSI: 0000000000000082 RDI: 0000000000000808 [29546.503566] RBP: ffff8803faa4fcf0 R08: 0000000000000000 R09: 0000000000000000 [29546.503591] R10: 000000000000055f R11: 0000000000000000 R12: 0000000000000002 [29546.503617] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001 [29546.503643] FS: 0000000000000000(0000) GS:ffff88041f380000(0000) knlGS:0000000000000000 [29546.503680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29546.503704] CR2: 0000000000000808 CR3: 00000000017d2000 CR4: 00000000000407e0 [29546.503729] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29546.503754] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [29546.503780] Process drbd_r_build (pid: 3251, threadinfo ffff8803faa4e000, task ffff88040784a000) [29546.503818] Stack: [29546.503836] ffff8803faa4fd00 ffffffff8105ee29 ffff8803faa4fd20 ffffffff81606529 [29546.503885] ffff8803fab02000 0000000000000002 ffff8803faa4fd58 ffffffff81095238 [29546.503935] ffff8803fe188b98 0000000000000002 0000000000000000 0000000000000001 [29546.503984] Call Trace: [29546.504004] [<ffffffff8105ee29>] default_spin_lock_flags+0x9/0x10 [29546.504029] [<ffffffff81606529>] _raw_spin_lock_irqsave+0x29/0x40 [29546.504055] [<ffffffff81095238>] force_sig_info+0x38/0x100 [29546.504079] [<ffffffff81095311>] force_sig+0x11/0x20 [29546.504106] [<ffffffffa01596b3>] _drbd_thread_stop+0x93/0x100 [drbd] [29546.504134] [<ffffffffa016f2aa>] conn_disconnect.part.48+0x40/0x444 [drbd] [29546.504164] [<ffffffffa016f0f3>] ? conn_request_state+0x53/0x80 [drbd] [29546.504193] [<ffffffffa0151705>] drbdd_init+0x225/0x230 [drbd] [29546.504221] [<ffffffffa0158a90>] ? drbd_open+0xa0/0xa0 [drbd] [29546.504249] [<ffffffffa0158ae8>] drbd_thread_setup+0x58/0x130 [drbd] [29546.504277] [<ffffffffa0158a90>] ? drbd_open+0xa0/0xa0 [drbd] [29546.504301] [<ffffffff810a2f5b>] kthread+0xbb/0xc0 [29546.504325] [<ffffffff810a2ea0>] ? kthread_create_on_node+0x120/0x120 [29546.504350] [<ffffffff81606f3c>] ret_from_fork+0x7c/0xb0 [29546.504374] [<ffffffff810a2ea0>] ? kthread_create_on_node+0x120/0x120 [29546.504398] Code: 00 00 48 c7 c1 91 eb 05 81 48 c7 c2 8e eb 05 81 e9 dd fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 55 b8 00 01 00 00 48 89 e5 <f0> 66 0f c1 07 0f b6 d4 38 c2 74 0c 0f 1f 00 f3 90 0f b6 07 38 [29546.504683] RIP [<ffffffff8105ed79>] __ticket_spin_lock+0x9/0x30 [29546.504711] RSP <ffff8803faa4fcf0> [29546.504730] CR2: 0000000000000808 [29546.504750] ---[ end trace 75eb563333dd96e3 ]---
Cheers -- Laurent
| |