lkml.org 
[lkml]   [2021]   [May]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Date
    SubjectRe: soft lockup in __inet_lookup_established() function
    Does anyone have some suggestions? I've been haunted for a while.

    thanks,
    jason

    On Thu, Apr 29, 2021 at 8:15 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
    >
    > Hello,
    >
    > I've encountered one big issue which causes infinite loop in
    > __inet_lookup_established() function until I reboot manually. it's
    > happening randomly among thousands of machines with the 4.19 kernel
    > running. Once the soft lockup issue is triggered, whatever I try I
    > still cannot ping or ssh to the machine anymore until reboot.
    >
    > Does anyone have any clue on how to dig into this part of code? I
    > highly suspect that it has something to do with the corruption of
    > nulls_list, so the lookup of sk could never break the infinite loop of
    > hashinfo.
    >
    > These call traces are totally identical attached below:
    > [1048271.465028] watchdog: BUG: soft lockup - CPU#20 stuck for 22s!
    > [swapper/20:0]
    > [1048271.473669] Modules linked in: vxlan ip6_udp_tunnel udp_tunnel
    > udp_diag tcp_diag inet_diag nf_conntrack_netlink nfnetlink
    > br_netfilter bridge stp llc xt_statistic xt_nat ipt_MASQUERADE
    > ipt_REJECT nf_reject_ipv4 xt_mark xt_addrtype xt_comment xt_conntrack
    > ...
    > [1048271.553597] RIP: 0010:__inet_lookup_established+0x5a/0x190
    > ...
    > [1048271.660309] Call Trace:
    > [1048271.663135] <IRQ>
    > [1048271.665432] tcp_v4_early_demux+0xaa/0x150
    > [1048271.669812] ip_rcv_finish+0x171/0x410
    > [1048271.673941] ip_rcv+0x273/0x362
    > [1048271.677360] ? inet_add_protocol.cold.1+0x1e/0x1e
    > [1048271.682354] __netif_receive_skb_core+0xac2/0xbb0
    > [1048271.687351] ? inet_gro_receive+0x22a/0x2d0
    > [1048271.692001] ? ktime_get_with_offset+0x4d/0xc0
    > [1048271.696725] netif_receive_skb_internal+0x42/0xf0
    > [1048271.701717] napi_gro_receive+0xba/0xe0
    > [1048271.705839] receive_buf+0x165/0xa50 [virtio_net]
    > [1048271.710839] ? receive_buf+0x165/0xa50 [virtio_net]
    > [1048271.716053] ? vring_unmap_one+0x16/0x80
    > [1048271.720308] ? detach_buf+0x69/0x110
    > [1048271.724218] virtnet_poll+0xc0/0x2ea [virtio_net]
    > [1048271.729202] net_rx_action+0x149/0x3b0
    > [1048271.733234] __do_softirq+0xe3/0x30a
    > [1048271.737095] irq_exit+0x100/0x110
    > [1048271.740882] do_IRQ+0x85/0xd0
    > [1048271.744143] common_interrupt+0xf/0xf
    > [1048271.748104] </IRQ>
    >
    >
    > thanks,
    > jason

    \
     
     \ /
      Last update: 2021-05-08 05:36    [W:4.321 / U:1.144 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site