Messages in this thread | | | From | DaeRyong Jeong <> | Date | Thu, 19 Apr 2018 15:45:15 +0900 | Subject | Re: kernel BUG at /home/blee/project/race-fuzzer/kernels/kernel_v4.16-rc3/net/packet/af_packet.c:LINE! |
| |
Hello. We have analyzed the cause of the crash, kernel BUG at net/packet/af_packet.c:LINE!, which is found by RaceFuzzer (a modified version of Syzkaller) in v4.16-rc7.
Since struct packet_sock's member variables, running, has_vnet_hdr, origdev and auxdata are declared as bitfields, accessing these variables can race if there is no synchronization mechanism.
We think racing between following lines in af_packet.c causes the crash. In function __unregister_prot_hook, po->running = 0; In function packet_setsockopt, po->has_vnet_hdr = !!val;
Analysis: CPU0 pakcet_setsockopt po->has_vnet_hdr = !!val;
CPU1 packet_do_bind __unregister_prot_hook po->running = 0;
In the CPU1, the value of po->running should become 0, but because of racing, it is possible that po->running can keep the value 1. Consequently, after returning from __unregister_prot_hook, BUG_ON at net/packet/af_packet.c:3107 can be triggered.
Possible interleaving between racy C source lines is as follows (built with gcc-7.1.0). CPU0 (po->has_vnet_hdr = !!val) CPU1 (po->running = 0) movzbl 0x6e0(%r15),%eax andb $0xfe,0x6e0(%r13) shl $0x3,%r12d and $0xfffffff7,%eax or %r12d,%eax mov %al,0x6e0(%r15)
Please, check out the following reproducer. C repro code : https://kiwi.cs.purdue.edu/static/race-fuzzer/afpacket-setsockopt-bind-repro.c kernel config v4.16-rc3 : https://kiwi.cs.purdue.edu/static/race-fuzzer/afpacket-setsockopt-bind-v4.16-rc3.config kernel config v4.16-rc7 : https://kiwi.cs.purdue.edu/static/race-fuzzer/afpacket-setsockopt-bind-v4.16-rc7.config kernel config v4.15.14 : https://kiwi.cs.purdue.edu/static/race-fuzzer/afpacket-setsockopt-bind-v4.15.14.config
= About RaceFuzzer
RaceFuzzer is a customized version of Syzkaller, specifically tailored to find race condition bugs in the Linux kernel. While we leverage many different technique, the notable feature of RaceFuzzer is in leveraging a custom hypervisor (QEMU/KVM) to interleave the scheduling. In particular, we modified the hypervisor to intentionally stall a per-core execution, which is similar to supporting per-core breakpoint functionality. This allows RaceFuzzer to force the kernel to deterministically trigger racy condition (which may rarely happen in practice due to randomness in scheduling).
RaceFuzzer's C repro always pinpoints two racy syscalls. Since C repro's scheduling synchronization should be performed at the user space, its reproducibility is limited (reproduction may take from 1 second to 10 minutes (or even more), depending on a bug). This is because, while RaceFuzzer precisely interleaves the scheduling at the kernel's instruction level when finding this bug, C repro cannot fully utilize such a feature. Please disregard all code related to "should_hypercall" in the C repro, as this is only for our debugging purposes using our own hypervisor.
On Sat, Mar 31, 2018 at 1:33 AM, DaeRyong Jeong <threeearcat@gmail.com> wrote: > We report the crash: kernel BUG at > /home/blee/project/race-fuzzer/kernels/kernel_v4.16-rc3/net/packet/af_packet.c:LINE! > > This crash has been found in v4.16-rc3 using RaceFuzzer (a modified > version of Syzkaller), which we describe more at the end of this > report. Our analysis shows that the race occurs when invoking two > syscalls concurrently, (setsockopt$packet_int) and (bind$packet). > We have confirmed that the kernel v4.16-rc3, v4.16-rc7, and v4.15.14 > built with gcc 7.1.0 are crashing by running the provided C repro > program within a few minutes (5 minutes). > Note that this crash can be triggered from the user space. > > C repro code : https://kiwi.cs.purdue.edu/static/race-fuzzer/afpacket-setsockopt-bind-repro.c > kernel config v4.16-rc3 : > https://kiwi.cs.purdue.edu/static/race-fuzzer/afpacket-setsockopt-bind-v4.16-rc3.config > kernel config v4.16-rc7 : > https://kiwi.cs.purdue.edu/static/race-fuzzer/afpacket-setsockopt-bind-v4.16-rc7.config > kernel config v4.15.14 : > https://kiwi.cs.purdue.edu/static/race-fuzzer/afpacket-setsockopt-bind-v4.15.14.config > > [ 881.047513] ------------[ cut here ]------------ > [ 881.048416] kernel BUG at > /home/blee/project/race-fuzzer/kernels/kernel_v4.16-rc3/net/packet/af_packet.c:3107! > [ 881.050014] invalid opcode: 0000 [#1] SMP KASAN > [ 881.050698] Dumping ftrace buffer: > [ 881.051244] (ftrace buffer empty) > [ 881.051768] Modules linked in: > [ 881.052236] CPU: 1 PID: 18247 Comm: syz-executor0 Not tainted 4.16.0-rc3 #1 > [ 881.053247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 > [ 881.054880] RIP: 0010:packet_do_bind+0x88d/0x950 > [ 881.055553] RSP: 0018:ffff8802231d7b08 EFLAGS: 00010212 > [ 881.056310] RAX: 0000000000010000 RBX: ffff8800af831740 RCX: ffffc900025ce000 > [ 881.057318] RDX: 00000000000000a5 RSI: ffffffff838b257d RDI: 0000000000000001 > [ 881.058301] RBP: ffff8802231d7c10 R08: ffff8802342f2480 R09: 0000000000000000 > [ 881.059298] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8802309f8f00 > [ 881.060314] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000001000 > [ 881.061320] FS: 00007f7fab50d700(0000) GS:ffff88023fc00000(0000) > knlGS:0000000000000000 > [ 881.062467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 881.063285] CR2: 0000000020038000 CR3: 00000000b11c9000 CR4: 00000000000006e0 > [ 881.064317] Call Trace: > [ 881.064686] ? compat_packet_setsockopt+0x100/0x100 > [ 881.065430] ? __sanitizer_cov_trace_const_cmp8+0x18/0x20 > [ 881.066188] packet_bind+0xa2/0xe0 > [ 881.066690] SYSC_bind+0x279/0x2f0 > [ 881.067180] ? move_addr_to_kernel.part.19+0xc0/0xc0 > [ 881.067896] ? do_futex+0x1e90/0x1e90 > [ 881.068435] ? SyS_sched_getaffinity+0xe3/0x100 > [ 881.069112] ? mark_held_locks+0x25/0xb0 > [ 881.069677] ? SyS_socketpair+0x4a0/0x4a0 > [ 881.070265] SyS_bind+0x24/0x30 > [ 881.070732] do_syscall_64+0x209/0x5d0 > [ 881.071270] ? syscall_return_slowpath+0x3e0/0x3e0 > [ 881.071929] ? __sanitizer_cov_trace_const_cmp4+0x16/0x20 > [ 881.072675] ? syscall_return_slowpath+0x260/0x3e0 > [ 881.073365] ? mark_held_locks+0x25/0xb0 > [ 881.073950] ? entry_SYSCALL_64_after_hwframe+0x52/0xb7 > [ 881.074693] ? trace_hardirqs_off_caller+0xb5/0x120 > [ 881.075390] ? trace_hardirqs_off_thunk+0x1a/0x1c > [ 881.076079] entry_SYSCALL_64_after_hwframe+0x42/0xb7 > [ 881.076797] RIP: 0033:0x453909 > [ 881.077238] RSP: 002b:00007f7fab50caf8 EFLAGS: 00000212 ORIG_RAX: > 0000000000000031 > [ 881.078268] RAX: ffffffffffffffda RBX: 00000000007080d8 RCX: 0000000000453909 > [ 881.079239] RDX: 0000000000000014 RSI: 000000002001f000 RDI: 0000000000000015 > [ 881.080268] RBP: 0000000000000250 R08: 0000000000000000 R09: 0000000000000000 > [ 881.081256] R10: 0000000000000000 R11: 0000000000000212 R12: 00000000004a82d3 > [ 881.082272] R13: 00000000ffffffff R14: 0000000000000015 R15: 000000002001f000 > [ 881.083251] Code: c0 fd 48 c7 c2 00 c8 d9 84 be ab 02 00 00 48 c7 > c7 60 c8 d9 84 c6 05 e7 a2 48 02 01 e8 3f 17 af fd e9 60 fb ff ff e8 > 43 b3 c0 fd <0f> 0b e8 3c b3 c0 fd 48 8b bd 20 ff ff ff e8 60 1e e7 fd > 4c 89 > [ 881.085828] RIP: packet_do_bind+0x88d/0x950 RSP: ffff8802231d7b08 > [ 881.086619] ---[ end trace 9c461502752b4f3e ]--- > [ 881.087181] Kernel panic - not syncing: Fatal exception > [ 881.088352] Dumping ftrace buffer: > [ 881.088877] (ftrace buffer empty) > [ 881.089414] Kernel Offset: disabled > [ 881.089950] Rebooting in 86400 seconds.. > > = About RaceFuzzer > > RaceFuzzer is a customized version of Syzkaller, specifically tailored > to find race condition bugs in the Linux kernel. While we leverage > many different technique, the notable feature of RaceFuzzer is in > leveraging a custom hypervisor (QEMU/KVM) to interleave the > scheduling. In particular, we modified the hypervisor to intentionally > stall a per-core execution, which is similar to supporting per-core > breakpoint functionality. This allows RaceFuzzer to force the kernel > to deterministically trigger racy condition (which may rarely happen > in practice due to randomness in scheduling). > > RaceFuzzer's C repro always pinpoints two racy syscalls. Since C > repro's scheduling synchronization should be performed at the user > space, its reproducibility is limited (reproduction may take from 1 > second to 10 minutes (or even more), depending on a bug). This is > because, while RaceFuzzer precisely interleaves the scheduling at the > kernel's instruction level when finding this bug, C repro cannot fully > utilize such a feature. Please disregard all code related to > "should_hypercall" in the C repro, as this is only for our debugging > purposes using our own hypervisor.
| |