lkml.org 
[lkml]   [2013]   [Jan]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Subject[PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers
    Date
    Version 4 of this series is a rebase over latest 3.8-rc4+. Moreover, I
    updated the mechanism that implements automatic symbol loading for new
    modules. It was affected by the refactorings around finit_module.

    While waiting for feedback who could imagine picking this up for merge,
    I wrote a tiny tutorial, see below.


    Here is the original series intro again:

    This adds the infrastructure and first tools that make kernel debugging
    through gdb more comfortable. Since 7.0, gdb supports python scripting.
    And this opens the doors to automate steps like the tedious loading of
    module symbols at the right address, resolving per-cpu variables or even
    retrieving the current kernel log without resuming an stopped target.

    Many of the helpers naturally depend on the layout of structures or
    internal mechanics of the kernel. So the best place to maintain such
    things, keeping them consistent with the corresponding kernel is, well,
    the kernel itself.

    While these scripts have been originally developed for debugging via
    QEMU/KVM, I've now also added the required bits for KGDB. Works fine,
    but as QEMU/KVM tends to outperform KGDB it remains the recommendation
    - when available.

    There are two architecture dependencies so far, one regarding per-cpu,
    the other regarding thread_info calculation. None of them I was able to
    test on a target, so I'm counting on review/testing by the corresponding
    communities.

    This series should be considered the foundation of much more kernel
    state exploration helpers, e.g. around tasks, timers, locks, sockets -
    I guess people will have even more ideas.


    And this is a tutorial for the gdb extension using QEMU/KVM as target
    platform:

    o Set up a virtual Linux machine for KVM (see www.linux-kvm.org and
    www.qemu.org for more details)

    o Build the kernel with this series applied, enabling CONFIG_DEBUG_INFO
    (but leave CONFIG_DEBUG_INFO_REDUCED off)

    o Install that kernel on the guest

    o Enable the gdb stub of QEMU/KVM, either
    - at VM startup time by appending "-s" to the QEMU command line
    or
    - during runtime by issuing "gdbserver" from the QEMU monitor
    console

    o cd /path/to/linux-build

    o Start gdb: gdb vmlinux

    o Attach to the booted guest:
    (gdb) target remote :1234

    o Load module (and main kernel) symbols:
    (gdb) lx-symbols
    loading vmlinux
    scanning for modules in /home/user/linux/build
    loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
    loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
    loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
    loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
    loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
    ...
    loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko

    o Set a breakpoint on some not yet loaded module function, e.g.:
    (gdb) b btrfs_init_sysfs
    Function "btrfs_init_sysfs" not defined.
    Make breakpoint pending on future shared library load? (y or [n]) y
    Breakpoint 1 (btrfs_init_sysfs) pending.

    o Continue the target

    o Load the module on the target and watch what happens:
    loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
    loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
    loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
    loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko

    Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
    36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);

    o Let's examine the current task a bit:
    (gdb) p ().pid
    = 4998
    (gdb) p ().comm
    = "modprobe\000\000\000\000\000\000\000"

    o Dump the log buffer of target kernel:
    (gdb) lx-dmesg
    [ 0.000000] Initializing cgroup subsys cpuset
    [ 0.000000] Initializing cgroup subsys cpu
    [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
    [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
    [ 0.000000] e820: BIOS-provided physical RAM map:
    [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
    [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
    ....

    o Make use of the per-cpu helper for the current or a specified CPU:
    (gdb) p ("runqueues").nr_running
    = 1
    (gdb) p ("runqueues", 2).nr_running
    = 0

    o And now we are digging deep into hrtimers using the container_of
    helper:
    (gdb) set = ("hrtimer_bases").clock_base[0].active.next
    (gdb) p *(, "struct hrtimer", "node")
    = {
    node = {
    node = {
    __rb_parent_color = 18446612133355256072,
    rb_right = 0x0 <irq_stack_union>,
    rb_left = 0x0 <irq_stack_union>
    },
    expires = {
    tv64 = 1835268000000
    }
    },
    _softexpires = {
    tv64 = 1835268000000
    },
    function = 0xffffffff81078232 <tick_sched_timer>,
    base = 0xffff88003fd0d6f0,
    state = 1,
    start_pid = 0,
    start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
    start_comm = "swapper/2\000\000\000\000\000\000"
    }

    Hope this provided some ideas and inspirations on how the commands and
    helper functions can support kernel development.

    Enjoy,
    Jan

    PS: Also available via git://git.kiszka.org/linux.git queues/gdb-scripts

    CC: "David S. Miller" <davem@davemloft.net>
    CC: Fenghua Yu <fenghua.yu@intel.com>
    CC: Kay Sievers <kay@vrfy.org>
    CC: linux-ia64@vger.kernel.org
    CC: linux-kbuild@vger.kernel.org
    CC: Michal Marek <mmarek@suse.cz>
    CC: sparclinux@vger.kernel.org
    CC: Tony Luck <tony.luck@intel.com>

    Jan Kiszka (13):
    scripts/gdb: Add infrastructure
    scripts/gdb: Add container_of helper and convenience function
    scripts/gdb: Add lx-symbols command
    scripts/gdb: Add get_target_endianness helper
    scripts/gdb: Add read_u16/32/64 helpers
    scripts/gdb: Add lx-dmesg command
    scripts/gdb: Add task iteration helper
    scripts/gdb: Add helper and convenience function to look up tasks
    scripts/gdb: Add is_target_arch helper
    scripts/gdb: Add internal helper and convenience function to retrieve
    thread_info
    scripts/gdb: Add get_gdbserver_type helper
    scripts/gdb: Add internal helper and convenience function for per-cpu
    lookup
    scripts/gdb: Add lx_current convenience function

    Makefile | 5 +-
    scripts/Makefile | 3 +-
    scripts/gdb/Makefile | 9 +++
    scripts/gdb/dmesg.py | 63 ++++++++++++++++++
    scripts/gdb/percpu.py | 76 ++++++++++++++++++++++
    scripts/gdb/symbols.py | 153 ++++++++++++++++++++++++++++++++++++++++++++
    scripts/gdb/task.py | 108 +++++++++++++++++++++++++++++++
    scripts/gdb/utils.py | 137 +++++++++++++++++++++++++++++++++++++++
    scripts/gdb/vmlinux-gdb.py | 28 ++++++++
    9 files changed, 580 insertions(+), 2 deletions(-)
    create mode 100644 scripts/gdb/Makefile
    create mode 100644 scripts/gdb/dmesg.py
    create mode 100644 scripts/gdb/percpu.py
    create mode 100644 scripts/gdb/symbols.py
    create mode 100644 scripts/gdb/task.py
    create mode 100644 scripts/gdb/utils.py
    create mode 100644 scripts/gdb/vmlinux-gdb.py

    --
    1.7.3.4



    \
     
     \ /
      Last update: 2013-01-21 19:02    [W:4.605 / U:0.076 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site