lkml.org 
[lkml]   [2011]   [Dec]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: Kswapd in 3.2.0-rc5 is a CPU hog
    [Let's CC linux-mm]

    On Wed 21-12-11 07:10:36, Nikolay S. wrote:
    > Hello,
    >
    > I'm using 3.2-rc5 on a machine, which atm does almost nothing except
    > file system operations and network i/o (i.e. file server). And there is
    > a problem with kswapd.

    What kind of filesystem do you use?

    >
    > I'm playing with dd:
    > dd if=/some/big/file of=/dev/null bs=8M
    >
    > I.e. I'm filling page cache.
    >
    > So when the machine is just rebooted, kswapd during this operation is
    > almost idle, just 5-8 percent according to top.
    >
    > After ~5 days of uptime (5 days, 2:10), the same operation demands ~70%
    > for kswapd:
    >
    > PID USER S %CPU %MEM TIME+ SWAP COMMAND
    > 420 root R 70 0.0 22:09.60 0 kswapd0
    > 17717 nowhere D 27 0.2 0:01.81 10m dd
    >
    > In fact, kswapd cpu usage on this operation steadily increases over
    > time.
    >
    > Also read performance degrades over time. After reboot:
    > dd if=/some/big/file of=/dev/null bs=8M
    > 1019+1 records in
    > 1019+1 records out
    > 8553494018 bytes (8.6 GB) copied, 16.211 s, 528 MB/s
    >
    > After ~5 days uptime:
    > dd if=/some/big/file of=/dev/null bs=8M
    > 1019+1 records in
    > 1019+1 records out
    > 8553494018 bytes (8.6 GB) copied, 29.0507 s, 294 MB/s
    >
    > Whereas raw disk sequential read performance stays the same:
    > dd if=/some/big/file of=/dev/null bs=8M iflag=direct
    > 1019+1 records in
    > 1019+1 records out
    > 8553494018 bytes (8.6 GB) copied, 14.7286 s, 581 MB/s
    >
    > Also after dropping caches, situation somehow improves, but not to the
    > state of freshly restarted system:
    > PID USER S %CPU %MEM TIME+ SWAP COMMAND
    > 420 root S 39 0.0 23:31.17 0 kswapd0
    > 19829 nowhere D 24 0.2 0:02.72 7764 dd
    >
    > perf shows:
    >
    > 31.24% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock
    > 26.19% kswapd0 [kernel.kallsyms] [k] shrink_slab
    > 16.28% kswapd0 [kernel.kallsyms] [k] prune_super
    > 6.55% kswapd0 [kernel.kallsyms] [k] grab_super_passive
    > 5.35% kswapd0 [kernel.kallsyms] [k] down_read_trylock
    > 4.03% kswapd0 [kernel.kallsyms] [k] up_read
    > 2.31% kswapd0 [kernel.kallsyms] [k] put_super
    > 1.81% kswapd0 [kernel.kallsyms] [k] drop_super
    > 0.99% kswapd0 [kernel.kallsyms] [k] __put_super
    > 0.25% kswapd0 [kernel.kallsyms] [k] __isolate_lru_page
    > 0.23% kswapd0 [kernel.kallsyms] [k] free_pcppages_bulk
    > 0.19% kswapd0 [r8169] [k] rtl8169_interrupt
    > 0.15% kswapd0 [kernel.kallsyms] [k] twa_interrupt

    Quite a lot of time spent shrinking slab (dcache I guess) and a lot of
    spin lock contention.
    Could you also take few snapshots of /proc/420/stack to see what kswapd
    is doing.

    >
    > P.S.: The message above was written couple of days ago. Now I'm at 10
    > days uptime, and this is the result as of today
    > PID USER S %CPU %MEM TIME+ SWAP COMMAND
    > 420 root R 93 0.0 110:48.48 0 kswapd0
    > 30085 nowhere D 42 0.2 0:04.36 10m dd
    >
    > PPS: Please CC me.

    --
    Michal Hocko
    SUSE Labs
    SUSE LINUX s.r.o.
    Lihovarska 1060/12
    190 00 Praha 9
    Czech Republic


    \
     
     \ /
      Last update: 2011-12-21 10:55    [W:6.953 / U:0.052 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site