lkml.org 
[lkml]   [2012]   [Oct]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: [PATCH RFC 1/2] kvm: Handle undercommitted guest case in PLE handler
    From
    Date
    On Wed, 2012-10-10 at 00:21 +0530, Raghavendra K T wrote:
    > * Avi Kivity <avi@redhat.com> [2012-10-04 17:00:28]:
    >
    > > On 10/04/2012 03:07 PM, Peter Zijlstra wrote:
    > > > On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
    > > >>
    > > >> Again the numbers are ridiculously high for arch_local_irq_restore.
    > > >> Maybe there's a bad perf/kvm interaction when we're injecting an
    > > >> interrupt, I can't believe we're spending 84% of the time running the
    > > >> popf instruction.
    > > >
    > > > Smells like a software fallback that doesn't do NMI, hrtimer based
    > > > sampling typically hits popf where we re-enable interrupts.
    > >
    > > Good nose, that's probably it. Raghavendra, can you ensure that the PMU
    > > is properly exposed? 'dmesg' in the guest will tell. If it isn't, -cpu
    > > host will expose it (and a good idea anyway to get best performance).
    > >
    >
    > Hi Avi, you are right. SandyBridge machine result was not proper.
    > I cleaned up the services, enabled PMU, re-ran all the test again.
    >
    > Here is the summary:
    > We do get good benefit by increasing ple window. Though we don't
    > see good benefit for kernbench and sysbench, for ebizzy, we get huge
    > improvement for 1x scenario. (almost 2/3rd of ple disabled case).
    >
    > Let me know if you think we can increase the default ple_window
    > itself to 16k.
    >
    > I am experimenting with V2 version of undercommit improvement(this) patch
    > series, But I think if you wish to go for increase of
    > default ple_window, then we would have to measure the benefit of patches
    > when ple_window = 16k.
    >
    > I can respin the whole series including this default ple_window change.
    >
    > I also have the perf kvm top result for both ebizzy and kernbench.
    > I think they are in expected lines now.
    >
    > Improvements
    > ================
    >
    > 16 core PLE machine with 16 vcpu guest
    >
    > base = 3.6.0-rc5 + ple handler optimization patches
    > base_pleopt_16k = base + ple_window = 16k
    > base_pleopt_32k = base + ple_window = 32k
    > base_pleopt_nople = base + ple_gap = 0
    > kernbench, hackbench, sysbench (time in sec lower is better)
    > ebizzy (rec/sec higher is better)
    >
    > % improvements w.r.t base (ple_window = 4k)
    > ---------------+---------------+-----------------+-------------------+
    > |base_pleopt_16k| base_pleopt_32k | base_pleopt_nople |
    > ---------------+---------------+-----------------+-------------------+
    > kernbench_1x | 0.42371 | 1.15164 | 0.09320 |
    > kernbench_2x | -1.40981 | -17.48282 | -570.77053 |
    > ---------------+---------------+-----------------+-------------------+
    > sysbench_1x | -0.92367 | 0.24241 | -0.27027 |
    > sysbench_2x | -2.22706 |-0.30896 | -1.27573 |
    > sysbench_3x | -0.75509 | 0.09444 | -2.97756 |
    > ---------------+---------------+-----------------+-------------------+
    > ebizzy_1x | 54.99976 | 67.29460 | 74.14076 |
    > ebizzy_2x | -8.83386 |-27.38403 | -96.22066 |
    > ---------------+---------------+-----------------+-------------------+
    >
    > perf kvm top observation for kernbench and ebizzy (nople, 4k, 32k window)
    > ========================================================================

    Is the perf data for 1x overcommit?

    > pleopt ple_gap=0
    > --------------------
    > ebizzy : 18131 records/s
    > 63.78% [guest.kernel] [g] _raw_spin_lock_irqsave
    > 5.65% [guest.kernel] [g] smp_call_function_many
    > 3.12% [guest.kernel] [g] clear_page
    > 3.02% [guest.kernel] [g] down_read_trylock
    > 1.85% [guest.kernel] [g] async_page_fault
    > 1.81% [guest.kernel] [g] up_read
    > 1.76% [guest.kernel] [g] native_apic_mem_write
    > 1.70% [guest.kernel] [g] find_vma

    Does 'perf kvm top' not give host samples at the same time? Would be
    nice to see the host overhead as a function of varying ple window. I
    would expect that to be the major difference between 4/16/32k window
    sizes.

    A big concern I have (if this is 1x overcommit) for ebizzy is that it
    has just terrible scalability to begin with. I do not think we should
    try to optimize such a bad workload.

    > kernbench :Elapsed Time 29.4933 (27.6007)
    > 5.72% [guest.kernel] [g] async_page_fault
    > 3.48% [guest.kernel] [g] pvclock_clocksource_read
    > 2.68% [guest.kernel] [g] copy_user_generic_unrolled
    > 2.58% [guest.kernel] [g] clear_page
    > 2.09% [guest.kernel] [g] page_cache_get_speculative
    > 2.00% [guest.kernel] [g] do_raw_spin_lock
    > 1.78% [guest.kernel] [g] unmap_single_vma
    > 1.74% [guest.kernel] [g] kmem_cache_alloc

    >
    > pleopt ple_window = 4k
    > ---------------------------
    > ebizzy: 10176 records/s
    > 69.17% [guest.kernel] [g] _raw_spin_lock_irqsave
    > 3.34% [guest.kernel] [g] clear_page
    > 2.16% [guest.kernel] [g] down_read_trylock
    > 1.94% [guest.kernel] [g] async_page_fault
    > 1.89% [guest.kernel] [g] native_apic_mem_write
    > 1.63% [guest.kernel] [g] smp_call_function_many
    > 1.58% [guest.kernel] [g] SetPageLRU
    > 1.37% [guest.kernel] [g] up_read
    > 1.01% [guest.kernel] [g] find_vma
    >
    >
    > kernbench: 29.9533
    > nts: 240K cycles
    > 6.04% [guest.kernel] [g] async_page_fault
    > 4.17% [guest.kernel] [g] pvclock_clocksource_read
    > 3.28% [guest.kernel] [g] clear_page
    > 2.57% [guest.kernel] [g] copy_user_generic_unrolled
    > 2.30% [guest.kernel] [g] do_raw_spin_lock
    > 2.13% [guest.kernel] [g] _raw_spin_lock_irqsave
    > 1.93% [guest.kernel] [g] page_cache_get_speculative
    > 1.92% [guest.kernel] [g] unmap_single_vma
    > 1.77% [guest.kernel] [g] kmem_cache_alloc
    > 1.61% [guest.kernel] [g] __d_lookup_rcu
    > 1.19% [guest.kernel] [g] find_vma
    > 1.19% [guest.kernel] [g] __list_del_entry
    >
    >
    > pleopt: ple_window=16k
    > -------------------------
    > ebizzy: 16990
    > 62.35% [guest.kernel] [g] _raw_spin_lock_irqsave
    > 5.22% [guest.kernel] [g] smp_call_function_many
    > 3.57% [guest.kernel] [g] down_read_trylock
    > 3.20% [guest.kernel] [g] clear_page
    > 2.16% [guest.kernel] [g] up_read
    > 1.89% [guest.kernel] [g] find_vma
    > 1.86% [guest.kernel] [g] async_page_fault
    > 1.81% [guest.kernel] [g] native_apic_mem_write
    >
    > kernbench: 28.5
    > 6.24% [guest.kernel] [g] async_page_fault
    > 4.16% [guest.kernel] [g] pvclock_clocksource_read
    > 3.33% [guest.kernel] [g] clear_page
    > 2.50% [guest.kernel] [g] copy_user_generic_unrolled
    > 2.08% [guest.kernel] [g] do_raw_spin_lock
    > 1.98% [guest.kernel] [g] unmap_single_vma
    > 1.89% [guest.kernel] [g] kmem_cache_alloc
    > 1.82% [guest.kernel] [g] page_cache_get_speculative
    > 1.46% [guest.kernel] [g] __d_lookup_rcu
    > 1.42% [guest.kernel] [g] _raw_spin_lock_irqsave
    > 1.15% [guest.kernel] [g] __list_del_entry
    > 1.10% [guest.kernel] [g] find_vma
    >
    >
    >
    > Detailed result for the run
    > =============================
    > patched = base_pleopt_16k
    > +-----------+-----------+-----------+------------+-----------+
    > kernbench
    > +-----------+-----------+-----------+------------+-----------+
    > base stddev patched stdev %improve
    > +-----------+-----------+-----------+------------+-----------+
    > 1x 30.0440 1.1896 29.9167 1.6755 0.42371
    > 2x 62.0083 3.4884 62.8825 2.5509 -1.40981
    > +-----------+-----------+-----------+------------+-----------+
    > +-----------+-----------+-----------+------------+-----------+
    > sysbench
    > +-----------+-----------+-----------+------------+-----------+
    > 1x 7.1779 0.0577 7.2442 0.0479 -0.92367
    > 2x 15.5362 0.3370 15.8822 0.3591 -2.22706
    > 3x 23.8249 0.1513 24.0048 0.1844 -0.75509
    > +-----------+-----------+-----------+------------+-----------+
    > +-----------+-----------+-----------+------------+-----------+
    > ebizzy
    > +-----------+-----------+-----------+------------+-----------+
    > 1x 10358.0000 442.6598 16054.8750 252.5088 54.99976
    > 2x 2705.5000 130.0286 2466.5000 120.0024 -8.83386
    > +-----------+-----------+-----------+------------+-----------+
    >
    > patched = base_pleopt_32k
    > +-----------+-----------+-----------+------------+-----------+
    > kernbench
    > +-----------+-----------+-----------+------------+-----------+
    > base stddev patched stdev %improve
    > +-----------+-----------+-----------+------------+-----------+
    > 1x 30.0440 1.1896 29.6980 0.6760 1.15164
    > 2x 62.0083 3.4884 72.8491 4.4616 -17.48282
    > +-----------+-----------+-----------+------------+-----------+
    > +-----------+-----------+-----------+------------+-----------+
    > sysbench
    > +-----------+-----------+-----------+------------+-----------+
    > 1x 7.1779 0.0577 7.1605 0.0447 0.24241
    > 2x 15.5362 0.3370 15.5842 0.1731 -0.30896
    > 3x 23.8249 0.1513 23.8024 0.2342 0.09444
    > +-----------+-----------+-----------+------------+-----------+
    > +-----------+-----------+-----------+------------+-----------+
    > ebizzy
    > +-----------+-----------+-----------+------------+-----------+
    > 1x 10358.0000 442.6598 17328.3750 281.4569 67.29460
    > 2x 2705.5000 130.0286 1964.6250 143.0793 -27.38403
    > +-----------+-----------+-----------+------------+-----------+
    >
    > patched = base_pleopt_nople
    > +-----------+-----------+-----------+------------+-----------+
    > kernbench
    > +-----------+-----------+-----------+------------+-----------+
    > base stddev patched stdev %improve
    > +-----------+-----------+-----------+------------+-----------+
    > 1x 30.0440 1.1896 30.0160 0.7523 0.09320
    > 2x 62.0083 3.4884 415.9334 189.9901 -570.77053
    > +-----------+-----------+-----------+------------+-----------+
    > +-----------+-----------+-----------+------------+-----------+
    > sysbench
    > +-----------+-----------+-----------+------------+-----------+
    > 1x 7.1779 0.0577 7.1973 0.0354 -0.27027
    > 2x 15.5362 0.3370 15.7344 0.2315 -1.27573
    > 3x 23.8249 0.1513 24.5343 0.3437 -2.97756
    > +-----------+-----------+-----------+------------+-----------+
    > +-----------+-----------+-----------+------------+-----------+
    > ebizzy
    > +-----------+-----------+-----------+------------+-----------+
    > 1x 10358.0000 442.6598 18037.5000 315.2074 74.14076
    > 2x 2705.5000 130.0286 102.2500 104.3521 -96.22066
    > +-----------+-----------+-----------+------------+-----------+
    >




    \
     
     \ /
      Last update: 2012-10-10 05:41    [W:3.113 / U:0.044 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site