lkml.org 
[lkml]   [2019]   [Dec]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: [PATCH v4 2/2] xen/blkback: Squeeze page pools if a memory pressure is detected
From
Date
On 09.12.19 20:43, SeongJae Park wrote:
> From: SeongJae Park <sjpark@amazon.de>
>
> Each `blkif` has a free pages pool for the grant mapping. The size of
> the pool starts from zero and be increased on demand while processing
> the I/O requests. If current I/O requests handling is finished or 100
> milliseconds has passed since last I/O requests handling, it checks and
> shrinks the pool to not exceed the size limit, `max_buffer_pages`.
>
> Therefore, `blkfront` running guests can cause a memory pressure in the
> `blkback` running guest by attaching a large number of block devices and
> inducing I/O. System administrators can avoid such problematic
> situations by limiting the maximum number of devices each guest can
> attach. However, finding the optimal limit is not so easy. Improper
> set of the limit can results in the memory pressure or a resource
> underutilization. This commit avoids such problematic situations by
> squeezing the pools (returns every free page in the pool to the system)
> for a while (users can set this duration via a module parameter) if a
> memory pressure is detected.
>
> Discussions
> ===========
>
> The `blkback`'s original shrinking mechanism returns only pages in the
> pool, which are not currently be used by `blkback`, to the system. In
> other words, the pages are not mapped with foreign pages. Because this
> commit is changing only the shrink limit but uses the mechanism as is,
> this commit does not introduce improper mappings related security
> issues.
>
> Once a memory pressure is detected, this commit keeps the squeezing
> limit for a user-specified time duration. The duration should be
> neither too long nor too short. If it is too long, the squeezing
> incurring overhead can reduce the I/O performance. If it is too short,
> `blkback` will not free enough pages to reduce the memory pressure.
> This commit sets the value as `10 milliseconds` by default because it is
> a short time in terms of I/O while it is a long time in terms of memory
> operations. Also, as the original shrinking mechanism works for at
> least every 100 milliseconds, this could be a somewhat reasonable
> choice. I also tested other durations (refer to the below section for
> more details) and confirmed that 10 milliseconds is the one that works
> best with the test. That said, the proper duration depends on actual
> configurations and workloads. That's why this commit is allowing users
> to set it as their optimal value via the module parameter.
>
> Memory Pressure Test
> ====================
>
> To show how this commit fixes the memory pressure situation well, I
> configured a test environment on a xen-running virtualization system.
> On the `blkfront` running guest instances, I attach a large number of
> network-backed volume devices and induce I/O to those. Meanwhile, I
> measure the number of pages that swapped in and out on the `blkback`
> running guest. The test ran twice, once for the `blkback` before this
> commit and once for that after this commit. As shown below, this commit
> has dramatically reduced the memory pressure:
>
> pswpin pswpout
> before 76,672 185,799
> after 212 3,325
>
> Optimal Aggressive Shrinking Duration
> -------------------------------------
>
> To find a best squeezing duration, I repeated the test with three
> different durations (1ms, 10ms, and 100ms). The results are as below:
>
> duration pswpin pswpout
> 1 852 6,424
> 10 212 3,325
> 100 203 3,340
>
> As expected, the memory pressure has decreased as the duration is
> increased, but the reduction stopped from the `10ms`. Based on this
> results, I chose the default duration as 10ms.
>
> Performance Overhead Test
> =========================
>
> This commit could incur I/O performance degradation under severe memory
> pressure because the squeezing will require more page allocations per
> I/O. To show the overhead, I artificially made a worst-case squeezing
> situation and measured the I/O performance of a `blkfront` running
> guest.
>
> For the artificial squeezing, I set the `blkback.max_buffer_pages` using
> the `/sys/module/xen_blkback/parameters/max_buffer_pages` file. We set
> the value to `1024` and `0`. The `1024` is the default value. Setting
> the value as `0` is same to a situation doing the squeezing always
> (worst-case).
>
> For the I/O performance measurement, I use a simple `dd` command.
>
> Default Performance
> -------------------
>
> [dom0]# echo 1024 > /sys/module/xen_blkback/parameters/max_buffer_pages
> [instance]$ for i in {1..5}; do dd if=/dev/zero of=file bs=4k count=$((256*512)); sync; done
> 131072+0 records in
> 131072+0 records out
> 536870912 bytes (537 MB) copied, 11.7257 s, 45.8 MB/s
> 131072+0 records in
> 131072+0 records out
> 536870912 bytes (537 MB) copied, 13.8827 s, 38.7 MB/s
> 131072+0 records in
> 131072+0 records out
> 536870912 bytes (537 MB) copied, 13.8781 s, 38.7 MB/s
> 131072+0 records in
> 131072+0 records out
> 536870912 bytes (537 MB) copied, 13.8737 s, 38.7 MB/s
> 131072+0 records in
> 131072+0 records out
> 536870912 bytes (537 MB) copied, 13.8702 s, 38.7 MB/s
>
> Worst-case Performance
> ----------------------
>
> [dom0]# echo 0 > /sys/module/xen_blkback/parameters/max_buffer_pages
> [instance]$ for i in {1..5}; do dd if=/dev/zero of=file bs=4k count=$((256*512)); sync; done
> 131072+0 records in
> 131072+0 records out
> 536870912 bytes (537 MB) copied, 11.7257 s, 45.8 MB/s
> 131072+0 records in
> 131072+0 records out
> 536870912 bytes (537 MB) copied, 13.878 s, 38.7 MB/s
> 131072+0 records in
> 131072+0 records out
> 536870912 bytes (537 MB) copied, 13.8746 s, 38.7 MB/s
> 131072+0 records in
> 131072+0 records out
> 536870912 bytes (537 MB) copied, 13.8786 s, 38.7 MB/s
> 131072+0 records in
> 131072+0 records out
> 536870912 bytes (537 MB) copied, 13.8749 s, 38.7 MB/s
>
> In short, even worst case squeezing makes no visible performance
> degradation. I think this is due to the slow speed of the I/O. In
> other words, the additional page allocation overhead is hidden under the
> much slower I/O latency.
>
> Nevertheless, pleaset note that this is just a very simple and minimal
> test.
>
> Signed-off-by: SeongJae Park <sjpark@amazon.de>

Reviewed-by: Juergen Gross <jgross@suse.com>

When dropping the domid parameter you can keep the Reviewed-by, of
course.


Juergen

\
 
 \ /
  Last update: 2019-12-10 07:25    [W:0.055 / U:0.556 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site