mirror of
https://mirrors.bfsu.edu.cn/git/linux.git
synced 2024-11-15 00:04:15 +08:00
cb9369bdbb
Each `blkif` has a free pages pool for the grant mapping. The size of
the pool starts from zero and is increased on demand while processing
the I/O requests. If current I/O requests handling is finished or 100
milliseconds has passed since last I/O requests handling, it checks and
shrinks the pool to not exceed the size limit, `max_buffer_pages`.
Therefore, host administrators can cause memory pressure in blkback by
attaching a large number of block devices and inducing I/O. Such
problematic situations can be avoided by limiting the maximum number of
devices that can be attached, but finding the optimal limit is not so
easy. Improper set of the limit can results in memory pressure or a
resource underutilization. This commit avoids such problematic
situations by squeezing the pools (returns every free page in the pool
to the system) for a while (users can set this duration via a module
parameter) if memory pressure is detected.
Discussions
===========
The `blkback`'s original shrinking mechanism returns only pages in the
pool which are not currently be used by `blkback` to the system. In
other words, the pages that are not mapped with granted pages. Because
this commit is changing only the shrink limit but still uses the same
freeing mechanism it does not touch pages which are currently mapping
grants.
Once memory pressure is detected, this commit keeps the squeezing limit
for a user-specified time duration. The duration should be neither too
long nor too short. If it is too long, the squeezing incurring overhead
can reduce the I/O performance. If it is too short, `blkback` will not
free enough pages to reduce the memory pressure. This commit sets the
value as `10 milliseconds` by default because it is a short time in
terms of I/O while it is a long time in terms of memory operations.
Also, as the original shrinking mechanism works for at least every 100
milliseconds, this could be a somewhat reasonable choice. I also tested
other durations (refer to the below section for more details) and
confirmed that 10 milliseconds is the one that works best with the test.
That said, the proper duration depends on actual configurations and
workloads. That's why this commit allows users to set the duration as a
module parameter.
Memory Pressure Test
====================
To show how this commit fixes the memory pressure situation well, I
configured a test environment on a xen-running virtualization system.
On the `blkfront` running guest instances, I attach a large number of
network-backed volume devices and induce I/O to those. Meanwhile, I
measure the number of pages that swapped in (pswpin) and out (pswpout)
on the `blkback` running guest. The test ran twice, once for the
`blkback` before this commit and once for that after this commit. As
shown below, this commit has dramatically reduced the memory pressure:
pswpin pswpout
before 76,672 185,799
after 867 3,967
Optimal Aggressive Shrinking Duration
-------------------------------------
To find a best squeezing duration, I repeated the test with three
different durations (1ms, 10ms, and 100ms). The results are as below:
duration pswpin pswpout
1 707 5,095
10 867 3,967
100 362 3,348
As expected, the memory pressure decreases as the duration increases,
but the reduction become slow from the `10ms`. Based on this results, I
chose the default duration as 10ms.
Performance Overhead Test
=========================
This commit could incur I/O performance degradation under severe memory
pressure because the squeezing will require more page allocations per
I/O. To show the overhead, I artificially made a worst-case squeezing
situation and measured the I/O performance of a `blkfront` running
guest.
For the artificial squeezing, I set the `blkback.max_buffer_pages` using
the `/sys/module/xen_blkback/parameters/max_buffer_pages` file. In this
test, I set the value to `1024` and `0`. The `1024` is the default
value. Setting the value as `0` is same to a situation doing the
squeezing always (worst-case).
If the underlying block device is slow enough, the squeezing overhead
could be hidden. For the reason, I use a fast block device, namely the
rbd[1]:
# xl block-attach guest phy:/dev/ram0 xvdb w
For the I/O performance measurement, I run a simple `dd` command 5 times
directly to the device as below and collect the 'MB/s' results.
$ for i in {1..5}; do dd if=/dev/zero of=/dev/xvdb \
bs=4k count=$((256*512)); sync; done
The results are as below. 'max_pgs' represents the value of the
`blkback.max_buffer_pages` parameter.
max_pgs Min Max Median Avg Stddev
0 417 423 420 419.4 2.5099801
1024 414 425 416 417.8 4.4384682
No difference proven at 95.0% confidence
In short, even worst case squeezing on ramdisk based fast block device
makes no visible performance degradation. Please note that this is just
a very simple and minimal test. On systems using super-fast block
devices and a special I/O workload, the results might be different. If
you have any doubt, test on your machine with your workload to find the
optimal squeezing duration for you.
[1] https://www.kernel.org/doc/html/latest/admin-guide/blockdev/ramdisk.html
Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: SeongJae Park <sjpark@amazon.de>
Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
|
||
|---|---|---|
| .. | ||
| obsolete | ||
| removed | ||
| stable | ||
| testing | ||
| README | ||
This directory attempts to document the ABI between the Linux kernel and userspace, and the relative stability of these interfaces. Due to the everchanging nature of Linux, and the differing maturity levels, these interfaces should be used by userspace programs in different ways. We have four different levels of ABI stability, as shown by the four different subdirectories in this location. Interfaces may change levels of stability according to the rules described below. The different levels of stability are: stable/ This directory documents the interfaces that the developer has defined to be stable. Userspace programs are free to use these interfaces with no restrictions, and backward compatibility for them will be guaranteed for at least 2 years. Most interfaces (like syscalls) are expected to never change and always be available. testing/ This directory documents interfaces that are felt to be stable, as the main development of this interface has been completed. The interface can be changed to add new features, but the current interface will not break by doing this, unless grave errors or security problems are found in them. Userspace programs can start to rely on these interfaces, but they must be aware of changes that can occur before these interfaces move to be marked stable. Programs that use these interfaces are strongly encouraged to add their name to the description of these interfaces, so that the kernel developers can easily notify them if any changes occur (see the description of the layout of the files below for details on how to do this.) obsolete/ This directory documents interfaces that are still remaining in the kernel, but are marked to be removed at some later point in time. The description of the interface will document the reason why it is obsolete and when it can be expected to be removed. removed/ This directory contains a list of the old interfaces that have been removed from the kernel. Every file in these directories will contain the following information: What: Short description of the interface Date: Date created KernelVersion: Kernel version this feature first showed up in. Contact: Primary contact for this interface (may be a mailing list) Description: Long description of the interface and how to use it. Users: All users of this interface who wish to be notified when it changes. This is very important for interfaces in the "testing" stage, so that kernel developers can work with userspace developers to ensure that things do not break in ways that are unacceptable. It is also important to get feedback for these interfaces to make sure they are working in a proper way and do not need to be changed further. How things move between levels: Interfaces in stable may move to obsolete, as long as the proper notification is given. Interfaces may be removed from obsolete and the kernel as long as the documented amount of time has gone by. Interfaces in the testing state can move to the stable state when the developers feel they are finished. They cannot be removed from the kernel tree without going through the obsolete state first. It's up to the developer to place their interfaces in the category they wish for it to start out in. Notable bits of non-ABI, which should not under any circumstances be considered stable: - Kconfig. Userspace should not rely on the presence or absence of any particular Kconfig symbol, in /proc/config.gz, in the copy of .config commonly installed to /boot, or in any invocation of the kernel build process. - Kernel-internal symbols. Do not rely on the presence, absence, location, or type of any kernel symbol, either in System.map files or the kernel binary itself. See Documentation/process/stable-api-nonsense.rst.