Messages in this thread | | | From | Oscar Shiang <> | Subject | Re: [patch v11 00/13] extensible prctl task isolation interface and vmstat sync | Date | Wed, 9 Mar 2022 23:31:59 +0800 |
| |
On Mar 8, 2022, at 9:12 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote: > On Tue, Mar 08, 2022 at 02:32:46PM +0800, Oscar Shiang wrote: > > On Feb 24, 2022, at 1:31 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote: > > > Hi Oscar, > > > > > > On Sat, Feb 19, 2022 at 04:02:10PM +0800, Oscar Shiang wrote: > > > > Hi Marcelo, > > > > > > > > I tried to apply your patches to kernel v5.15.18-rt28 and measured > > > > the latencies through oslat [1]. > > > > > > > > It turns out that the peak latency (around 100us) can drop to about 90us. > > > > The result is impressive since I only changed the guest's kernel > > > > instead of installing the patched kernel to both host and guest. > > > > > > > > However, I am still curious about: > > > > 1) Why did I catch a bigger maximum latency in almost each of the > > > > results of applying task isolation patches? Or does it come from > > > > other reasons? > > > > > > There are a number of things that need to be done in order to have an > > > "well enough" isolated CPU so you can measure latency reliably: > > > > > > * Boot a kernel with isolated CPU (or better, use realtime-virtual-host profile of > > > https://github.com/redhat-performance/tuned.git, which does a bunch of > > > other things to avoid interruptions to isolated CPUs). > > > * Apply the userspace patches at https://people.redhat.com/~mtosatti/task-isol-v6-userspace-patches/ > > > to util-linux and rt-tests. > > > > > > Run oslat with chisol: > > > > > > chisol -q vmstat_sync -I conf oslat -c ... > > > > > > Where chisol is from patched util-linux and oslat from patched rt-tests. > > > > > > If you had "-f 1" (FIFO priority), on oslat, then the vmstat work would be hung. > > > > > > Are you doing those things? > > > > > > > 2) Why did we only get a 10us improvement on quiescing vmstat? > > > > > > If you did not have FIFO priority on oslat, then other daemons > > > could be interrupting it, so better make sure the 10us improvement > > > you see is due to vmstat_flush workqueue work not executing anymore. > > > > > > The testcase i use is: > > > > > > Stock kernel: > > > > > > terminal 1: > > > # oslat -f 1 -c X ... > > > > > > terminal 2: > > > # echo 1 > /proc/sys/vm/stat_refresh > > > (hang) > > > > > > Patched kernel: > > > > > > terminal 1: > > > # chisol -q vmstat_sync -I conf oslat -f 1 -c X ... > > > > > > terminal 2: > > > # echo 1 > /proc/sys/vm/stat_refresh > > > # > > > > Sure, I did see the terminal hung during oslat with FIFO priority. > > > > BTW, thanks for providing this test case. I used to run all workload stuff to just > > verify the improvement of task isolation. It is a more straightr- forward way to do. > > > > > > [1]: The result and the test scripts I used can be found at > > > > https://gist.github.com/OscarShiang/8b530a00f472fd1c39f5979ee601516d#testing-task-isolation-via-oslat > > > > > > OK, you seem to be doing everything necessary for chisol > > > to work. Does /proc/pid/task_isolation of the oslat worker thread > > > (note its not the same pid as the main oslat thread) show "vmstat" > > > configured and activated for quiesce? > > > > The status of task_isolation seems to be set properly with "vmstat" and activated > > > > > However 100us is really high. You should be able to get < 10us with > > > realtime-virtual-host (i see 4us on an idle system). > > > > > > The answer might be: because 10us is what it takes to execute > > > vmstat_worker on the isolated CPU (you can verify with tracepoints). > > > > > > That time depends on the number of per-CPU vmstat variables that need flushing, > > > i suppose... > > > > Considering the interferences outside of the KVM, I have redone the measurements > > directly on my x86_64 computer [1]. > > > > As result, most of the latencies are down to 60us (and below). There are still > > some latencies larger than 80us, I am working on and trying to figure out the reason. > > > > [1]: https://gist.github.com/OscarShiang/202eb691e649557fe3eaa5ec67a5aa82 > > Oscar, > > Did you confirm with hwlatdetect that the BIOS does not have long > running SMIs?
Marcelo,
I have run hwlatdetect and the result is shown below:
hwlatdetect: test duration 900 seconds detector: tracer parameters: Latency threshold: 0us Sample window: 1000000us Sample width: 500000us Non-sampling period: 500000us Output File: test.report
Starting test test finished Max Latency: 48us Samples recorded: 37 Samples exceeding threshold: 37 SMIs during run: 0 report saved to test.report (37 samples) ts: 1646837340.346151918, inner:5, outer:9 ts: 1646837351.550312752, inner:46, outer:45 ts: 1646837381.549331178, inner:45, outer:0 ts: 1646837400.008623200, inner:0, outer:9 ts: 1646837425.578093371, inner:0, outer:8 ts: 1646837429.587363003, inner:0, outer:1 ts: 1646837436.549050243, inner:45, outer:45 ts: 1646837580.005173999, inner:0, outer:9 ts: 1646837605.591017161, inner:7, outer:8 ts: 1646837635.552410329, inner:0, outer:1 ts: 1646837639.246489489, inner:9, outer:5 ts: 1646837645.426611917, inner:9, outer:5 ts: 1646837651.550721975, inner:0, outer:1 ts: 1646837728.549928137, inner:40, outer:47 ts: 1646837756.281606376, inner:1, outer:8 ts: 1646837757.492693661, inner:0, outer:1 ts: 1646837759.355807689, inner:13, outer:13 ts: 1646837761.590570928, inner:0, outer:1 ts: 1646837762.382475433, inner:5, outer:9 ts: 1646837764.185172836, inner:0, outer:9 ts: 1646837768.675668348, inner:1, outer:0 ts: 1646837776.485184319, inner:0, outer:1 ts: 1646837777.544517878, inner:45, outer:41 ts: 1646837855.549140154, inner:45, outer:0 ts: 1646837886.492523509, inner:1, outer:0 ts: 1646837897.544172247, inner:45, outer:0 ts: 1646837933.550015925, inner:48, outer:48 ts: 1646837981.546557947, inner:45, outer:45 ts: 1646837998.051615598, inner:0, outer:1 ts: 1646838030.550099263, inner:45, outer:0 ts: 1646838072.549664559, inner:0, outer:46 ts: 1646838106.571038324, inner:0, outer:1 ts: 1646838107.547228682, inner:1, outer:0 ts: 1646838114.549686904, inner:45, outer:45 ts: 1646838162.549477219, inner:46, outer:47 ts: 1646838180.014465679, inner:5, outer:9 ts: 1646838237.486873064, inner:0, outer:1
It seems that there is no SMI occurring (but some other latencies around 40us?)
> Also, for the software part, you could save time by using the > realtime-virtual-host profile (check /usr/lib/tuned/realtime-virtual-host/ > to see what its doing in addition to isolcpus=).
Yes, I have switched to realtime-virtual-host profile with its kernel cmdline args.
| |