Messages in this thread | | | Date | Mon, 23 Aug 1999 11:08:47 -0700 | From | John Hawkes <> | Subject | [PATCH] 2.3.11 spinlock metering |
| |
Please consider this SMP i386 patch against 2.3.11 (or an analogous patch against 2.2.10, previously posted) for spinlock "metering".
Spinlock metering is the runtime recording of data about spinlock usage -- how often each spinlock is acquired by each locker, how often
an acquisition faced contention and had to wait because someone else owned that spinlock, and how much time passed before the contention
went away.
The patch also contains a new command, "lockstat", which turns the functional act of metering on or off in the kernel, and which
retrieves the kernel's data and displays it to the user in a useful human-readable tabular format. Spinlock names and their callers are
displayed, whenever possible, using their symbolic names from the System.map, not their virtual addresses, and mean wait-times are
displayed in microseconds.
After the kernel patch is applied to the kernel, a new config variable (in the "Kernel hacking" subsection) controls whether or not
lockmetering gets compiled into the kernel. A lockmeter-capable kernel has essentially the same size as a non-lockmeter-capable kernel
-- even smaller, in fact -- because the non-metered kernel's inline locking code gets replaced by procedure calls, and the
multiple-reader-single-writer locks gets significantly smaller. A lockmeter-capable kernel is negligibly slower than a normal
non-lockmeter-capable kernel when metering is turned off, and is 1-2% slower when metering (the act of collecting the data itself) is
turned on.
Care has been taken to minimize runtime performance impact of lockmetering. For example, the data structures that record the counts and
times are separated per-CPU, which means there is no cache coherency overhead when different CPUs update counts for the same spinlock
being called by the same caller.
As an example of the usefulness of this lockmetering code, I exercised 2x, 3x, and 4xCPU (4x400 MHz Xeon) configurations with an AIM7
workload that had been modified to remove three synchronous disk subtests that otherwise would contend on a single disk spindle and
produce substantial idle time. My test workload ran with effectively zero idle time, about 75% user and 25% system time.
My test results:
The 4xCPU 2.3.11 kernel performs about 6-8% faster than the 2.2.10 kernel at the highest loads. The 2.3.11 kernel did almost 3x the
number of spinlocks-per-second vs. the 2.2.10 kernel, due to the finer granularity of 2.3's locking scheme, but 2.3 exhibits lock
contention on 2% of these calls vs. 18% in 2.2. When 2.3 does contend, the mean wait-times are almost 2x those in 2.2. One likely
hypothesis for the longer mean wait-times is that 2.3 has eliminated the quick, trivial contentions, leaving the longer contentions to
raise the mean.
With this workload on this 4xCPU Xeon hardware, spinlock contention in 2.2.10 consumed about 8% of theoretically available CPU cycles
(340 milliseconds/second of waiting per 4,000 milliseconds/second theoretically available) vs. about 2% in 2.3.11 (95 milliseconds of
waiting per 4,000 milliseconds).
The kernel_flag usage is still significant in 2.3, but its contention is greatly reduced. In 2.2 I saw contention on 40% of the
41K-per-second acquisitions, vs. 10% of the 19K-per-second acquisitions done in the 2.3 kernel. Despite the almost 2x increase in mean
wait-time on the kernel_flag in 2.3, the overall number of kernel_flag contention cycles is about 3x higher in 2.2.10 vs. 2.3.11. That
is, the 2.3 kernel goes after kernel_flag much less frequently and sees much less contention on it; but when there is contention, that
contention results in a mean wait that is twice as long as in 2.2.
Further investigation (using a hacked, unreleased version of the lockmetering code) shows that the biggest kernel_flag pig in 2.3.11 is
sys_close(), which holds the kernel_flag for as long as 10-13 milliseconds (on a 400 MHz Xeon CPU) on a regular basis.
A broader discussion of spinlock metering is available at the SGI Open Source website: http://oss.sgi.com/projects/lockmeter
John Hawkes (hawkes@engr.sgi.com)[unhandled content-type:application/x-gzip] | |