lkml.org 
[lkml]   [1999]   [Sep]   [3]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
Subject[PATCH] 2.3.15 spinlock metering
Please consider this SMP i386 patch against 2.3.15 for spinlock
"metering". Also attached is a new command, "lockstat", that tells the
kernel to turn the metering on or off, and to read the metering data and
display it in a human-readable form.

Spinlock metering is the runtime recording of data about spinlock usage
-- how often each spinlock is acquired by each locker, how often an
acquisition faced contention and had to wait because someone else owned
that spinlock, and how much time passed before the contention went away.

After the kernel patch is applied to the kernel, a new config variable
(in the "Kernel hacking" subsection) controls whether or not
lockmetering gets compiled into the kernel. A lockmeter-capable kernel
has essentially the same size as a non-lockmeter-capable kernel -- even
smaller, in fact -- because the non-metered kernel's inline locking code
gets replaced by procedure calls, and the multiple-reader-single-writer
locks gets significantly smaller. A lockmeter-capable kernel is
negligibly slower than a normal non-lockmeter-capable kernel when
metering is turned off, and is 1-2% slower when metering (the act of
collecting the data itself) is turned on.

Care has been taken to minimize runtime performance impact of
lockmetering. For example, the data structures that record the counts
and times are separated per-CPU, which means there is no cache coherency
overhead when different CPUs update counts for the same spinlock being
called by the same caller.

As an example of the usefulness of this lockmetering code, I exercised
2x, 3x, and 4xCPU (4x400 MHz Xeon) configurations with an AIM7 workload
that had been modified to remove three synchronous disk subtests that
otherwise would contend on a single disk spindle and produce substantial
idle time. My test workload ran with effectively zero idle time, about
75% user and 25% system time.

My test results:

The 4xCPU 2.3.11 kernel performs about 6-8% faster than the 2.2.10
kernel at the highest loads. The 2.3.x kernel did almost 3x the number
of spinlocks-per-second vs. the 2.2.10 kernel, due to the finer
granularity of 2.3's locking scheme, but 2.3 exhibits lock contention on
2% of these calls vs. 18% in 2.2. When 2.3 does contend, the mean
wait-times are almost 2x those in 2.2. One likely hypothesis for the
longer mean wait-times is that 2.3 has eliminated the quick, trivial
contentions, leaving the longer contentions to raise the mean.

With this workload on this 4xCPU Xeon hardware, spinlock contention in
2.2.10 consumed about 8% of theoretically available CPU cycles (340
milliseconds/second of waiting per 4,000 milliseconds/second
theoretically available) vs. about 2% in 2.3 (95 milliseconds of waiting
per 4,000 milliseconds).

The kernel_flag usage is still significant in 2.3, but its contention is
greatly reduced. In 2.2 I saw contention on 40% of the 41K-per-second
acquisitions, vs. 10% of the 19K-per-second acquisitions done in the 2.3
kernel. Despite the almost 2x increase in mean wait-time on the
kernel_flag in 2.3, the overall number of kernel_flag contention cycles
is about 3x higher in 2.2 vs. 2.3. That is, the 2.3 kernel goes after
kernel_flag much less frequently and sees much less contention on it;
but when there is contention, that contention results in a mean wait
that is twice as long as in 2.2.

Further investigation (using a hacked, unreleased version of the
lockmetering code) shows that the biggest kernel_flag pig in 2.3 is
sys_close(), which holds the kernel_flag for as long as 10-13
milliseconds (on a 400 MHz Xeon CPU) on a regular basis.

A broader discussion of spinlock metering is available at the SGI Open
Source website:
http://oss.sgi.com/projects/lockmeter
as well as a patch against 2.2.10.

John Hawkes
(hawkes@engr.sgi.com)[unhandled content-type:application/x-gzip][unhandled content-type:application/x-gzip]
\
 
 \ /
  Last update: 2005-03-22 13:53    [W:0.058 / U:0.072 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site