lkml.org 
[lkml]   [2021]   [Dec]   [30]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: cpufreq: intel_pstate: map utilization into the pstate range


On Thu, 30 Dec 2021, Rafael J. Wysocki wrote:

> On Thu, Dec 30, 2021 at 7:21 PM Julia Lawall <julia.lawall@inria.fr> wrote:
> >
> >
> >
> > On Thu, 30 Dec 2021, Rafael J. Wysocki wrote:
> >
> > > On Thu, Dec 30, 2021 at 6:54 PM Julia Lawall <julia.lawall@inria.fr> wrote:
> > > >
> > > > > > The effect is the same. But that approach is indeed simpler than patching
> > > > > > the kernel.
> > > > >
> > > > > It is also applicable when intel_pstate runs in the active mode.
> > > > >
> > > > > As for the results that you have reported, it looks like the package
> > > > > power on these systems is dominated by package voltage and going from
> > > > > P-state 20 to P-state 21 causes that voltage to increase significantly
> > > > > (the observed RAM energy usage pattern is consistent with that). This
> > > > > means that running at P-states above 20 is only really justified if
> > > > > there is a strict performance requirement that can't be met otherwise.
> > > > >
> > > > > Can you please check what value is there in the base_frequency sysfs
> > > > > attribute under cpuX/cpufreq/?
> > > >
> > > > 2100000, which should be pstate 21
> > > >
> > > > >
> > > > > I'm guessing that the package voltage level for P-states 10 and 20 is
> > > > > the same, so the power difference between them is not significant
> > > > > relative to the difference between P-state 20 and 21 and if increasing
> > > > > the P-state causes some extra idle time to appear in the workload
> > > > > (even though there is not enough of it to prevent to overall
> > > > > utilization from increasing), then the overall power draw when running
> > > > > at P-state 10 may be greater that for P-state 20.
> > > >
> > > > My impression is that the package voltage level for P-states 10 to 20 is
> > > > high enough that increasing the frequency has little impact. But the code
> > > > runs twice as fast, which reduces the execution time a lot, saving energy.
> > > >
> > > > My first experiment had only one running thread. I also tried running 32
> > > > spinning threads for 10 seconds, ie using up one package and leaving the
> > > > other idle. In this case, instead of staying around 600J for pstates
> > > > 10-20, the pstate rises from 743 to 946. But there is still a gap between
> > > > 20 and 21, with 21 being 1392J.
> > > >
> > > > > You can check if there is any C-state residency difference between
> > > > > these two cases by running the workload under turbostat in each of
> > > > > them.
> > > >
> > > > The C1 and C6 cases (CPU%c1 and CPU%c6) are about the same between 20 and
> > > > 21, whether with 1 thread or with 32 thread.
> > >
> > > I meant to compare P-state 10 and P-state 20.
> > >
> > > 20 and 21 are really close as far as the performance is concerned, so
> > > I wouldn't expect to see any significant C-state residency difference
> > > between them.
> >
> > There's also no difference between 10 and 20. This seems normal, because
> > the same cores are either fully used or fully idle in both cases. The
> > idle ones are almost always in C6.
>
> The turbostat output sent by you previously shows that the CPUs doing
> the work are only about 15-or-less percent busy, though, and you get
> quite a bit of C-state residency on them. I'm assuming that this is
> for 1 running thread.
>
> Can you please run the 32 spinning threads workload (ie. on one
> package) and with P-state locked to 10 and then to 20 under turbostat
> and send me the turbostat output for both runs?

Attached.

Pstate 10: spin_minmax_10_dahu-9_5.15.0freq_schedutil_11.turbo
Pstate 20: spin_minmax_20_dahu-9_5.15.0freq_schedutil_11.turbo

julia10.041290 sec
Package Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI POLL C1 C1E C6 POLL% C1% C1E% C6% CPU%c1 CPU%c6 CoreTmp PkgTmp Pkg%pc2 Pkg%pc6 Pkg_J RAM_J PKG_% RAM_%
- - - 16 1.57 1000 2101 4971 0 1 21 227 2512 0.00 0.00 0.02 98.69 1.86 96.56 36 36 49.15 0.25 580.08 318.91 0.00 0.00
0 0 0 2 0.18 1000 2095 105 0 0 0 0 355 0.00 0.00 0.00 99.84 1.90 97.92 32 36 0.29 0.25 300.17 153.07 0.00 0.00
0 0 32 0 0.02 1001 2095 22 0 0 0 1 27 0.00 0.00 0.01 99.98 2.06
0 1 4 0 0.01 1000 2095 22 0 0 0 2 25 0.00 0.00 0.01 99.98 0.12 99.87 34
0 1 36 0 0.01 1000 2095 23 0 0 0 1 25 0.00 0.00 0.01 99.98 0.12
0 2 8 0 0.02 1000 2095 47 0 0 0 2 48 0.00 0.00 0.01 99.97 0.25 99.73 34
0 2 40 0 0.01 1000 2095 27 0 0 0 1 23 0.00 0.00 0.01 99.99 0.26
0 3 12 0 0.01 1000 2095 24 0 0 0 1 25 0.00 0.00 0.00 99.99 0.13 99.86 33
0 3 44 0 0.01 1000 2095 21 0 0 0 1 21 0.00 0.00 0.00 99.99 0.13
0 4 14 0 0.01 1000 2095 23 0 0 0 0 26 0.00 0.00 0.00 99.99 0.13 99.85 35
0 4 46 0 0.01 1000 2095 21 0 0 0 0 22 0.00 0.00 0.00 99.99 0.14
0 5 10 991 99.31 1000 2095 2523 0 0 0 1 18 0.00 0.00 0.01 0.69 0.08 0.61 34
0 5 42 0 0.01 1000 2095 20 0 0 0 0 21 0.00 0.00 0.00 99.99 99.38
0 6 6 0 0.01 1000 2095 32 0 0 0 0 32 0.00 0.00 0.00 99.99 0.18 99.81 33
0 6 38 0 0.01 1000 2095 32 0 0 0 0 32 0.00 0.00 0.00 99.99 0.17
0 7 2 0 0.01 1000 2095 27 0 0 0 0 27 0.00 0.00 0.00 99.99 0.14 99.85 36
0 7 34 0 0.01 1000 2095 27 0 0 0 0 26 0.00 0.00 0.00 99.99 0.14
0 8 16 0 0.01 1000 2095 24 0 0 0 0 25 0.00 0.00 0.00 99.99 0.13 99.86 34
0 8 48 0 0.01 1000 2095 23 0 0 0 0 23 0.00 0.00 0.00 99.99 0.13
0 9 20 1 0.13 1000 2095 561 0 1 1 200 362 0.00 0.00 0.91 98.98 2.48 97.39 33
0 9 52 0 0.02 1001 2095 36 0 0 0 0 36 0.00 0.00 0.00 99.99 2.58
0 10 24 0 0.01 1000 2095 23 0 0 0 1 23 0.00 0.00 0.01 99.98 0.13 99.86 34
0 10 56 0 0.01 1000 2095 22 0 0 0 1 23 0.00 0.00 0.00 99.99 0.13
0 11 28 0 0.01 1000 2095 23 0 0 0 1 23 0.00 0.00 0.00 99.99 0.14 99.85 34
0 11 60 0 0.01 1000 2095 25 0 0 0 0 25 0.00 0.00 0.00 99.99 0.14
0 12 30 0 0.01 1000 2095 26 0 0 0 0 27 0.00 0.00 0.00 99.99 0.13 99.85 35
0 12 62 0 0.01 1000 2095 19 0 0 0 0 19 0.00 0.00 0.00 99.99 0.14
0 13 26 0 0.01 1000 2095 20 0 0 0 0 22 0.00 0.00 0.00 99.99 0.13 99.85 34
0 13 58 0 0.01 1000 2095 23 0 0 0 0 24 0.00 0.00 0.00 99.99 0.13
0 14 22 0 0.02 1000 2095 27 0 0 0 0 28 0.00 0.00 0.00 99.99 0.15 99.84 33
0 14 54 0 0.01 1000 2095 27 0 0 0 0 26 0.00 0.00 0.00 99.99 0.15
0 15 18 0 0.01 1000 2095 21 0 0 0 0 22 0.00 0.00 0.00 99.99 0.12 99.87 35
0 15 50 0 0.01 1000 2095 22 0 0 0 0 21 0.00 0.00 0.00 99.99 0.12
1 0 1 0 0.02 1000 2095 22 0 0 20 0 25 0.00 0.09 0.00 99.90 0.22 99.76 28 30 97.98 0.25 279.91 165.85 0.00 0.00
1 0 33 0 0.01 1000 2095 28 0 0 0 0 28 0.00 0.00 0.00 99.99 0.22
1 1 5 0 0.02 1000 2095 42 0 0 0 0 41 0.00 0.00 0.00 99.98 0.17 99.80 29
1 1 37 0 0.01 1000 2095 22 0 0 0 0 22 0.00 0.00 0.00 99.99 0.19
1 2 9 0 0.02 1000 2095 38 0 0 0 1 38 0.00 0.00 0.01 99.98 0.23 99.75 28
1 2 41 0 0.02 1000 2095 38 0 0 0 1 36 0.00 0.00 0.00 99.99 0.24
1 3 13 1 0.08 1000 2095 134 0 0 0 2 133 0.00 0.00 0.00 99.93 0.49 99.43 28
1 3 45 0 0.02 1001 2095 27 0 0 0 0 28 0.00 0.00 0.00 99.99 0.55
1 4 15 0 0.01 1000 2095 30 0 0 0 0 32 0.00 0.00 0.00 99.99 0.15 99.84 29
1 4 47 0 0.01 1000 2095 19 0 0 0 0 19 0.00 0.00 0.00 99.99 0.15
1 5 11 0 0.02 1000 2095 32 0 0 0 0 36 0.00 0.00 0.00 99.98 0.26 99.72 27
1 5 43 0 0.02 1000 2095 56 0 0 0 2 54 0.00 0.00 0.00 99.98 0.26
1 6 7 0 0.01 1000 2095 22 0 0 0 0 20 0.00 0.00 0.00 99.99 0.10 99.89 29
1 6 39 0 0.01 1000 2095 22 0 0 0 0 21 0.00 0.00 0.00 99.99 0.10
1 7 3 0 0.01 1000 2095 23 0 0 0 0 22 0.00 0.00 0.00 99.99 0.17 99.82 30
1 7 35 0 0.03 1000 2095 29 0 0 0 0 31 0.00 0.00 0.00 99.98 0.15
1 8 17 0 0.02 1000 2095 21 0 0 0 0 27 0.00 0.00 0.00 99.99 0.13 99.85 28
1 8 49 0 0.01 1000 2095 16 0 0 0 0 16 0.00 0.00 0.00 99.99 0.14
1 9 21 0 0.01 1000 2095 18 0 0 0 1 17 0.00 0.00 0.01 99.98 0.21 99.78 27
1 9 53 1 0.06 1000 2095 32 0 0 0 2 31 0.00 0.00 0.01 99.93 0.16
1 10 25 0 0.01 1000 2095 20 0 0 0 1 21 0.00 0.00 0.00 99.99 0.31 99.68 29
1 10 57 1 0.08 1000 2095 50 0 0 0 2 56 0.00 0.00 0.00 99.92 0.24
1 11 29 0 0.01 1000 2095 19 0 0 0 0 17 0.00 0.00 0.00 99.99 0.17 99.82 28
1 11 61 0 0.02 1000 2095 37 0 0 0 0 37 0.00 0.00 0.00 99.98 0.16
1 12 31 0 0.01 1000 2095 17 0 0 0 0 19 0.00 0.00 0.00 99.99 0.23 99.76 28
1 12 63 0 0.04 1000 2095 46 0 0 0 0 49 0.00 0.00 0.00 99.97 0.20
1 13 27 0 0.01 1000 2095 16 0 0 0 0 15 0.00 0.00 0.00 99.68 0.52 99.47 29
1 13 59 0 0.04 1000 2095 45 0 0 0 1 40 0.00 0.00 0.01 99.95 0.18
1 14 23 0 0.01 1000 2095 31 0 0 0 0 18 0.00 0.00 0.00 99.99 0.20 99.79 28
1 14 55 0 0.03 1000 2095 48 0 0 0 1 42 0.00 0.00 0.00 99.97 0.18
1 15 19 0 0.01 1000 2095 31 0 0 0 0 22 0.00 0.00 0.00 99.99 0.19 99.80 29
1 15 51 1 0.07 1000 2095 22 0 0 0 0 17 0.00 0.00 0.00 99.94 0.13
10.041491 sec
Package Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI POLL C1 C1E C6 POLL% C1% C1E% C6% CPU%c1 CPU%c6 CoreTmp PkgTmp Pkg%pc2 Pkg%pc6 Pkg_J RAM_J PKG_% RAM_%
- - - 16 1.57 1000 2097 4274 0 1 20 19 1863 0.00 0.01 0.00 98.49 1.80 96.63 38 38 49.14 0.07 579.96 320.76 0.00 0.00
0 0 0 2 0.18 1000 2095 97 0 0 0 0 332 0.00 0.00 0.00 99.84 1.80 98.02 34 38 0.37 0.13 300.26 154.74 0.00 0.00
0 0 32 0 0.02 1001 2095 17 0 0 0 0 19 0.00 0.00 0.00 99.99 1.96
0 1 4 0 0.02 1000 2095 39 0 0 0 0 39 0.00 0.00 0.00 99.98 0.20 99.78 36
0 1 36 0 0.01 1000 2095 20 0 0 0 0 19 0.00 0.00 0.00 99.99 0.21
0 2 8 0 0.02 1000 2095 32 0 0 0 0 36 0.00 0.00 0.00 99.98 0.23 99.74 35
0 2 40 0 0.01 1000 2095 27 0 0 0 0 27 0.00 0.00 0.00 99.99 0.24
0 3 12 0 0.01 1000 2095 22 0 0 0 0 24 0.00 0.00 0.00 99.99 0.16 99.83 34
0 3 44 0 0.01 1000 2095 18 0 0 0 0 18 0.00 0.00 0.00 99.99 0.16
0 4 14 0 0.01 1000 2095 23 0 0 0 0 24 0.00 0.00 0.00 99.99 0.13 99.86 36
0 4 46 0 0.01 1000 2095 24 0 0 0 0 23 0.00 0.00 0.00 99.99 0.13
0 5 10 991 99.35 1000 2095 2522 0 0 0 1 17 0.00 0.00 0.01 0.64 0.08 0.57 35
0 5 42 0 0.01 1000 2095 18 0 0 0 0 18 0.00 0.00 0.00 100.00 99.42
0 6 6 0 0.01 1000 2095 36 0 0 2 1 33 0.00 0.08 0.00 99.91 0.26 99.73 34
0 6 38 0 0.01 1000 2095 18 0 0 0 1 18 0.00 0.00 0.00 99.99 0.26
0 7 2 0 0.01 1000 2095 21 0 0 0 0 20 0.00 0.00 0.00 99.99 0.12 99.87 38
0 7 34 0 0.01 1000 2095 21 0 0 0 0 21 0.00 0.00 0.00 99.99 0.12
0 8 16 0 0.01 1000 2095 18 0 0 0 0 20 0.00 0.00 0.00 99.99 0.10 99.89 35
0 8 48 0 0.01 1000 2095 18 0 0 0 0 18 0.00 0.00 0.00 99.99 0.10
0 9 20 0 0.01 1000 2095 26 0 0 0 0 19 0.00 0.00 0.00 99.72 0.44 99.55 35
0 9 52 0 0.01 1000 2095 29 0 0 0 0 29 0.00 0.00 0.00 99.99 0.17
0 10 24 0 0.01 1000 2095 21 0 0 0 0 21 0.00 0.00 0.00 99.99 0.13 99.86 35
0 10 56 0 0.01 1000 2095 21 0 0 0 0 21 0.00 0.00 0.00 99.99 0.13
0 11 28 0 0.01 1000 2095 23 0 0 0 0 24 0.00 0.00 0.00 99.99 0.14 99.85 35
0 11 60 0 0.01 1000 2095 27 0 0 0 0 22 0.00 0.00 0.00 99.99 0.14
0 12 30 0 0.01 1000 2095 20 0 0 0 0 21 0.00 0.00 0.00 99.99 0.12 99.87 36
0 12 62 0 0.01 1000 2095 23 0 0 0 0 20 0.00 0.00 0.00 99.99 0.12
0 13 26 0 0.01 1000 2095 20 0 0 0 0 20 0.00 0.00 0.00 99.99 0.17 99.82 35
0 13 58 0 0.01 1000 2095 20 0 0 0 0 20 0.00 0.00 0.00 99.99 0.17
0 14 22 0 0.01 1000 2095 25 0 0 0 0 23 0.00 0.00 0.00 99.99 0.17 99.82 35
0 14 54 0 0.01 1001 2095 18 0 0 0 0 18 0.00 0.00 0.00 99.99 0.17
0 15 18 0 0.01 1000 2095 12 0 0 0 1 10 0.00 0.00 0.01 99.99 0.06 99.93 37
0 15 50 0 0.01 1000 2095 8 0 0 0 1 8 0.00 0.00 0.00 99.99 0.06
1 0 1 0 0.01 1000 2095 27 0 1 4 0 9 0.00 0.08 0.00 99.91 0.13 99.86 28 31 97.98 0.00 279.71 166.02 0.00 0.00
1 0 33 0 0.01 1000 2095 10 0 0 0 0 9 0.00 0.00 0.00 100.00 0.14
1 1 5 0 0.02 1000 2095 28 0 0 0 0 30 0.00 0.00 0.00 99.98 0.25 99.73 29
1 1 37 0 0.03 1000 2095 37 0 0 0 0 36 0.00 0.00 0.00 99.97 0.24
1 2 9 0 0.01 1000 2095 23 0 0 0 0 23 0.00 0.00 0.00 99.99 0.11 99.88 28
1 2 41 0 0.01 1000 2095 25 0 0 0 0 24 0.00 0.00 0.00 99.99 0.11
1 3 13 1 0.07 1000 2095 126 0 0 0 0 121 0.00 0.00 0.00 99.94 0.43 99.50 28
1 3 45 0 0.01 1001 2095 19 0 0 0 0 18 0.00 0.00 0.00 99.99 0.49
1 4 15 0 0.01 1000 2095 14 0 0 0 0 12 0.00 0.00 0.00 99.99 0.10 99.89 29
1 4 47 0 0.01 1000 2095 27 0 0 0 0 19 0.00 0.00 0.00 99.99 0.10
1 5 11 0 0.01 1000 2095 29 0 0 0 0 27 0.00 0.00 0.00 99.99 0.47 99.51 27
1 5 43 0 0.01 1000 2095 39 0 0 8 1 31 0.00 0.28 0.04 99.67 0.47
1 6 7 0 0.01 1000 2095 23 0 0 0 1 21 0.00 0.00 0.01 99.98 0.11 99.89 29
1 6 39 0 0.01 1000 2095 24 0 0 0 1 21 0.00 0.00 0.01 99.99 0.11
1 7 3 0 0.04 1000 2095 41 0 0 0 3 30 0.00 0.00 0.01 99.95 0.14 99.82 30
1 7 35 0 0.01 1000 2095 7 0 0 0 1 7 0.00 0.00 0.00 99.99 0.17
1 8 17 0 0.03 1000 2095 43 0 0 3 1 43 0.00 0.12 0.00 99.85 0.31 99.66 29
1 8 49 0 0.01 1000 2095 25 0 0 0 0 18 0.00 0.00 0.00 99.99 0.33
1 9 21 0 0.01 1000 2095 13 0 0 0 0 13 0.00 0.00 0.00 99.99 0.29 99.71 28
1 9 53 1 0.07 1000 2095 46 0 0 0 2 46 0.00 0.00 0.02 99.92 0.23
1 10 25 0 0.01 1000 2095 23 0 0 0 0 21 0.00 0.00 0.00 99.99 0.11 99.88 29
1 10 57 0 0.02 1000 2095 21 0 0 0 0 20 0.00 0.00 0.00 99.99 0.10
1 11 29 0 0.02 1000 2095 42 0 0 0 0 40 0.00 0.00 0.00 99.98 0.34 99.64 28
1 11 61 1 0.07 1000 2095 43 0 0 0 0 49 0.00 0.00 0.00 99.94 0.29
1 12 31 0 0.01 1000 2095 15 0 0 0 0 11 0.00 0.00 0.00 99.99 0.19 99.80 28
1 12 63 0 0.03 1000 2095 46 0 0 0 2 38 0.00 0.00 0.01 99.97 0.17
1 13 27 0 0.01 1000 2095 18 0 0 0 0 11 0.00 0.00 0.00 99.99 0.17 99.82 29
1 13 59 0 0.03 1000 2095 43 0 0 0 0 33 0.00 0.00 0.00 99.97 0.15
1 14 23 0 0.01 1000 2095 19 0 0 0 0 13 0.00 0.00 0.00 99.99 0.18 99.81 28
1 14 55 0 0.03 1000 2095 47 0 0 2 1 35 0.00 0.00 0.01 99.97 0.17
1 15 19 0 0.01 1000 2095 20 0 0 1 1 10 0.00 0.00 0.00 99.99 0.17 99.82 29
1 15 51 1 0.07 1000 2095 27 0 0 0 0 22 0.00 0.00 0.00 99.94 0.11
\
 
 \ /
  Last update: 2021-12-30 19:44    [W:0.158 / U:0.052 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site