lkml.org 
[lkml]   [2008]   [Apr]   [4]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: Scalability requirements for sysv ipc (+namespaces broken with SEM_UNDO)
Mike Galbraith wrote:
> On Sun, 2008-03-30 at 16:12 +0200, Manfred Spraul wrote:
>
>>Mike Galbraith wrote:
>>
>>>On Sat, 2008-03-22 at 20:08 +0100, Manfred Spraul wrote:
>>>
>>>
>>>
>>>>just the normal performance of 2.6.25-rc3 is abyssimal, 55 to 60% slower
>>>>than 2.6.18.8:
>>>>
>>>
>>>After manually reverting 3e148c79938aa39035669c1cfa3ff60722134535,
>>>2.6.25.git scaled linearly
>>
>>We can't just revert that patch: with IDR, a global lock is mandatory :-(
>>We must either revert the whole idea of using IDR or live with the
>>reduced scalability.
>
>
> Yeah, I looked at the problem, but didn't know what the heck to do about
> it, so just grabbed my axe to verify/quantify.
>
>
>>Actually, there are further bugs: the undo structures are not
>>namespace-aware, thus semop with SEM_UNDO, unshare, create new array
>>with same id, but more semaphores, another semop with SEM_UNDO will
>>corrupt kernel memory :-(
>>I'll try to clean up the bugs first, then I'll look at the scalability
>>again.
>
>
> Great!
>
> -Mike
>
>
>
>

I could get better results with the following solution:
wrote an RCU-based idr api (layers allocation is managed similarly to
the radix-tree one)

Using it in the ipc code makes me get rid of the read lock taken in
ipc_lock() (the one introduced in 3e148c79938aa39035669c1cfa3ff60722134535).

You'll find the results in attachment (kernel is 2.6.25-rc3-mm1).
output.25_rc3_mm1.ref.8 --> pmsg output for the 2.6.25-rc3-mm1
plot.25_rc3_mm1.ref.8 --> previous file results for use by gnuplot
output.25_rc3_mm1.ridr.8 --> pmsg output for the 2.6.25-rc3-mm1
+ rcu-based idrs
plot.25_rc3_mm1.ridr.8 --> previous file results for use by gnuplot


I think I should be able to send a patch next week. It is presently an
uggly code: I copied idr.c and idr.h into ridr.c and ridr.h to go fast,
so didn't do any code factorization.

Regards
Nadia



pmsg [nr queues] [timeout]
Using 1 queues/cpus (2 threads) for 5 seconds.
thread 0: sysvmsg 0 type 1 bound to 0001h
thread 1: sysvmsg 0 type 0 bound to 0001h
Result matrix:
Thread 0: 488650 1: 488650
Total: 977300
pmsg [nr queues] [timeout]
Using 2 queues/cpus (4 threads) for 5 seconds.
thread 0: sysvmsg 32768 type 1 bound to 0001h
thread 1: sysvmsg 65537 type 1 bound to 0002h
thread 3: sysvmsg 65537 type 0 bound to 0002h
thread 2: sysvmsg 32768 type 0 bound to 0001h
Result matrix:
Thread 0: 223991 2: 223991
Thread 1: 225588 3: 225588
Total: 899158
pmsg [nr queues] [timeout]
Using 3 queues/cpus (6 threads) for 5 seconds.
thread 0: sysvmsg 98304 type 1 bound to 0001h
thread 1: sysvmsg 131073 type 1 bound to 0002h
thread 2: sysvmsg 163842 type 1 bound to 0004h
thread 5: sysvmsg 163842 type 0 bound to 0004h
thread 4: sysvmsg 131073 type 0 bound to 0002h
thread 3: sysvmsg 98304 type 0 bound to 0001h
Result matrix:
Thread 0: 183407 3: 183407
Thread 1: 184030 4: 184030
Thread 2: 357875 5: 357876
Total: 1450625
pmsg [nr queues] [timeout]
Using 4 queues/cpus (8 threads) for 5 seconds.
thread 0: sysvmsg 196608 type 1 bound to 0001h
thread 1: sysvmsg 229377 type 1 bound to 0002h
thread 2: sysvmsg 262146 type 1 bound to 0004h
thread 3: sysvmsg 294915 type 1 bound to 0008h
thread 5: sysvmsg 229377 type 0 bound to 0002h
thread 6: sysvmsg 262146 type 0 bound to 0004h
thread 7: sysvmsg 294915 type 0 bound to 0008h
thread 4: sysvmsg 196608 type 0 bound to 0001h
Result matrix:
Thread 0: 166911 4: 166912
Thread 1: 159281 5: 159281
Thread 2: 166024 6: 166024
Thread 3: 167440 7: 167440
Total: 1319313
pmsg [nr queues] [timeout]
Using 5 queues/cpus (10 threads) for 5 seconds.
thread 0: sysvmsg 327680 type 1 bound to 0001h
thread 2: sysvmsg 393218 type 1 bound to 0004h
thread 3: sysvmsg 425987 type 1 bound to 0008h
thread 4: sysvmsg 458756 type 1 bound to 0010h
thread 9: sysvmsg 458756 type 0 bound to 0010h
thread 6: sysvmsg 360449 type 0 bound to 0002h
thread 8: sysvmsg 425987 type 0 bound to 0008h
thread 7: sysvmsg 393218 type 0 bound to 0004h
thread 1: sysvmsg 360449 type 1 bound to 0002h
thread 5: sysvmsg 327680 type 0 bound to 0001h
Result matrix:
Thread 0: 39740 5: 39740
Thread 1: 40399 6: 40399
Thread 2: 40326 7: 40327
Thread 3: 39290 8: 39290
Thread 4: 68684 9: 68685
Total: 456880
pmsg [nr queues] [timeout]
Using 6 queues/cpus (12 threads) for 5 seconds.
thread 0: sysvmsg 491520 type 1 bound to 0001h
thread 1: sysvmsg 524289 type 1 bound to 0002h
thread 2: sysvmsg 557058 type 1 bound to 0004h
thread 3: sysvmsg 589827 type 1 bound to 0008h
thread 4: sysvmsg 622596 type 1 bound to 0010h
thread 5: sysvmsg 655365 type 1 bound to 0020h
thread 6: sysvmsg 491520 type 0 bound to 0001h
thread 11: sysvmsg 655365 type 0 bound to 0020h
thread 10: sysvmsg 622596 type 0 bound to 0010h
thread 8: sysvmsg 557058 type 0 bound to 0004h
thread 9: sysvmsg 589827 type 0 bound to 0008h
thread 7: sysvmsg 524289 type 0 bound to 0002h
Result matrix:
Thread 0: 27901 6: 27901
Thread 1: 28554 7: 28555
Thread 2: 28471 8: 28472
Thread 3: 28015 9: 28016
Thread 4: 28213 10: 28213
Thread 5: 28396 11: 28396
Total: 339103
pmsg [nr queues] [timeout]
Using 7 queues/cpus (14 threads) for 5 seconds.
thread 0: sysvmsg 688128 type 1 bound to 0001h
thread 1: sysvmsg 720897 type 1 bound to 0002h
thread 2: sysvmsg 753666 type 1 bound to 0004h
thread 3: sysvmsg 786435 type 1 bound to 0008h
thread 4: sysvmsg 819204 type 1 bound to 0010h
thread 5: sysvmsg 851973 type 1 bound to 0020h
thread 6: sysvmsg 884742 type 1 bound to 0040h
thread 13: sysvmsg 884742 type 0 bound to 0040h
thread 7: sysvmsg 688128 type 0 bound to 0001h
thread 11: sysvmsg 819204 type 0 bound to 0010h
thread 12: sysvmsg 851973 type 0 bound to 0020h
thread 8: sysvmsg 720897 type 0 bound to 0002h
thread 10: sysvmsg 786435 type 0 bound to 0008h
thread 9: sysvmsg 753666 type 0 bound to 0004h
Result matrix:
Thread 0: 12201 7: 12201
Thread 1: 12451 8: 12452
Thread 2: 12345 9: 12345
Thread 3: 12277 10: 12278
Thread 4: 12259 11: 12259
Thread 5: 12364 12: 12365
Thread 6: 24666 13: 24666
Total: 197129
pmsg [nr queues] [timeout]
Using 8 queues/cpus (16 threads) for 5 seconds.
thread 0: sysvmsg 917504 type 1 bound to 0001h
thread 1: sysvmsg 950273 type 1 bound to 0002h
thread 2: sysvmsg 983042 type 1 bound to 0004h
thread 3: sysvmsg 1015811 type 1 bound to 0008h
thread 4: sysvmsg 1048580 type 1 bound to 0010h
thread 5: sysvmsg 1081349 type 1 bound to 0020h
thread 6: sysvmsg 1114118 type 1 bound to 0040h
thread 7: sysvmsg 1146887 type 1 bound to 0080h
thread 15: sysvmsg 1146887 type 0 bound to 0080h
thread 8: sysvmsg 917504 type 0 bound to 0001h
thread 14: sysvmsg 1114118 type 0 bound to 0040h
thread 13: sysvmsg 1081349 type 0 bound to 0020h
thread 12: sysvmsg 1048580 type 0 bound to 0010h
thread 11: sysvmsg 1015811 type 0 bound to 0008h
thread 10: sysvmsg 983042 type 0 bound to 0004h
thread 9: sysvmsg 950273 type 0 bound to 0002h
Result matrix:
Thread 0: 11082 8: 11083
Thread 1: 11461 9: 11461
Thread 2: 11430 10: 11431
Thread 3: 11184 11: 11185
Thread 4: 11373 12: 11374
Thread 5: 11290 13: 11291
Thread 6: 11265 14: 11266
Thread 7: 11324 15: 11325
Total: 180825
1 977300
2 899158
3 1450625
4 1319313
5 456880
6 339103
7 197129
8 180825
pmsg [nr queues] [timeout]
Using 1 queues/cpus (2 threads) for 5 seconds.
thread 0: sysvmsg 0 type 1 bound to 0001h
thread 1: sysvmsg 0 type 0 bound to 0001h
Result matrix:
Thread 0: 549365 1: 549365
Total: 1098730
pmsg [nr queues] [timeout]
Using 2 queues/cpus (4 threads) for 5 seconds.
thread 0: sysvmsg 32768 type 1 bound to 0001h
thread 1: sysvmsg 65537 type 1 bound to 0002h
thread 3: sysvmsg 65537 type 0 bound to 0002h
thread 2: sysvmsg 32768 type 0 bound to 0001h
Result matrix:
Thread 0: 245002 2: 245003
Thread 1: 246618 3: 246619
Total: 983242
pmsg [nr queues] [timeout]
Using 3 queues/cpus (6 threads) for 5 seconds.
thread 0: sysvmsg 98304 type 1 bound to 0001h
thread 1: sysvmsg 131073 type 1 bound to 0002h
thread 2: sysvmsg 163842 type 1 bound to 0004h
thread 5: sysvmsg 163842 type 0 bound to 0004h
thread 4: sysvmsg 131073 type 0 bound to 0002h
thread 3: sysvmsg 98304 type 0 bound to 0001h
Result matrix:
Thread 0: 231585 3: 231586
Thread 1: 233256 4: 233256
Thread 2: 509630 5: 509631
Total: 1948944
pmsg [nr queues] [timeout]
Using 4 queues/cpus (8 threads) for 5 seconds.
thread 0: sysvmsg 196608 type 1 bound to 0001h
thread 1: sysvmsg 229377 type 1 bound to 0002h
thread 2: sysvmsg 262146 type 1 bound to 0004h
thread 3: sysvmsg 294915 type 1 bound to 0008h
thread 5: sysvmsg 229377 type 0 bound to 0002h
thread 6: sysvmsg 262146 type 0 bound to 0004h
thread 7: sysvmsg 294915 type 0 bound to 0008h
thread 4: sysvmsg 196608 type 0 bound to 0001h
Result matrix:
Thread 0: 233392 4: 233392
Thread 1: 234485 5: 234486
Thread 2: 235604 6: 235604
Thread 3: 235683 7: 235683
Total: 1878329
pmsg [nr queues] [timeout]
Using 5 queues/cpus (10 threads) for 5 seconds.
thread 0: sysvmsg 327680 type 1 bound to 0001h
thread 2: sysvmsg 393218 type 1 bound to 0004h
thread 3: sysvmsg 425987 type 1 bound to 0008h
thread 4: sysvmsg 458756 type 1 bound to 0010h
thread 1: sysvmsg 360449 type 1 bound to 0002h
thread 9: sysvmsg 458756 type 0 bound to 0010h
thread 6: sysvmsg 360449 type 0 bound to 0002h
thread 7: sysvmsg 393218 type 0 bound to 0004h
thread 8: sysvmsg 425987 type 0 bound to 0008h
thread 5: sysvmsg 327680 type 0 bound to 0001h
Result matrix:
Thread 0: 216094 5: 216095
Thread 1: 227109 6: 227110
Thread 2: 222042 7: 222042
Thread 3: 222708 8: 222708
Thread 4: 467186 9: 467187
Total: 2710281
pmsg [nr queues] [timeout]
Using 6 queues/cpus (12 threads) for 5 seconds.
thread 0: sysvmsg 491520 type 1 bound to 0001h
thread 1: sysvmsg 524289 type 1 bound to 0002h
thread 2: sysvmsg 557058 type 1 bound to 0004h
thread 3: sysvmsg 589827 type 1 bound to 0008h
thread 4: sysvmsg 622596 type 1 bound to 0010h
thread 5: sysvmsg 655365 type 1 bound to 0020h
thread 6: sysvmsg 491520 type 0 bound to 0001h
thread 11: sysvmsg 655365 type 0 bound to 0020h
thread 8: sysvmsg 557058 type 0 bound to 0004h
thread 10: sysvmsg 622596 type 0 bound to 0010h
thread 9: sysvmsg 589827 type 0 bound to 0008h
thread 7: sysvmsg 524289 type 0 bound to 0002h
Result matrix:
Thread 0: 224027 6: 224028
Thread 1: 225394 7: 225394
Thread 2: 223545 8: 223545
Thread 3: 223599 9: 223599
Thread 4: 224632 10: 224633
Thread 5: 224511 11: 224512
Total: 2691419
pmsg [nr queues] [timeout]
Using 7 queues/cpus (14 threads) for 5 seconds.
thread 0: sysvmsg 688128 type 1 bound to 0001h
thread 1: sysvmsg 720897 type 1 bound to 0002h
thread 2: sysvmsg 753666 type 1 bound to 0004h
thread 3: sysvmsg 786435 type 1 bound to 0008h
thread 4: sysvmsg 819204 type 1 bound to 0010h
thread 5: sysvmsg 851973 type 1 bound to 0020h
thread 6: sysvmsg 884742 type 1 bound to 0040h
thread 13: sysvmsg 884742 type 0 bound to 0040h
thread 8: sysvmsg 720897 type 0 bound to 0002h
thread 9: sysvmsg 753666 type 0 bound to 0004h
thread 10: sysvmsg 786435 type 0 bound to 0008h
thread 11: sysvmsg 819204 type 0 bound to 0010h
thread 7: sysvmsg 688128 type 0 bound to 0001h
thread 12: sysvmsg 851973 type 0 bound to 0020h
Result matrix:
Thread 0: 188264 7: 188264
Thread 1: 190677 8: 190677
Thread 2: 188850 9: 188851
Thread 3: 188925 10: 188926
Thread 4: 190333 11: 190334
Thread 5: 189235 12: 189235
Thread 6: 386862 13: 386863
Total: 3046296
pmsg [nr queues] [timeout]
Using 8 queues/cpus (16 threads) for 5 seconds.
thread 0: sysvmsg 917504 type 1 bound to 0001h
thread 1: sysvmsg 950273 type 1 bound to 0002h
thread 2: sysvmsg 983042 type 1 bound to 0004h
thread 3: sysvmsg 1015811 type 1 bound to 0008h
thread 4: sysvmsg 1048580 type 1 bound to 0010h
thread 5: sysvmsg 1081349 type 1 bound to 0020h
thread 6: sysvmsg 1114118 type 1 bound to 0040h
thread 7: sysvmsg 1146887 type 1 bound to 0080h
thread 8: sysvmsg 917504 type 0 bound to 0001h
thread 10: sysvmsg 983042 type 0 bound to 0004h
thread 11: sysvmsg 1015811 type 0 bound to 0008h
thread 12: sysvmsg 1048580 type 0 bound to 0010h
thread 13: sysvmsg 1081349 type 0 bound to 0020h
thread 9: sysvmsg 950273 type 0 bound to 0002h
thread 15: sysvmsg 1146887 type 0 bound to 0080h
thread 14: sysvmsg 1114118 type 0 bound to 0040h
Result matrix:
Thread 0: 187613 8: 187614
Thread 1: 190488 9: 190489
Thread 2: 190112 10: 190113
Thread 3: 190374 11: 190375
Thread 4: 190658 12: 190658
Thread 5: 190508 13: 190508
Thread 6: 189222 14: 189223
Thread 7: 190272 15: 190272
Total: 3038499
1 1098730
2 983242
3 1948944
4 1878329
5 2710281
6 2691419
7 3046296
8 3038499
\
 
 \ /
  Last update: 2008-04-04 17:03    [W:0.083 / U:1.044 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site