lkml.org 
[lkml]   [2004]   [May]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: How can I optimize a process on a NUMA architecture(x86-64 specifically)?
Andrew Theurer wrote:

> On Sunday 23 May 2004 09:28, Andi Kleen wrote:
>
>>On Sat, May 22, 2004 at 05:28:09PM -0700, Bryan O'Sullivan wrote:
>>
>>>On Fri, 2004-05-21 at 16:42, Brett E. wrote:
>>>
>>>>Right now, 5 processes are running taking up a good deal of the CPU
>>>>doing memory-intensive work(cacheing) and I notice that none of the
>>>>processes seem to have CPU affinity.
>>>
>>>I don't know what kind of system you're running on, but if it's a
>>>multi-CPU Opteron, it is normally a sufficient fudge to just use
>>>sched_setaffinity to bind individual processes to specific CPUs. The
>>>mainline kernel memory allocator does the right thing in that case, and
>>>allocates memory locally when it can.
>>>
>>>You can use the taskset command to get at this from the command line, so
>>>you may not even need to modify your code.
>>
>>Linus also merged the NUMA API support into mainline now with 2.6.7rc1, so
>>you can use numactl for more finegrained tuning.
>
>
> FYI Brett, some Opteron systems have a BIOS option to interleave memory. If
> you are going to make use of NUMA, I think you want to not interleave.
Thanks for the heads up, I just disabled interleaving in the BIOS.

>
> Also, if you have a 25% imbalance within a domain/node, the scheduler can have
> a tendency to bounce around a task for fairness. That might be why you are
> seeing little/no affinity to a cpu (even top might be causing some of this).
> Not sure what the threshold is between domains/nodes, but I am curious if it
> still happens with CONFIG_NUMA on. If these are long lived cpu bound
> processes, I would try to have the number of processes be a multiple of the
> number of cpus.
I have CONFIG_NUMA on and yes these are long-lived processes(duration is
1 hour). And you and I are on the same wavelength, I'm imagining 3
processes per CPU with apache handing off work to the processes in
question. This is on a 1.6ghz 2-way opteron if that matters at all.
Hopefully I can make the modifications and test this tomorrow.



Thanks,

Brett

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 14:03    [W:0.033 / U:0.048 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site