Messages in this thread | | | From | "Stephen C. Tweedie" <> | Date | Tue, 7 Mar 2000 15:19:00 +0000 (GMT) | Subject | Re: Help in DSM design |
| |
Hi,
On Mon, 06 Mar 2000 11:36:06 -0800, Aman Singla <aman@cthulhu.engr.sgi.com> said:
> Both add that additional level of locality that a performance > conscious application needs to exploit; agreed that KSR was the only > one to have caching at the remote memory to local memory level (with a > page size granularity).
Yes, and the granularity was a _big_ problem.
> A DSM system could look very much like the current NUMA h/w with an > added remote-local memory caching happening at application specific > granularity. The main issue there is getting access control at the > desired granularity for the s/w DSM layer
Not just that. On SMP, and even more on NUMA, you only get scalable performance by going out of your way to avoid unnecessary sharing of data. It's just not transparent. And when you _do_ want to pass data from X to Y, you only want to do it once, which requires extra synchronisation to make sure that the receiver of the data doesn't start peeking at the data structures until the data is known to arrive.
With message passing, that synchronisation is implicit. With shared memory, about all you can do is spin waiting for the data to arrive, unless you add a separate synchronisation mechanism somewhere to do locking outside the shared memory channel.
It's not just granularity. The synchronisation hit is bad, on _all_ forms of DSM up to and including SMP. (The latest Linux development kernel includes "big reader" locks which allow a read-mostly data structure to be locked on each CPU separately with no inter-CPU memory traffic at all, simply because even on such tightly coupled systems the cost of passing even one cache line of shared memory traffic between CPUs is too high.)
--Stephen
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |