Messages in this thread | | | Date | Sun, 17 Feb 2019 16:52:35 -0500 | From | Rich Felker <> | Subject | Re: Regression in SYS_membarrier expedited |
| |
On Sun, Feb 17, 2019 at 04:34:45PM -0500, Mathieu Desnoyers wrote: > ----- On Feb 17, 2019, at 1:48 PM, Rich Felker dalias@libc.org wrote: > > > commit a961e40917fb14614d368d8bc9782ca4d6a8cd11 made it so that the > > MEMBARRIER_CMD_PRIVATE_EXPEDITED command cannot be used without first > > registering intent to use it. However, registration is an expensive > > operation since commit 3ccfebedd8cf54e291c809c838d8ad5cc00f5688, which > > added synchronize_sched() to it; this means it's no longer possible to > > lazily register intent at first use, and it's unreasonably expensive > > to preemptively register intent for possibly extremely-short-lived > > processes that will never use it. (My usage case is in libc (musl), > > where I can't know if the process will be short- or long-lived; > > unnecessary and potentially expensive syscalls can't be made > > preemptively, only lazily at first use.) > > > > Can we restore the functionality of MEMBARRIER_CMD_PRIVATE_EXPEDITED > > to work even without registration? The motivation of requiring > > registration seems to be: > > > > "Registering at this time removes the need to interrupt each and > > every thread in that process at the first expedited > > sys_membarrier() system call." > > > > but interrupting every thread in the process is exactly what I expect, > > and is not a problem. What does seem like a big problem is waiting for > > synchronize_sched() to synchronize with an unboundedly large number of > > cores (vs only a few threads in the process), especially in the > > presence of full_nohz, where it seems like latency would be at least a > > few ms and possibly unbounded. > > > > Short of a working SYS_membarrier that doesn't require expensive > > pre-registration, I'm stuck just implementing it in userspace with > > signals... > > Hi Rich, > > Let me try to understand the scenario first. > > musl libc support for using membarrier private expedited > would require to first register membarrier private expedited for > the process at musl library init (typically after exec). At that stage, the > process is still single-threaded, right ? So there is no reason > to issue a synchronize_sched() (or now synchronize_rcu() in newer > kernels): > > membarrier_register_private_expedited() > > if (!(atomic_read(&mm->mm_users) == 1 && get_nr_threads(p) == 1)) { > /* > * Ensure all future scheduler executions will observe the > * new thread flag state for this process. > */ > synchronize_rcu(); > } > > So considering that pre-registration carefully done before the process > becomes multi-threaded just costs a system call (and not a synchronize_sched()), > does it make the pre-registration approach more acceptable ?
It does get rid of the extreme cost, but I don't think it would be well-received by users who don't like random unnecessary syscalls at init time (each adding a few us of startup time cost). If it's so cheap, why isn't it just the default at kernel-side process creation? Why is there any requirement of registration to begin with? Reading the code, it looks like all it does is set a flag, and all this flag is used for is erroring-out if it's not set.
Rich
| |