Messages in this thread Patch in this message | | | Date | Wed, 21 Sep 2022 17:50:18 +0800 (GMT+08:00) | From | duoming@zju ... | Subject | Re: [PATCH V2] perf/x86/rapl: fix deadlock in rapl_pmu_event_stop |
| |
Hello,
On Wed, 21 Sep 2022 09:53:04 +0200 Ingo Molnar wrote:
> * Duoming Zhou <duoming@zju.edu.cn> wrote: > > > There is a deadlock in rapl_pmu_event_stop(), the process is > > shown below: > > > > (thread 1) | (thread 2) > > rapl_pmu_event_stop() | rapl_hrtimer_handle() > > ... | if (!pmu->n_active) > > raw_spin_lock_irqsave() //(1) | ... > > ... | > > hrtimer_cancel() | raw_spin_lock_irqsave() //(2) > > (block forever) > > > > We hold pmu->lock in position (1) and use hrtimer_cancel() to wait > > rapl_hrtimer_handle() to stop, but rapl_hrtimer_handle() also need > > pmu->lock in position (2). As a result, the rapl_pmu_event_stop() > > will be blocked forever. > > > > This patch extracts hrtimer_cancel() from the protection of > > raw_spin_lock_irqsave(). As a result, the rapl_hrtimer_handle() > > could obtain the pmu->lock. > > > > Fixes: 65661f96d3b3 ("perf/x86: Add RAPL hrtimer support") > > Signed-off-by: Duoming Zhou <duoming@zju.edu.cn> > > --- > > Changes in v2: > > - Move hrtimer_cancel() to the end of rapl_pmu_event_stop() function. > > > > arch/x86/events/rapl.c | 7 +++++-- > > 1 file changed, 5 insertions(+), 2 deletions(-) > > > > diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c > > index 77e3a47af5a..7c110092c83 100644 > > --- a/arch/x86/events/rapl.c > > +++ b/arch/x86/events/rapl.c > > @@ -281,8 +281,6 @@ static void rapl_pmu_event_stop(struct perf_event *event, int mode) > > if (!(hwc->state & PERF_HES_STOPPED)) { > > WARN_ON_ONCE(pmu->n_active <= 0); > > pmu->n_active--; > > - if (pmu->n_active == 0) > > - hrtimer_cancel(&pmu->hrtimer); > > > > list_del(&event->active_entry); > > > > @@ -300,6 +298,11 @@ static void rapl_pmu_event_stop(struct perf_event *event, int mode) > > hwc->state |= PERF_HES_UPTODATE; > > } > > > > + if (!pmu->n_active) { > > + raw_spin_unlock_irqrestore(&pmu->lock, flags); > > + hrtimer_cancel(&pmu->hrtimer); > > + return; > > + } > > Looks racy now: AFAICS now it's possible for rapl_hrtimer_handle() to > execute at an arbitrary moment after pmu->lock is dropped - which could be > use-after-free after cleanup_rapl_pmus() executes and the PMU is freed, > right? > > There's also the quality-of-implementation issue of the hrtimer executing > in a delayed fashion for the *next* event that may have been added, leading > to possibly unexpected results.
Thank your for your suggestions! In order to solve the above problems, I come up with the following solution.
diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c index 77e3a47af5a..a526a08ee6e 100644 --- a/arch/x86/events/rapl.c +++ b/arch/x86/events/rapl.c @@ -219,11 +219,13 @@ static enum hrtimer_restart rapl_hrtimer_handle(struct hrtimer *hrtimer) struct perf_event *event; unsigned long flags;
- if (!pmu->n_active) - return HRTIMER_NORESTART; - raw_spin_lock_irqsave(&pmu->lock, flags);
+ if (!pmu->n_active) { + raw_spin_unlock_irqrestore(&pmu->lock, flags); + return HRTIMER_NORESTART; + } + list_for_each_entry(event, &pmu->active_list, active_entry) rapl_event_update(event);
@@ -282,7 +284,7 @@ static void rapl_pmu_event_stop(struct perf_event *event, int mode) WARN_ON_ONCE(pmu->n_active <= 0); pmu->n_active--; if (pmu->n_active == 0) - hrtimer_cancel(&pmu->hrtimer); + hrtimer_try_to_cancel(&pmu->hrtimer);
list_del(&event->active_entry);
Firstly, the deadlock could be mitigated. Because if the timer callback function is running, the hrtimer_try_to_cancel() will directly return. Secondly, the race could be avoided. Because we use pmu->lock to synchronize and move the check "if (!pmu->n_active)" into the protection scope of pmu->lock. If the rapl_pmu_event_stop() has finished, the "pmu->n_active" equals to 0 and the rapl_hrtimer_handle() will return "HRTIMER_NORESTART". Thirdly, this solution will not cause quality-of-implementation issue of the hrtimer.
Best regards, Duoming Zhou
| |