Messages in this thread | | | From | "Liang, Kan" <> | Subject | RE: [PATCH RFC V4 5/6] perf top: switch to backward overwrite mode | Date | Mon, 2 Oct 2017 17:19:41 +0000 |
| |
> On Fri, Sep 29, 2017 at 07:47:56AM -0700, kan.liang@intel.com wrote: > > From: Kan Liang <kan.liang@intel.com> > > > > perf_top__mmap_read has severe performance issue in Knights > > Landing/Mill, when monitoring in heavy load system. It costs several > > minutes to finish, which is unacceptable. > > > > perf top was overwrite mode. But it is changed to non overwrite mode > > since commit 93fc64f14472 ("perf top: Switch to non overwrite mode"). > > For non overwrite mode, it tries to read everything in the ring buffer > > and does not check the messup. Once there are lots of samples > > delivered shortly, the processing time could be very long. > > Knights Landing/Mill as a manycore processor contains a large number > > of small cores. Because of the huge core number, it will generated > > lots of samples in a heavy load system. Also, since the huge sample#, > > the mmap writer probably bite the tail and mess up the samples. > > > > Also, to avoid the problems which is described in commit 9ecda41acb97 > > ("perf/core: Add ::write_backward attribute to perf event"), switch to > > backward overwrite mode. > > Pausing the ring-buffer during perf_top__mmap_read to ensure the > > ring-buffer is stable. > > There would be some records lost in backward overwrite mode. Removing > > the lost events checking. > > I'm getting perf top hogging the cpu completely with this change >
I think I find the root cause of the cpu hogging. perf_mmap__read_catchup discards the md->prev from previous mmap_read. Current mmap_read doesn't know which data has already been processed by previous mmap_read. So it has to go through all the valid data in the ring buffer, even most of the data has been processed by previous mmap_read.
Also, it looks perf record has the similar issue. The previous location will be discarded as well in backward overwrite mode. That will be an issue when --overwrite and --switch-output are enabled. The new output will always include the old data in the previous output, which should be wrong.
I think I will rewrite the perf_mmap__read_backward and perf_mmap__read_catchup to fix this issue in a separate thread. Those functions should be common backward mmap_read functions for all tools and tests.
BTW, are you OK with patch 1-4? Those patches multithreading the machine__synthesize_threads, which is irrelevant with the overwrite mode. I think they can be merged separately.
Thanks, Kan
| |