Messages in this thread | | | Date | Mon, 2 Oct 2017 15:03:46 -0300 | From | "" <> | Subject | Re: [PATCH RFC V4 5/6] perf top: switch to backward overwrite mode |
| |
Em Mon, Oct 02, 2017 at 05:19:41PM +0000, Liang, Kan escreveu: > > > > On Fri, Sep 29, 2017 at 07:47:56AM -0700, kan.liang@intel.com wrote: > > > From: Kan Liang <kan.liang@intel.com> > > > > > > perf_top__mmap_read has severe performance issue in Knights > > > Landing/Mill, when monitoring in heavy load system. It costs several > > > minutes to finish, which is unacceptable. > > > > > > perf top was overwrite mode. But it is changed to non overwrite mode > > > since commit 93fc64f14472 ("perf top: Switch to non overwrite mode"). > > > For non overwrite mode, it tries to read everything in the ring buffer > > > and does not check the messup. Once there are lots of samples > > > delivered shortly, the processing time could be very long. > > > Knights Landing/Mill as a manycore processor contains a large number > > > of small cores. Because of the huge core number, it will generated > > > lots of samples in a heavy load system. Also, since the huge sample#, > > > the mmap writer probably bite the tail and mess up the samples. > > > > > > Also, to avoid the problems which is described in commit 9ecda41acb97 > > > ("perf/core: Add ::write_backward attribute to perf event"), switch to > > > backward overwrite mode. > > > Pausing the ring-buffer during perf_top__mmap_read to ensure the > > > ring-buffer is stable. > > > There would be some records lost in backward overwrite mode. Removing > > > the lost events checking. > > > > I'm getting perf top hogging the cpu completely with this change > > > > I think I find the root cause of the cpu hogging. > perf_mmap__read_catchup discards the md->prev from previous mmap_read. > Current mmap_read doesn't know which data has already been processed by > previous mmap_read. So it has to go through all the valid data in the ring buffer, > even most of the data has been processed by previous mmap_read. > > Also, it looks perf record has the similar issue. > The previous location will be discarded as well in backward overwrite mode. > That will be an issue when --overwrite and --switch-output are enabled. > The new output will always include the old data in the previous output, which > should be wrong. > > I think I will rewrite the perf_mmap__read_backward and perf_mmap__read_catchup > to fix this issue in a separate thread. Those functions should be common backward > mmap_read functions for all tools and tests. > > BTW, are you OK with patch 1-4? > Those patches multithreading the machine__synthesize_threads, which is > irrelevant with the overwrite mode. > I think they can be merged separately.
I'll try and process 1-4 while you work on the read_backward case.
- Arnaldo
| |