Messages in this thread | | | Date | Wed, 18 May 2022 11:03:24 +0800 | Subject | Re: DAMON VA regions don't split on an large Android APP | From | Rongwei Wang <> |
| |
On 5/17/22 7:14 PM, Barry Song wrote: > On Tue, May 17, 2022 at 3:00 AM Rongwei Wang > <rongwei.wang@linux.alibaba.com> wrote: >> >> >> >> On 5/16/22 3:03 PM, Barry Song wrote: >>> On Thu, Apr 28, 2022 at 7:37 PM Barry Song <21cnbao@gmail.com> wrote: >>>> >>>> On Thu, Apr 28, 2022 at 2:05 PM Rongwei Wang >>>> <rongwei.wang@linux.alibaba.com> wrote: >>>>> >>>>> >>>>> >>>>> On 4/27/22 5:22 PM, Barry Song wrote: >>>>>> On Wed, Apr 27, 2022 at 7:44 PM Barry Song <21cnbao@gmail.com> wrote: >>>>>>> >>>>>>> On Wed, Apr 27, 2022 at 6:56 PM Rongwei Wang >>>>>>> <rongwei.wang@linux.alibaba.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 4/27/22 7:19 AM, Barry Song wrote: >>>>>>>>> Hi SeongJae & Andrew, >>>>>>>>> (also Cc-ed main damon developers) >>>>>>>>> On an Android phone, I tried to use the DAMON vaddr monitor and found >>>>>>>>> that vaddr regions don't split well on large Android Apps though >>>>>>>>> everything works well on native Apps. >>>>>>>>> >>>>>>>>> I have tried the below two cases on an Android phone with 12GB memory >>>>>>>>> and snapdragon 888 CPU. >>>>>>>>> 1. a native program with small memory working set as below, >>>>>>>>> #define size (1024*1024*100) >>>>>>>>> main() >>>>>>>>> { >>>>>>>>> volatile int *p = malloc(size); >>>>>>>>> memset(p, 0x55, size); >>>>>>>>> >>>>>>>>> while(1) { >>>>>>>>> int i; >>>>>>>>> for (i = 0; i < size / 4; i++) >>>>>>>>> (void)*(p + i); >>>>>>>>> usleep(1000); >>>>>>>>> >>>>>>>>> for (i = 0; i < size / 16; i++) >>>>>>>>> (void)*(p + i); >>>>>>>>> usleep(1000); >>>>>>>>> >>>>>>>>> } >>>>>>>>> } >>>>>>>>> For this application, the Damon vaddr monitor works very well. >>>>>>>>> I have modified monitor.py in the damo userspace tool a little bit to >>>>>>>>> show the raw data getting from the kernel. >>>>>>>>> Regions can split decently on this kind of applications, a typical raw >>>>>>>>> data is as below, >>>>>>>>> >>>>>>>>> monitoring_start: 2.224 s >>>>>>>>> monitoring_end: 2.329 s >>>>>>>>> monitoring_duration: 104.336 ms >>>>>>>>> target_id: 0 >>>>>>>>> nr_regions: 24 >>>>>>>>> 005fb37b2000-005fb734a000( 59.594 MiB): 0 >>>>>>>>> 005fb734a000-005fbaf95000( 60.293 MiB): 0 >>>>>>>>> 005fbaf95000-005fbec0b000( 60.461 MiB): 0 >>>>>>>>> 005fbec0b000-005fc2910000( 61.020 MiB): 0 >>>>>>>>> 005fc2910000-005fc6769000( 62.348 MiB): 0 >>>>>>>>> 005fc6769000-005fca33f000( 59.836 MiB): 0 >>>>>>>>> 005fca33f000-005fcdc8b000( 57.297 MiB): 0 >>>>>>>>> 005fcdc8b000-005fd115a000( 52.809 MiB): 0 >>>>>>>>> 005fd115a000-005fd45bd000( 52.387 MiB): 0 >>>>>>>>> 007661c59000-007661ee4000( 2.543 MiB): 2 >>>>>>>>> 007661ee4000-0076623e4000( 5.000 MiB): 3 >>>>>>>>> 0076623e4000-007662837000( 4.324 MiB): 2 >>>>>>>>> 007662837000-0076630f1000( 8.727 MiB): 3 >>>>>>>>> 0076630f1000-007663494000( 3.637 MiB): 2 >>>>>>>>> 007663494000-007663753000( 2.746 MiB): 1 >>>>>>>>> 007663753000-007664251000( 10.992 MiB): 3 >>>>>>>>> 007664251000-0076666fd000( 36.672 MiB): 2 >>>>>>>>> 0076666fd000-007666e73000( 7.461 MiB): 1 >>>>>>>>> 007666e73000-007667c89000( 14.086 MiB): 2 >>>>>>>>> 007667c89000-007667f97000( 3.055 MiB): 0 >>>>>>>>> 007667f97000-007668112000( 1.480 MiB): 1 >>>>>>>>> 007668112000-00766820f000(1012.000 KiB): 0 >>>>>>>>> 007ff27b7000-007ff27d6000( 124.000 KiB): 0 >>>>>>>>> 007ff27d6000-007ff27d8000( 8.000 KiB): 8 >>>>>>>>> >>>>>>>>> 2. a large Android app like Asphalt 9 >>>>>>>>> For this case, basically regions can't split very well, but monitor >>>>>>>>> works on small vma: >>>>>>>>> >>>>>>>>> monitoring_start: 2.220 s >>>>>>>>> monitoring_end: 2.318 s >>>>>>>>> monitoring_duration: 98.576 ms >>>>>>>>> target_id: 0 >>>>>>>>> nr_regions: 15 >>>>>>>>> 000012c00000-0001c301e000( 6.754 GiB): 0 >>>>>>>>> 0001c301e000-000371b6c000( 6.730 GiB): 0 >>>>>>>>> 000371b6c000-000400000000( 2.223 GiB): 0 >>>>>>>>> 005c6759d000-005c675a2000( 20.000 KiB): 0 >>>>>>>>> 005c675a2000-005c675a3000( 4.000 KiB): 3 >>>>>>>>> 005c675a3000-005c675a7000( 16.000 KiB): 0 >>>>>>>>> 0072f1e14000-0074928d4000( 6.510 GiB): 0 >>>>>>>>> 0074928d4000-00763c71f000( 6.655 GiB): 0 >>>>>>>>> 00763c71f000-0077e863e000( 6.687 GiB): 0 >>>>>>>>> 0077e863e000-00798e214000( 6.590 GiB): 0 >>>>>>>>> 00798e214000-007b0e48a000( 6.002 GiB): 0 >>>>>>>>> 007b0e48a000-007c62f00000( 5.323 GiB): 0 >>>>>>>>> 007c62f00000-007defb19000( 6.199 GiB): 0 >>>>>>>>> 007defb19000-007f794ef000( 6.150 GiB): 0 >>>>>>>>> 007f794ef000-007fe8f53000( 1.745 GiB): 0 >>>>>>>>> >>>>>>>>> As you can see, we have some regions which are very very big and they >>>>>>>>> are losing the chance to be splitted. But >>>>>>>>> Damon can still monitor memory access for those small VMA areas very well like: >>>>>>>>> 005c675a2000-005c675a3000( 4.000 KiB): 3 >>>>>>>> Hi, Barry >>>>>>>> >>>>>>>> Actually, we also had found the same problem in redis by ourselves >>>>>>>> tool[1]. The DAMON can not split the large anon VMA well, and the anon >>>>>>>> VMA has 10G~20G memory. I guess the whole region doesn't have sufficient >>>>>>>> hot areas to been monitored or found by DAMON, likes one or more address >>>>>>>> choose by DAMON not been accessed during sample period. >>>>>>> >>>>>>> Hi Rongwei, >>>>>>> Thanks for your comments and thanks for sharing your tools. >>>>>>> >>>>>>> I guess the cause might be: >>>>>>> in case a region is very big like 10GiB, we have only 1MiB hot pages >>>>>>> in this large region. >>>>>>> damon will randomly pick one page to sample, but the page has only >>>>>>> 1MiB/10GiB, thus >>>>>>> less than 1/10000 chance to hit the hot 1MiB. so probably we need >>>>>>> 10000 sample periods >>>>>>> to hit the hot 1MiB in order to split this large region? >>>>>>> >>>>>>> @SeongJae, please correct me if I am wrong. >>>>>>> >>>>>>>> >>>>>>>> I'm not sure whether sets init_regions can deal with the above problem, >>>>>>>> or dynamic choose one or limited number VMA to monitor. >>>>>>>> >>>>>>> >>>>>>> I won't set a limited number of VMA as this will make the damon too hard to use >>>>>>> as nobody wants to make such complex operations, especially an Android >>>>>>> app might have more than 8000 VMAs. >>>>>>> >>>>>>> I agree init_regions might be the right place to enhance the situation. >>>>>>> >>>>>>>> I'm not sure, just share my idea. >>>>>>>> >>>>>>>> [1] https://github.com/aliyun/data-profile-tools.git >>>>>>> >>>>>>> I suppose this tool is based on damon? How do you finally resolve the problem >>>>>>> that large anon VMAs can't be splitted? >>>>>>> Anyway, I will give your tool a try. >>>>>> >>>>>> Unfortunately, data-profile-tools.git doesn't build on aarch64 ubuntu >>>>>> though autogen.sh >>>>>> runs successfully. >>>>>> >>>>>> /usr/bin/ld: ./.libs/libdatop.a(disp.o): in function `cons_handler': >>>>>> /root/data-profile-tools/src/disp.c:625: undefined reference to `stdscr' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/disp.c:625: undefined >>>>>> reference to `stdscr' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/disp.c:625: undefined >>>>>> reference to `wgetch' >>>>>> /usr/bin/ld: ./.libs/libdatop.a(reg.o): in function `reg_win_create': >>>>>> /root/data-profile-tools/src/reg.c:108: undefined reference to `stdscr' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:108: undefined >>>>>> reference to `stdscr' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:108: undefined >>>>>> reference to `subwin' >>>>>> /usr/bin/ld: ./.libs/libdatop.a(reg.o): in function `reg_erase': >>>>>> /root/data-profile-tools/src/reg.c:161: undefined reference to `werase' >>>>>> /usr/bin/ld: ./.libs/libdatop.a(reg.o): in function `reg_refresh': >>>>>> /root/data-profile-tools/src/reg.c:171: undefined reference to `wrefresh' >>>>>> /usr/bin/ld: ./.libs/libdatop.a(reg.o): in function `reg_refresh_nout': >>>>>> /root/data-profile-tools/src/reg.c:182: undefined reference to `wnoutrefresh' >>>>>> /usr/bin/ld: ./.libs/libdatop.a(reg.o): in function `reg_update_all': >>>>>> /root/data-profile-tools/src/reg.c:191: undefined reference to `doupdate' >>>>>> /usr/bin/ld: ./.libs/libdatop.a(reg.o): in function `reg_win_destroy': >>>>>> /root/data-profile-tools/src/reg.c:200: undefined reference to `delwin' >>>>>> /usr/bin/ld: ./.libs/libdatop.a(reg.o): in function `reg_line_write': >>>>>> /root/data-profile-tools/src/reg.c:226: undefined reference to `mvwprintw' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:230: undefined >>>>>> reference to `wattr_off' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:217: undefined >>>>>> reference to `wattr_on' >>>>>> /usr/bin/ld: ./.libs/libdatop.a(reg.o): in function `reg_highlight_write': >>>>>> /root/data-profile-tools/src/reg.c:245: undefined reference to `wattr_on' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:255: undefined >>>>>> reference to `wattr_off' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:252: undefined >>>>>> reference to `mvwprintw' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:255: undefined >>>>>> reference to `wattr_off' >>>>>> /usr/bin/ld: ./.libs/libdatop.a(reg.o): in function `reg_curses_fini': >>>>>> /root/data-profile-tools/src/reg.c:367: undefined reference to `stdscr' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:367: undefined >>>>>> reference to `stdscr' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:367: undefined >>>>>> reference to `wclear' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:368: undefined >>>>>> reference to `wrefresh' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:369: undefined >>>>>> reference to `endwin' >>>>>> /usr/bin/ld: ./.libs/libdatop.a(reg.o): in function `reg_curses_init': >>>>>> /root/data-profile-tools/src/reg.c:382: undefined reference to `stdscr' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:381: undefined >>>>>> reference to `initscr' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:382: undefined >>>>>> reference to `stdscr' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:382: undefined >>>>>> reference to `wrefresh' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:383: undefined >>>>>> reference to `use_default_colors' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:384: undefined >>>>>> reference to `start_color' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:385: undefined >>>>>> reference to `keypad' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:386: undefined >>>>>> reference to `nonl' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:387: undefined >>>>>> reference to `cbreak' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:388: undefined >>>>>> reference to `noecho' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:389: undefined >>>>>> reference to `curs_set' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:401: undefined >>>>>> reference to `stdscr' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:401: undefined >>>>>> reference to `mvwprintw' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:403: undefined >>>>>> reference to `mvwprintw' >>>>>> /usr/bin/ld: /root/data-profile-tools/src/reg.c:405: undefined >>>>>> reference to `wrefresh' >>>>>> collect2: error: ld returned 1 exit status >>>>>> make[1]: *** [Makefile:592: datop] Error 1 >>>>>> make[1]: Leaving directory '/root/data-profile-tools' >>>>>> make: *** [Makefile:438: all] Error 2 >>>>> Hi, Barry >>>>> >>>>> Now, the question made me realize that the compatibility of this tool is >>>>> very poor. I built a ubuntu environment at yesterday, and fixed above >>>>> errors by: >>>>> >>>>> diff --git a/configure.ac b/configure.ac >>>>> index 7922f27..1ed823c 100644 >>>>> --- a/configure.ac >>>>> +++ b/configure.ac >>>>> @@ -21,13 +21,9 @@ AC_PROG_INSTALL >>>>> AC_CHECK_LIB([numa], [numa_free]) >>>>> AC_CHECK_LIB([pthread], [pthread_create]) >>>>> >>>>> -PKG_CHECK_MODULES([CHECK], [check]) >>>>> - >>>>> -PKG_CHECK_MODULES([NCURSES], [ncursesw ncurses], [LIBS="$LIBS >>>>> $ncurses_LIBS"], [ >>>>> - AC_SEARCH_LIBS([delwin], [ncursesw ncurses], [], [ >>>>> - AC_MSG_ERROR([ncurses is required but was not found]) >>>>> - ], []) >>>>> -]) >>>>> +AC_SEARCH_LIBS([stdscr], [ncurses ncursesw], [], [ >>>>> + AC_MSG_ERROR([required library libncurses or ncurses not found]) >>>>> + ]) >>>>> >>>> >>>> I can confirm the patch fixed the issue I reported yesterday, thanks! >>>> >>>>> It works. But I found an another thing will hinder you using this tool. >>>>> We had developed other patches about DAMON base on upstream. This tool >>>>> only works well in ourselves kernel(anolis kernel, already open source). >>>>> Of course, I think it's unnecessary for you to change kernel, just let >>>>> you know this tool still has this problem. >>>>> >>>> >>>> Although I can't use this tool directly as I am not a NUMA right now, >>>> ~/data-profile-tools # ./datop --help >>>> Not support NUMA fault stat (DAMON)! >>>> >>> >>> I wonder if you can extend it to non-numa by setting "remote" to 0% >>> and local to "100%" always for non-numa machines rather than death. >> Hi Barry >> >> That's a great suggestion. Actually, I have removed 'numa_stat' check in >> datop. Maybe you can found. It does not enable numa stat when >> 'numa_stat' sysfs not found in the current system. > > yep. i am able to run it on a non-numa machine, but datop immediately crashes > due to some memory corruption issues: > > Monitoring 270 processes (interval: 5.5s) Barry, it's known bug. I remember the maximum number of processes that is 32 in datop. The reason that setting like this is that I feel impossible to monitor so many processes at the beginning.
And it seems that the error message should been printed here, instead of crash. Thank you for reminding me. > > PID PROC TYPE START END > SIZE(KiB) ACCESS AGE > 1693 Binder:1693 ---- 0 0 > 0 0 0 > 428 ueventd ---- 0 0 > 0 0 0 > 28654 adbd ---- 0 0 > 0 0 0 > 971 usb@1.2-ser ---- 0 0 > 0 0 0 > 619 logd ---- 0 0 > 0 0 0 > 4311 a... > > <- Hotkey for sorting: 1(PID), 2(START), 3(SIZE), 4(ACCESS), 5(RMA) -> > CPU% = system CPU utilization > > Q: Quit; H: Home; B: Back; R: Refresh; D: DAMON > double free or corruption (!prev) > > Aborted > > if i move to monitor only one process, datop doesn't crash but it > doesn't show any > data either: > > # pgrep youtube > 4311 > # ./datop -p 4311 > > Monitoring 1 processes (interval: 5.0s) Oh, it's ever happen to me. Does It always show like this when monitoring one process in your environment? > > PID PROC TYPE START END > SIZE(KiB) *ACCESS AGE > 4311 youtube ---- 0 0 0 > 0 0 > > >> >> What's more, a new hot key 'f' will be introduced which can enable some >> features dynamically, such as numa stat. Others features can be used >> only in our internal version, likes 'f' in top, and will be open source >> when stable. >> >>> as your tools can map regions to .so, which seems to be quite useful. >> enen, I'm agree with you. But you know, one region maybe covers one or >> more VMAs, hard to map access count of regions to the related .so or >> anon. A lazy way used by me now. I still think it's valuable in the future. >> > > it seems really an interesting topic worth more investigation. I wonder if > damon vaddr monitor should actually take vmas, or at least the types of > vmas into consideration while splitting. > > Different vma types should be inherently different in hotness. for example, > if 1mb text and 1mb data are put in the same region, the monitored data > to reflect the hotness for the whole 2mb seems to be pointless at all. > > Hi SeongJae, > what do you think about it? > >> Anyway, any idea are welcome. >> >> Thanks, >> -wrw >> >>> >>>> I am still quite interested in your design and the purpose of this project. >>>> Unfortunately the project seems to be lacking some design doc. >>>> >>>> And would you like to send patches to lkml regarding what you >>>> have changed atop DAMON? >>>> >>>>> Anyway, the question that you reported was valuable, made me realize >>>>> what we need to improve next. >>>>> >>>>> Thanks, >>>>> Rongwei Wang >>>>>> >>>>>>> >>>>>>>>> >>>>>>>>> Typical characteristics of a large Android app is that it has >>>>>>>>> thousands of vma and very large virtual address spaces: >>>>>>>>> ~/damo # pmap 2550 | wc -l >>>>>>>>> 8522 >>>>>>>>> >>>>>>>>> ~/damo # pmap 2550 >>>>>>>>> ... >>>>>>>>> 0000007992bbe000 4K r---- [ anon ] >>>>>>>>> 0000007992bbf000 24K rw--- [ anon ] >>>>>>>>> 0000007fe8753000 4K ----- [ anon ] >>>>>>>>> 0000007fe8754000 8188K rw--- [ stack ] >>>>>>>>> total 36742112K >>>>>>>>> >>>>>>>>> Because the whole vma list is too long, I have put the list here for >>>>>>>>> you to download: >>>>>>>>> wget http://www.linuxep.com/patches/android-app-vmas >>>>>>>>> >>>>>>>>> I can reproduce this problem on other Apps like youtube as well. >>>>>>>>> I suppose we need to boost the algorithm of splitting regions for this >>>>>>>>> kind of application. >>>>>>>>> Any thoughts? >>>>>>>>> >>>>>>> >>> > > Thanks > Barry
| |