Messages in this thread | | | From | Michael Nazzareno Trimarchi <> | Date | Mon, 19 Sep 2022 14:11:36 +0200 | Subject | Re: Correlation CMA size and FORCE_MAX_ZONEORDER |
| |
Hi
On Mon, Sep 19, 2022 at 2:08 PM David Hildenbrand <david@redhat.com> wrote: > > On 19.09.22 13:59, Michael Nazzareno Trimarchi wrote: > > Hi David > > > > > > On Mon, Sep 19, 2022 at 1:28 PM David Hildenbrand <david@redhat.com> wrote: > >> > >> On 19.09.22 13:17, Michael Nazzareno Trimarchi wrote: > >>> Hi > >>> > >>> On Mon, Sep 19, 2022 at 1:03 PM David Hildenbrand <david@redhat.com> wrote: > >>>> > >>>> On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote: > >>>>> Hi > >>>>> > >>>>> On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@redhat.com> wrote: > >>>>>> > >>>>>> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote: > >>>>>>> Hi David > >>>>>>> > >>>>>>> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote: > >>>>>>>> > >>>>>>>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote: > >>>>>>>>> Hi all > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>>> > >>>>>>>>> Working on a small device with 128MB of memory and using imx_v6_v7 > >>>>>>>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE > >>>>>>>>> are not respected. The calculation done does not allow the requested > >>>>>>>>> size. I think that this should be somehow documented and described but > >>>>>>>>> I did not > >>>>>>>>> find the documentation. Does it work this way? > >>>>>>>>> > >>>>>>>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have > >>>>>>>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb > >>>>>>>> > >>>>>>>> The underlying constraint is that CMA regions require a certain minimum > >>>>>>>> alignment+size. They cannot be arbitrarily in size. > >>>>>>>> > >>>>>>>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream > >>>>>>>> kernels to the size of a single pageblock. > >>>>>>>> > >>>>>>>> In previous kernels, it used to be the size of the largest buddy > >>>>>>>> allocation granularity (derived from MAX_ORDER, derived from > >>>>>>>> FORCE_MAX_ZONEORDER). > >>>>>>>> > >>>>>>>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer > >>>>>>>> apply. On most archs, the minimum alignment+size should be 2 MiB > >>>>>>>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock. > >>>>>>>> > >>>>>>>> So far the theory. Are you still running into this limitation on > >>>>>>>> upstream kernels? > >>>>>>>> > >>>>>>> > >>>>>>> I can run 6-rc2 on my board. I test again but according to it, if I > >>>>>>> put 4M as CMA in cma=4M in boot > >>>>>>> parameters, the result is 32Mb of CMA. Apart of that seems that > >>>>>>> process lime tiny membench can not even start > >>>>>>> to mblock memory > >>>>>>> > >>>>>> > >>>>>> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you > > >>>>>> 4M, can you post /proc/meminfo and the early console output? > >>>>>> > >>>>> > >>>>> cat /proc/cmdline > >>>>> cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs) > >>>>> root=ubi0:root rw ubi.mtd=ro > >>>>> otfs rootfstype=ubifs rootwait=1 > >>>>> # cat /proc/meminfo > >>>>> MemTotal: 109560 kB > >>>>> MemFree: 56084 kB > >>>>> MemAvailable: 56820 kB > >>>>> Buffers: 0 kB > >>>>> Cached: 39680 kB > >>>>> SwapCached: 0 kB > >>>>> Active: 44 kB > >>>>> Inactive: 644 kB > >>>>> Active(anon): 44 kB > >>>>> Inactive(anon): 644 kB > >>>>> Active(file): 0 kB > >>>>> Inactive(file): 0 kB > >>>>> Unevictable: 39596 kB > >>>>> Mlocked: 0 kB > >>>>> HighTotal: 0 kB > >>>>> HighFree: 0 kB > >>>>> LowTotal: 109560 kB > >>>>> LowFree: 56084 kB > >>>>> SwapTotal: 0 kB > >>>>> SwapFree: 0 kB > >>>>> Dirty: 0 kB > >>>>> Writeback: 0 kB > >>>>> AnonPages: 628 kB > >>>>> Mapped: 1480 kB > >>>>> Shmem: 84 kB > >>>>> KReclaimable: 4268 kB > >>>>> Slab: 8456 kB > >>>>> SReclaimable: 4268 kB > >>>>> SUnreclaim: 4188 kB > >>>>> KernelStack: 392 kB > >>>>> PageTables: 88 kB > >>>>> NFS_Unstable: 0 kB > >>>>> Bounce: 0 kB > >>>>> WritebackTmp: 0 kB > >>>>> CommitLimit: 54780 kB > >>>>> Committed_AS: 1876 kB > >>>>> VmallocTotal: 901120 kB > >>>>> VmallocUsed: 2776 kB > >>>>> VmallocChunk: 0 kB > >>>>> Percpu: 72 kB > >>>>> CmaTotal: 32768 kB > >>>>> CmaFree: 32484 kB > >>>>> # uname -a > >>>>> Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux > >>>>> # > >>>>> > >>>>> Then here https://pastebin.com/6MUB2VBM dmesg > >>>>> > >>>>> CONFIG_ARM_MODULE_PLTS=y > >>>>> CONFIG_FORCE_MAX_ZONEORDER=14 > >>>>> CONFIG_ALIGNMENT_TRAP=y > >>>>> ... > >>>>> CONFIG_CMA > >>>>> CONFIG_CMA_AREAS=7 > >>>>> ... > >>>>> > >>>>> CONFIG_CMA_SIZE_MBYTES=8 > >>>>> CONFIG_CMA_SIZE_SEL_MBYTES=y > >>>>> > >>>> > >>>> Thanks! > >>>> > >>>> I assume that in your setup, the pageblock size depends on MAX_ORDER > >>>> and, therefore, FORCE_MAX_ZONEORDER. > >>>> > >>>> This should be the case especially if CONFIG_HUGETLB_PAGE is not defined > >>>> (include/linux/pageblock-flags.h). > >>>> > >>>> In contrast to what I remember, the pageblock size does not seem to > >>>> depend on the THP size (weird) as well. > >>>> > >>>> > >>>> So, yes, that limitation is still in effect for some kernel configs. > >>>> > >>>> One could make the pageblock size configurable (similar to > >>>> CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller > >>>> pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE. > >>>> > >>>> I imagine something reasonable might be to set the pageblock size to > >>>> 2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA. > >>>> > >>> > >>> I don't think making more configuration options makes things clear. > >> > >> Yes, in an ideal case it should be automatic. > >> > >>> When we enable some configuration > >>> we can force down the configuration. You need to explain clearly how > >>> you envision it. FORCE_MAX_ZONEORDER > >>> for me is the largest allocation that you can get from a zone (ex CMA > >>> one). Any request allocation that is align to the > >> > >> FORCE_MAX_ZONEORDER is just a way to increase/decrease the maximum > >> allocation size of the buddy in general. > >> > >>> CMA align and can fit inside a region should be allowed > >>> > >>> What am I missing? > >> > >> I think that the issue is that the CMA alignments nowadays depend on the > >> pageblock size. And the pageblock size depends on *some* configurations > >> on the maximum allocation size of the buddy. > >> > >> Documenting the interaction between FORCE_MAX_ZONEORDER and CMA size > >> alignment is not trivial. > >> > >> For example, with CONFIG_HUGETLB_PAGE there might not be such an > >> interaction. With CONFIG_HUGETLB_PAGE, there clearly is one. > >> > >> > >> Let me phrase it this way: is it good enough in you setup to get 32mb vs > >> 8mb or do you want/need to reduce it without adjusting > >> FORCE_MAX_ZONEORDER ? > > > > Wait we have: > > - CMA kconfig alignment that in most config we have not considered > > natural dma alignment but is put to 1Mb in a lot of embedded > > - We have CMA_SIZE, CMA_SIZE_PERCENTAGE etc. Those seems that are not > > effect if ZONEORDER is not reasonable and without > > HUGETLB_PAGE > > - etc > > > > Changing MAX_ZONEORDER is ok and yes if you have an IOT device that > > you know about your CMA allocation, it makes no sense to have > > it 32MB for a 128Mb device. What I point out is that I need to figure > > it out because in Kconfig there is no mention of it. Should it be > > added there? > > Something like > > "Note that in some configurations, the CMA size must be aligned to the > maximum allocation granularity of the buddy allocator, consequently > depending on FORCE_MAX_ZONEORDER. The requested CMA size might get > increased accordingly." > > Maybe we'd want some kind of a warning in the kernel as well. If someone > specifies "cma=2MB" but gets 32MB or more that might be a problem. > > Does something like that make sense to you?
Yes we need to warn the user and add to Kconfig. Maybe wait for other to get a feedback
Michael
> > -- > Thanks, > > David / dhildenb >
-- Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 michael@amarulasolutions.com __________________________________
Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 info@amarulasolutions.com www.amarulasolutions.com
| |