Messages in this thread | | | Date | Mon, 30 Oct 2023 19:32:46 +0800 | Subject | Re: [RFC] mm/kasan: Add Allocation, Free, Error timestamps to KASAN report | From | Juntong Deng <> |
| |
On 2023/10/30 18:10, Dmitry Vyukov wrote: > On Mon, 30 Oct 2023 at 10:28, Juntong Deng <juntong.deng@outlook.com> wrote: >> >> On 2023/10/30 14:29, Dmitry Vyukov wrote: >>> On Sun, 29 Oct 2023 at 10:05, Juntong Deng <juntong.deng@outlook.com> wrote: >>>> >>>> On 2023/10/26 3:22, Andrey Konovalov wrote: >>>>> On Tue, Oct 17, 2023 at 9:40 PM Juntong Deng <juntong.deng@outlook.com> wrote: >>>>>> >>>>>> The idea came from the bug I was fixing recently, >>>>>> 'KASAN: slab-use-after-free Read in tls_encrypt_done'. >>>>>> >>>>>> This bug is caused by subtle race condition, where the data structure >>>>>> is freed early on another CPU, resulting in use-after-free. >>>>>> >>>>>> Like this bug, some of the use-after-free bugs are caused by race >>>>>> condition, but it is not easy to quickly conclude that the cause of the >>>>>> use-after-free is race condition if only looking at the stack trace. >>>>>> >>>>>> I did not think this use-after-free was caused by race condition at the >>>>>> beginning, it took me some time to read the source code carefully and >>>>>> think about it to determine that it was caused by race condition. >>>>>> >>>>>> By adding timestamps for Allocation, Free, and Error to the KASAN >>>>>> report, it will be much easier to determine if use-after-free is >>>>>> caused by race condition. >>>>> >>>>> An alternative would be to add the CPU number to the alloc/free stack >>>>> traces. Something like: >>>>> >>>>> Allocated by task 42 on CPU 2: >>>>> (stack trace) >>>>> >>>>> The bad access stack trace already prints the CPU number. >>>> >>>> Yes, that is a great idea and the CPU number would help a lot. >>>> >>>> But I think the CPU number cannot completely replace the free timestamp, >>>> because some freeing really should be done at another CPU. >>>> >>>> We need the free timestamp to help us distinguish whether it was freed >>>> a long time ago or whether it was caused to be freed during the >>>> current operation. >>>> >>>> I think both the CPU number and the timestamp should be displayed, more >>>> information would help us find the real cause of the error faster. >>>> >>>> Should I implement these features? >>> >>> Hi Juntong, >>> >>> There is also an aspect of memory consumption. KASAN headers increase >>> the size of every heap object. So we tried to keep them as compact as >>> possible. At some point CPU numbers and timestamps (IIRC) were already >>> part of the header, but we removed them to shrink the header to 16 >>> bytes. >>> PID gives a good approximation of potential races. I usually look at >>> PIDs to understand if it's a "plain old single-threaded >>> use-after-free", or free and access happened in different threads. >>> Re timestamps, I see you referenced a syzbot report. With syzkaller >>> most timestamps will be very close even for non-racing case. >>> So if this is added, this should be added at least under a separate config. >>> >>> If you are looking for potential KASAN improvements, here is a good list: >>> https://bugzilla.kernel.org/buglist.cgi?bug_status=__open__&component=Sanitizers&list_id=1134168&product=Memory%20Management >> >> Hi Dmitry, >> >> I think PID cannot completely replace timestamp for reason similar to >> CPU number, some frees really should be done in another thread, but it >> is difficult for us to distinguish if it is a free that was done some >> time ago, or under subtle race conditions. > > I agree it's not a complete replacement, it just does not consume > additional memory. > >> As to whether most of the timestamps will be very close even for >> non-racing case, this I am not sure, because I do not have >> enough samples. >> >> I agree that these features should be in a separate config and >> the user should be free to choose whether to enable them or not. >> >> We can divide KASAN into normal mode and depth mode. Normal mode >> records only minimal critical information, while depth mode records >> more potentially useful information. >> >> Also, honestly, I think a small amount of extra memory consumption >> should not stop us from recording more information. >> >> Because if someone enables KASAN for debugging, then memory consumption >> and performance are no longer his main concern. > > There are a number of debugging tools created with the "performance > does not matter" attitude. They tend to be barely usable, not usable > in wide scale testing, not usable in canaries, etc. > All of sanitizers were created with lots of attention to performance, > attention on the level of the most performance critical production > code (sanitizer code is hotter than any production piece of code). > That's what made them so widely used. Think of interactive uses, > smaller devices, etc. Please let's keep this attitude.
Yes, I agree that debugging tools used at a wide scale need to have more rigorous performance considerations.
Do you think it is worth using the extra bytes to record more information? If this is a user-configurable feature.
| |