lkml.org 
[lkml]   [2019]   [May]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework
    From
    Date
    On Fri, 2019-05-10 at 10:12 +0200, Daniel Vetter wrote:
    > On Fri, May 10, 2019 at 7:49 AM Knut Omang <knut.omang@oracle.com> wrote:
    > >
    > > On Thu, 2019-05-09 at 22:18 -0700, Frank Rowand wrote:
    > > > On 5/9/19 4:40 PM, Logan Gunthorpe wrote:
    > > > >
    > > > >
    > > > > On 2019-05-09 5:30 p.m., Theodore Ts'o wrote:
    > > > >> On Thu, May 09, 2019 at 04:20:05PM -0600, Logan Gunthorpe wrote:
    > > > >>>
    > > > >>> The second item, arguably, does have significant overlap with kselftest.
    > > > >>> Whether you are running short tests in a light weight UML environment or
    > > > >>> higher level tests in an heavier VM the two could be using the same
    > > > >>> framework for writing or defining in-kernel tests. It *may* also be valuable
    > > > >>> for some people to be able to run all the UML tests in the heavy VM
    > > > >>> environment along side other higher level tests.
    > > > >>>
    > > > >>> Looking at the selftests tree in the repo, we already have similar items to
    > > > >>> what Kunit is adding as I described in point (2) above. kselftest_harness.h
    > > > >>> contains macros like EXPECT_* and ASSERT_* with very similar intentions to
    > > > >>> the new KUNIT_EXECPT_* and KUNIT_ASSERT_* macros.
    > > > >>>
    > > > >>> However, the number of users of this harness appears to be quite small. Most
    > > > >>> of the code in the selftests tree seems to be a random mismash of scripts
    > > > >>> and userspace code so it's not hard to see it as something completely
    > > > >>> different from the new Kunit:
    > > > >>>
    > > > >>> $ git grep --files-with-matches kselftest_harness.h *
    > > > >>
    > > > >> To the extent that we can unify how tests are written, I agree that
    > > > >> this would be a good thing. However, you should note that
    > > > >> kselftest_harness.h is currently assums that it will be included in
    > > > >> userspace programs. This is most obviously seen if you look closely
    > > > >> at the functions defined in the header files which makes calls to
    > > > >> fork(), abort() and fprintf().
    > > > >
    > > > > Ah, yes. I obviously did not dig deep enough. Using kunit for
    > > > > in-kernel tests and kselftest_harness for userspace tests seems like
    > > > > a sensible line to draw to me. Trying to unify kernel and userspace
    > > > > here sounds like it could be difficult so it's probably not worth
    > > > > forcing the issue unless someone wants to do some really fancy work
    > > > > to get it done.
    > > > >
    > > > > Based on some of the other commenters, I was under the impression
    > > > > that kselftests had in-kernel tests but I'm not sure where or if they
    > > > > exist.
    > > >
    > > > YES, kselftest has in-kernel tests. (Excuse the shouting...)
    > > >
    > > > Here is a likely list of them in the kernel source tree:
    > > >
    > > > $ grep module_init lib/test_*.c
    > > > lib/test_bitfield.c:module_init(test_bitfields)
    > > > lib/test_bitmap.c:module_init(test_bitmap_init);
    > > > lib/test_bpf.c:module_init(test_bpf_init);
    > > > lib/test_debug_virtual.c:module_init(test_debug_virtual_init);
    > > > lib/test_firmware.c:module_init(test_firmware_init);
    > > > lib/test_hash.c:module_init(test_hash_init); /* Does everything */
    > > > lib/test_hexdump.c:module_init(test_hexdump_init);
    > > > lib/test_ida.c:module_init(ida_checks);
    > > > lib/test_kasan.c:module_init(kmalloc_tests_init);
    > > > lib/test_list_sort.c:module_init(list_sort_test);
    > > > lib/test_memcat_p.c:module_init(test_memcat_p_init);
    > > > lib/test_module.c:static int __init test_module_init(void)
    > > > lib/test_module.c:module_init(test_module_init);
    > > > lib/test_objagg.c:module_init(test_objagg_init);
    > > > lib/test_overflow.c:static int __init test_module_init(void)
    > > > lib/test_overflow.c:module_init(test_module_init);
    > > > lib/test_parman.c:module_init(test_parman_init);
    > > > lib/test_printf.c:module_init(test_printf_init);
    > > > lib/test_rhashtable.c:module_init(test_rht_init);
    > > > lib/test_siphash.c:module_init(siphash_test_init);
    > > > lib/test_sort.c:module_init(test_sort_init);
    > > > lib/test_stackinit.c:module_init(test_stackinit_init);
    > > > lib/test_static_key_base.c:module_init(test_static_key_base_init);
    > > > lib/test_static_keys.c:module_init(test_static_key_init);
    > > > lib/test_string.c:module_init(string_selftest_init);
    > > > lib/test_ubsan.c:module_init(test_ubsan_init);
    > > > lib/test_user_copy.c:module_init(test_user_copy_init);
    > > > lib/test_uuid.c:module_init(test_uuid_init);
    > > > lib/test_vmalloc.c:module_init(vmalloc_test_init)
    > > > lib/test_xarray.c:module_init(xarray_checks);
    > > >
    > > >
    > > > > If they do exists, it seems like it would make sense to
    > > > > convert those to kunit and have Kunit tests run-able in a VM or
    > > > > baremetal instance.
    > > >
    > > > They already run in a VM.
    > > >
    > > > They already run on bare metal.
    > > >
    > > > They already run in UML.
    > > >
    > > > This is not to say that KUnit does not make sense. But I'm still trying
    > > > to get a better description of the KUnit features (and there are
    > > > some).
    > >
    > > FYI, I have a master student who looks at converting some of these to KTF, such as for
    > > instance the XArray tests, which lended themselves quite good to a semi-automated
    > > conversion.
    > >
    > > The result is also a somewhat more compact code as well as the flexibility
    > > provided by the Googletest executor and the KTF frameworks, such as running selected
    > > tests, output formatting, debugging features etc.
    >
    > So is KTF already in upstream?

    No..

    > Or is the plan to unify the KTF and
    > Kunit in-kernel test harnesses?

    Since KTF delegates reporting and test running to user space - reporting is based on
    netlink communication with the user land frontend (Googletest based, but can in principle
    be ported to any user land framework if need be) - little specific test harness features
    are needed for KTF, so there's little if no direct overlap with the infrastructure in
    kselftests (as far as I understand, I'd like to spend more time on the details there..)

    > Because there's tons of these
    > in-kernel unit tests already, and every merge we get more (Frank's
    > list didn't even look into drivers or anywhere else, e.g. it's missing
    > the locking self tests I worked on in the past), and a more structured
    > approach would really be good.

    That's been my thinking too. Having a unified way to gradually move to would benefit
    anyone who needs to relate to these tests in that there will be less to learn.
    But that would be a long term evolutionary goal of course.

    Also I think each of these test suites/sets would benefit from being more available for
    running even in kernels not made specifically for testing, and I think using the module
    structure and a lean, common module (ktf.ko) as a library for the tests, but no
    "polluting" of the "production" kernel code with anything else, is a kind of a policy that
    comes implicitly with KTF that can make it easier to maintain a sort of standard
    "language" for writing kernel tests.

    Wrt KTF, we're working on making a suitable patch set for the kernel,
    but also have projects running internally that uses KTF as a standalone git repository,
    and that inspires new features and other changes, as well as a growing set of tests.
    I want to make sure we find a good candidate kernel integration that can coexist nicely
    with the separate git repo version without becoming too much of a back/forward porting
    challenge.

    Knut

    > -Daniel

    \
     
     \ /
      Last update: 2019-05-10 13:39    [W:8.221 / U:0.320 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site