lkml.org 
[lkml]   [2022]   [Nov]   [1]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [igt-dev] [PATCH i-g-t v2 3/4] lib/igt_kmod: add compatibility for KUnit
On Mon, 19 Sep 2022 17:25:38 -0700
Daniel Latypov <dlatypov@google.com> wrote:

> On Mon, Sep 19, 2022 at 1:43 PM Isabella Basso <isabbasso@riseup.net> wrote:
> > >> + * "(K)TAP version XX" should be the first line on all (sub)tests as per
> > >> + * https://www.kernel.org/doc/html/latest/dev-tools/ktap.html#version-lines
> > >> + * but actually isn't, as it currently depends on whoever writes the
> > >> + * test to print this info
> > >
> > > FYI: we're really trying to fix cases of "missing version lines",
> > > largely by making the kunit_test_suites() macro work in more
> > > circumstances.
> > >
> > > So while it may be worth still handling the case where this is
> > > missing, I don't think there are any tests in the latest versions of
> > > the kernel which should have this missing.
> >
> > I’m not sure if I totally get how these work. Every time I run a KUnit test I
> > get something like this: https://pastebin.com/7Ff31PMC
> >
> > As you can see it has been loaded as a module, just like we intend to do it
> > from IGT, and I see no version lines whatsoever. Am I doing something wrong?
>
> You're doing everything right.
>
> The problem is we only print the version line for the *built-in* tests.
> It never gets printed for tests in loadable modules.

IGT tests require loadable modules, as some tests check if bind/unbind
and load/unload works. Also, some tests require modprobing drivers
with different parameters.

Unfortunately, kunit.py is currently broken on such cases: its parser
simply ignores everything if the subtest doesn't have a TAP version
just before it. See:

$ ./tools/testing/kunit/kunit.py parse /var/log/dmesg
[12:37:05] ============================================================
[12:37:05] [ERROR] Test: main: 0 tests run!
[12:37:05] ============================================================
[12:37:05] Testing complete. Ran 0 tests: errors: 1

Here, I loaded the test module twice. I can force it to parse it by
doing

$ (dmesg|grep "TAP version"; dmesg|grep -A9999 intel_i915_mock) >logs

But then, it gets confused with two subtests with the same name:

$ ./tools/testing/kunit/kunit.py parse logs
[12:37:34] ============================================================
[12:37:34] ============== intel_i915_mock (18 subtests) ===============
[12:37:34] [PASSED] sanitycheck
[12:37:34] [PASSED] shmem
[12:37:34] [PASSED] fence
[12:37:34] [PASSED] scatterlist
[12:37:34] [PASSED] syncmap
[12:37:34] [PASSED] uncore
[12:37:34] [PASSED] ring
[12:37:34] [PASSED] engine
[12:37:34] [PASSED] timelines
[12:37:34] [PASSED] requests
[12:37:34] [PASSED] objects
[12:37:34] [PASSED] phys
[12:37:34] [PASSED] dmabuf
[12:37:34] [PASSED] vma
[12:37:34] [PASSED] evict
[12:37:34] [PASSED] gtt
[12:37:34] [PASSED] hugepages
[12:37:34] [PASSED] memory_region
[12:37:34] ================= [PASSED] intel_i915_mock =================
[12:37:34] ============== intel_i915_mock (18 subtests) ===============
[12:37:34] [PASSED] sanitycheck
[12:37:34] [PASSED] shmem
[12:37:34] [PASSED] fence
[12:37:34] [PASSED] scatterlist
[12:37:34] [PASSED] syncmap
[12:37:34] [PASSED] uncore
[12:37:34] [PASSED] ring
[12:37:34] [PASSED] engine
[12:37:34] [PASSED] timelines
[12:37:34] [PASSED] requests
[12:37:34] [PASSED] objects
[12:37:34] [PASSED] phys
[12:37:34] [PASSED] dmabuf
[12:37:34] [PASSED] vma
[12:37:34] [PASSED] evict
[12:37:34] [PASSED] gtt
[12:37:34] [PASSED] hugepages
[12:37:34] [PASSED] memory_region
[12:37:34] [ERROR] Test: intel_i915_mock: Expected test number 2 but found 1
[12:37:34] ================= [PASSED] intel_i915_mock =================
[12:37:34] ============================================================
[12:37:34] Testing complete. Ran 36 tests: passed: 36, errors: 1

> Some more details below, if interested.
> Specifically, it happens in
> https://elixir.bootlin.com/linux/latest/C/ident/kunit_print_tap_header
>
> What David is referring to is we had tests that weren't in modules,
> but didn't use the normal built-in code path. Those were also missing
> TAP version lines.
> But KUnit needs to do better here in the case of modules.
>
> The goal was that you'd be able to parse the result of module based
> tests by something like
> $ cat /sys/kernel/debug/kunit/*/results | ./tools/testing/kunit/kunit.py parse
> but this doesn't work because of the lack of a version line.

At least here, debug/kunit doesn't exist:

sudo ls -lctra /sys/kernel/debug/|grep kunit

>
> If we add it to each module's test results, then we'll need to update
> the parse to allow multiple verison lines, I think.

Yeah, the parser is currently broken when used with modules.

Regards,
Mauro

\
 
 \ /
  Last update: 2022-11-01 13:44    [W:0.125 / U:0.488 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site