lkml.org 
[lkml]   [2020]   [Jun]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: [PATCH] kselftest: runner: fix TAP output for skipped tests
From
Date
On 10/06/20 19:43, Bird, Tim wrote:
>>> (rc=$?; \
>>> if [ $rc -eq $skip_rc ]; then \
>>> - echo "not ok $test_num $TEST_HDR_MSG # SKIP"
>>> + echo "ok $test_num $TEST_HDR_MSG # SKIP"
>
> This is a pretty big change, and might break upstream CIs that have come to
> rely on kselftest's existing behavior. I know it's going to break Fuego's parsing
> of results.

Do you have a pointer to this code?

> kselftest has a few conventions that are different from the TAP spec,
> and a few items it does that are extensions to the TAP spec.

Yes, there are extensions to directives are not a problem and parsers
might raise an error on them. That can be an issue, but it's a separate
one (and it's easier to ignore it as long as test pass...).

> IMHO, the TAP spec got this one wrong, but I could be convinced
> otherwise.

Here the TAP spec says that a skip starts with "ok" and has a "SKIP"
directive, and anyone can parse it to treat as it as a failure if
desirable. But doing something else should be treated simply as a
violation of the spec, it's not a matter of "right" or "wrong".

So, if you want to use "not ok ... # SKIP", don't call it TAP.

However, I noticed now that there is another instance of "not ok.*SKIP"
in testing/selftests/kselftest.h (and also one in a comment). So they
should all be fixed at the same time, and I'm okay with holding this patch.

Paolo

> But I think we should discuss this among CI users of
> kselftest before making the change.
>
> I started work quite a while ago on an effort to document the
> conventions used by kselftest (particularly where it deviates
> from the TAP spec), but never submitted it.
>
> I'm going to submit what I've got as an RFC now, for discussion,
> even though it's not finished. I'll do that in a separate thread.
>
>
>>> elif [ $rc -eq $timeout_rc ]; then \
>>> echo "#"
>>> echo "not ok $test_num $TEST_HDR_MSG # TIMEOUT"
>>>
>>
>> Thanks. I will pull this in for Linux 5.8-rc2
> Shuah - can you hold off on this until we discuss it?
>
> Thanks,
> -- Tim
>

\
 
 \ /
  Last update: 2020-06-10 20:25    [W:0.145 / U:0.796 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site