Re: [PATCH v3 06/19] test: Avoid failing skipped tests

25 Jun 2024

On Tue, Jun 25, 2024 at 01:38:00PM +0100, Simon Glass wrote:
...
Hi Tom,
On Mon, 24 Jun 2024 at 19:06, Tom Rini trini@konsulko.com wrote:
...
On Sun, Jun 23, 2024 at 02:32:00PM -0600, Simon Glass wrote:
...
When a test returns -EAGAIN this should not be considered a failure.
Fix what seems to be a problem case, where the pytests see a failure
when a test has merely been skipped.
Signed-off-by: Simon Glass sjg@chromium.org
(no changes since v1)
test/test-main.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/test/test-main.c b/test/test-main.c
index 3fa6f6e32ec..cda1a186390 100644
--- a/test/test-main.c
+++ b/test/test-main.c
@@ -448,7 +448,7 @@ static int ut_run_test(struct unit_test_state *uts, struct unit_test *test,
 static int ut_run_test_live_flat(struct unit_test_state *uts,
                               struct unit_test *test)
 {

int runs;




int runs, ret;

if ((test->flags & UT_TESTF_OTHER_FDT) && !IS_ENABLED(CONFIG_SANDBOX))
        return skip_test(uts);



@@ -458,8 +458,11 @@ static int ut_run_test_live_flat(struct unit_test_state *uts,
      if (CONFIG_IS_ENABLED(OF_LIVE)) {
              if (!(test->flags & UT_TESTF_FLAT_TREE)) {
                      uts->of_live = true;

                ut_assertok(ut_run_test(uts, test, test->name));


                runs++;




                ret = ut_run_test(uts, test, test->name);


                if (ret != -EAGAIN) {


                        ut_assertok(ret);


                        runs++;


                }
        }
}




@@ -483,8 +486,11 @@ static int ut_run_test_live_flat(struct unit_test_state *uts,
          (!runs || ut_test_run_on_flattree(test)) &&
          !(gd->flags & GD_FLG_FDT_CHANGED)) {
              uts->of_live = false;

        ut_assertok(ut_run_test(uts, test, test->name));


        runs++;




        ret = ut_run_test(uts, test, test->name);


        if (ret != -EAGAIN) {


                ut_assertok(ret);


                runs++;


        }
}

return 0;



How did you trigger this case exactly?
I noticed this in CI, where some skipped tests were shown as failed in
the log, even though they were not counted as failures in the final
results.
That's really really strange, do you have an example log or something
around still?
-- 
Tom

    