Add a comment about this option in the documentation. Also mention the script that runs these combinations.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v2)

Changes in v2: - Expand docs on how each type of test is marked

doc/develop/testing.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst index bc74eb53e35..f01ca4dc408 100644 --- a/doc/develop/testing.rst +++ b/doc/develop/testing.rst @@ -15,10 +15,20 @@ To run most tests on sandbox, type this:: in the U-Boot directory. Note that only the pytest suite is run using this command.

-Some tests take ages to run. To run just the quick ones, type this:: +Some tests take ages to run and are marked with @pytest.mark.slow. To run just +the quick ones, type this::

make qcheck

+It is also possible to run just the tests for tools (patman, binman, etc.). +Such tests are included with those tools, i.e. no actual U-Boot unit tests are +run. Type this:: + + make tcheck + +All of the above use the test/run script with a paremeter to select which tests +are run. +

Sandbox -------

This was inadvertently disabled after a recent change. Re-enable it.

Signed-off-by: Simon Glass sjg@chromium.org ---

Changes in v3: - Add new patch to re-enable test_ofplatdata

test/py/tests/test_ofplatdata.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/py/tests/test_ofplatdata.py b/test/py/tests/test_ofplatdata.py index 92d09b7aa19..e9cce4daf48 100644 --- a/test/py/tests/test_ofplatdata.py +++ b/test/py/tests/test_ofplatdata.py @@ -4,7 +4,7 @@ import pytest import u_boot_utils as util

-@pytest.mark.boardspec('sandbox') +@pytest.mark.boardspec('sandbox_spl') @pytest.mark.buildconfigspec('spl_of_platdata') def test_spl_devicetree(u_boot_console): """Test content of spl device-tree"""

Add a few notes about the sandbox_spl tests, since they are special.

Signed-off-by: Simon Glass sjg@chromium.org ---

Changes in v3: - Reword the SPL tests section for clarity

Changes in v2: - Add a note that SPL tests can in fact be run individualy - Document how to run all C tests with 'ut all' - Fix 'get list' typo

doc/develop/testing.rst | 5 +++ doc/develop/tests_sandbox.rst | 81 +++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+)

diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst index 87c90eee271..b181c2e2e41 100644 --- a/doc/develop/testing.rst +++ b/doc/develop/testing.rst @@ -58,6 +58,11 @@ also select particular tests with -k::

./test/py/test.py --bd sandbox_flattree --build -k hello

+There are some special tests that run in SPL. For this you need the sandbox_spl +build:: + + ./test/py/test.py --bd sandbox_spl --build -k test_spl + See test/py/README.md for more information about the pytest suite.

See :doc:`tests_sandbox` for how to run tests directly (not through pytest). diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst index 85bbd4f6734..f05de2f2b7a 100644 --- a/doc/develop/tests_sandbox.rst +++ b/doc/develop/tests_sandbox.rst @@ -77,3 +77,84 @@ You can easily use gdb on these tests, without needing --gdbserver::

You can then single-step and look at variables as needed.

+ +Running sandbox_spl tests directly +---------------------------------- + +SPL is the phase before U-Boot proper. It is present in the sandbox_spl build, +so you can run SPL like this:: + + ./spl/u-boot-spl + +SPL tests are special in that they run (only in the SPL phase, of course) if the +-u flag is given:: + + ./spl/u-boot-spl -u + + U-Boot SPL 2021.01-00723-g43c77b51be5-dirty (Jan 24 2021 - 16:38:24 -0700) + Running 5 driver model tests + Test: dm_test_of_plat_base: of_platdata.c (flat tree) + Test: dm_test_of_plat_dev: of_platdata.c (flat tree) + Test: dm_test_of_plat_parent: of_platdata.c (flat tree) + Test: dm_test_of_plat_phandle: of_platdata.c (flat tree) + Test: dm_test_of_plat_props: of_platdata.c (flat tree) + Failures: 0 + + + U-Boot 2021.01-00723-g43c77b51be5-dirty (Jan 24 2021 - 16:38:24 -0700) + + DRAM: 128 MiB + ... + +It is not possible to run SPL tests in U-Boot proper, firstly because they are +not built into U-Boot proper and secondly because the environment is very +different, e.g. some SPL tests rely on of-pldata which is only available in SPL. + +Note that after running, SPL continues to boot into U-Boot proper. You can add +'-c exit' to make U-Boot quit without doing anything further. It is not +currently possible to run SPL tests and then stop, since the pytests require +that U-Boot produces the expected banner. + +You can use the -k flag to select which tests run:: + + ./spl/u-boot-spl -u -k dm_test_of_plat_parent + +Of course you can use gdb with sandbox_spl, just as with sandbox. + + +Running all tests directly +-------------------------- + +A fast way to run all sandbox tests is:: + + ./u-boot -T -c "ut all" + +It typically runs single-thread in 6 seconds on 2021 hardware, with 2s of that +to the delays in the time test. + +This should not be considered a substitute for 'make check', but can be helpful +for git bisect, etc. + + +What tests are built in? +------------------------ + +Whatever sandbox build is used, which tests are present is determined by which +source files are built. For sandbox_spl, the of_platdata tests are built +because of the build rule in test/dm/Makefile:: + + ifeq ($(CONFIG_SPL_BUILD),y) + obj-$(CONFIG_SPL_OF_PLATDATA) += of_platdata.o + else + ...other tests for non-spl + endif + +You can get a list of tests in a U-Boot ELF file by looking for the +linker_list:: + + $ nm /tmp/b/sandbox_spl/spl/u-boot-spl |grep 2_dm_test + 000000000001f200 D _u_boot_list_2_dm_test_2_dm_test_of_plat_base + 000000000001f220 D _u_boot_list_2_dm_test_2_dm_test_of_plat_dev + 000000000001f240 D _u_boot_list_2_dm_test_2_dm_test_of_plat_parent + 000000000001f260 D _u_boot_list_2_dm_test_2_dm_test_of_plat_phandle + 000000000001f280 D _u_boot_list_2_dm_test_2_dm_test_of_plat_props

This prefix should be for setexpr, not mem. This means that trying to select just these tests to run does not work. Fix it.

For some reason this provokes an assertion failure due to memory not being freed. Move the env_set() in setexpr_test_str() to before the malloc() heap size size is recorded and disable the rest in setexpr_test_str_oper().

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

test/cmd/setexpr.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)

diff --git a/test/cmd/setexpr.c b/test/cmd/setexpr.c index fd6d869c0ed..b483069ff0f 100644 --- a/test/cmd/setexpr.c +++ b/test/cmd/setexpr.c @@ -306,8 +306,8 @@ static int setexpr_test_str(struct unit_test_state *uts) ut_asserteq(1, run_command("setexpr.s fred 0", 0)); ut_assertok(ut_check_delta(start_mem));

- start_mem = ut_check_free(); ut_assertok(env_set("fred", "12345")); + start_mem = ut_check_free(); ut_assertok(run_command("setexpr.s fred *0", 0)); ut_asserteq_str("hello", env_get("fred")); ut_assertok(ut_check_delta(start_mem)); @@ -345,7 +345,22 @@ static int setexpr_test_str_oper(struct unit_test_state *uts) start_mem = ut_check_free(); ut_assertok(run_command("setexpr.s fred *0 + *10", 0)); ut_asserteq_str("hello there", env_get("fred")); - ut_assertok(ut_check_delta(start_mem)); + + /* + * This check does not work with sandbox_flattree, apparently due to + * memory allocations in env_set(). + * + * The truetype console produces lots of memory allocations even though + * the LCD display is not visible. But even without these, it does not + * work. + * + * A better test would be for dlmalloc to record the allocs and frees + * for a particular caller, but that is not supported. + * + * For now, drop this test. + * + * ut_assertok(ut_check_delta(start_mem)); + */

unmap_sysmem(buf);

@@ -379,6 +394,6 @@ int do_ut_setexpr(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) setexpr_test); const int n_ents = ll_entry_count(struct unit_test, setexpr_test);

- return cmd_ut_category("cmd_setexpr", "cmd_mem_", tests, n_ents, argc, - argv); + return cmd_ut_category("cmd_setexpr", "setexpr_test_", tests, n_ents, + argc, argv); }

This is the main test function for driver model but not for other tests. Rename the file and the function so this is clear.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

arch/sandbox/cpu/spl.c | 2 +- include/test/test.h | 4 ++-- test/dm/Makefile | 2 +- test/dm/{test-main.c => test-dm.c} | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) rename test/dm/{test-main.c => test-dm.c} (98%)

diff --git a/arch/sandbox/cpu/spl.c b/arch/sandbox/cpu/spl.c index 9a77da15619..fa5055bb9fc 100644 --- a/arch/sandbox/cpu/spl.c +++ b/arch/sandbox/cpu/spl.c @@ -60,7 +60,7 @@ void spl_board_init(void) if (state->run_unittests) { int ret;

- ret = dm_test_main(state->select_unittests); + ret = dm_test_run(state->select_unittests); /* continue execution into U-Boot */ } } diff --git a/include/test/test.h b/include/test/test.h index 27585507d8c..d282cb2362d 100644 --- a/include/test/test.h +++ b/include/test/test.h @@ -106,7 +106,7 @@ enum { struct udevice *testbus_get_clear_removed(void);

/** - * dm_test_main() - Run driver model tests + * dm_test_run() - Run driver model tests * * Run all the available driver model tests, or a selection * @@ -114,6 +114,6 @@ struct udevice *testbus_get_clear_removed(void); * "fdt_pre_reloc"), or NULL to run all * @return 0 if all tests passed, 1 if not */ -int dm_test_main(const char *test_name); +int dm_test_run(const char *test_name);

#endif /* __TEST_TEST_H */ diff --git a/test/dm/Makefile b/test/dm/Makefile index e70e50f4024..3a85ed8996d 100644 --- a/test/dm/Makefile +++ b/test/dm/Makefile @@ -2,7 +2,7 @@ # # Copyright (c) 2013 Google, Inc

-obj-$(CONFIG_UT_DM) += test-main.o +obj-$(CONFIG_UT_DM) += test-dm.o

# Tests for particular subsystems - when enabling driver model for a new # subsystem you must add sandbox tests here. diff --git a/test/dm/test-main.c b/test/dm/test-dm.c similarity index 98% rename from test/dm/test-main.c rename to test/dm/test-dm.c index 69458d62c86..74e12bd8bcc 100644 --- a/test/dm/test-main.c +++ b/test/dm/test-dm.c @@ -145,7 +145,7 @@ static bool test_matches(const char *test_name, const char *find_name) return false; }

-int dm_test_main(const char *test_name) +int dm_test_run(const char *test_name) { struct unit_test *tests = ll_entry_start(struct unit_test, dm_test); const int n_ents = ll_entry_count(struct unit_test, dm_test); @@ -225,5 +225,5 @@ int do_ut_dm(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) if (argc > 1) test_name = argv[1];

- return dm_test_main(test_name); + return dm_test_run(test_name); }

Split out the test preparation into a separation function before expanding it. Add a post-run function as well, currently empty.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

include/test/ut.h | 20 ++++++++++++++++++++ test/test-main.c | 41 +++++++++++++++++++++++++++++++---------- 2 files changed, 51 insertions(+), 10 deletions(-)

diff --git a/include/test/ut.h b/include/test/ut.h index 88e75ab791c..7cb5e10f3af 100644 --- a/include/test/ut.h +++ b/include/test/ut.h @@ -356,6 +356,26 @@ void ut_silence_console(struct unit_test_state *uts); */ void ut_unsilence_console(struct unit_test_state *uts);

+/** + * test_pre_run() - Handle any preparation needed to run a test + * + * @uts: Test state + * @test: Test to prepare for + * @return 0 if OK, -EAGAIN to skip this test since some required feature is not + * available, other -ve on error (meaning that testing cannot likely + * continue) + */ +int test_pre_run(struct unit_test_state *uts, struct unit_test *test); + +/** + * test_post_run() - Handle cleaning up after a test + * + * @uts: Test state + * @test: Test to clean up after + * @return 0 if OK, -ve on error (meaning that testing cannot likely continue) + */ +int test_post_run(struct unit_test_state *uts, struct unit_test *test); + /** * ut_run_tests() - Run a set of tests * diff --git a/test/test-main.c b/test/test-main.c index 376e7ebd3d2..7961fd8aa3e 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -8,6 +8,27 @@ #include <console.h> #include <test/test.h>

+int test_pre_run(struct unit_test_state *uts, struct unit_test *test) +{ + uts->start = mallinfo(); + + if (test->flags & UT_TESTF_CONSOLE_REC) { + int ret = console_record_reset_enable(); + + if (ret) { + printf("Skipping: Console recording disabled\n"); + return -EAGAIN; + } + } + + return 0; +} + +int test_post_run(struct unit_test_state *uts, struct unit_test *test) +{ + return 0; +} + int ut_run_tests(struct unit_test_state *uts, const char *prefix, struct unit_test *tests, int count, const char *select_name) { @@ -17,6 +38,7 @@ int ut_run_tests(struct unit_test_state *uts, const char *prefix,

for (test = tests; test < tests + count; test++) { const char *test_name = test->name; + int ret;

/* Remove the prefix */ if (prefix && !strncmp(test_name, prefix, prefix_len)) @@ -27,18 +49,17 @@ int ut_run_tests(struct unit_test_state *uts, const char *prefix, printf("Test: %s\n", test_name); found++;

- if (test->flags & UT_TESTF_CONSOLE_REC) { - int ret = console_record_reset_enable(); - - if (ret) { - printf("Skipping: Console recording disabled\n"); - continue; - } - } - - uts->start = mallinfo(); + ret = test_pre_run(uts, test); + if (ret == -EAGAIN) + continue; + if (ret) + return ret;

test->func(uts); + + ret = test_post_run(uts, test); + if (ret) + return ret; } if (select_name && !found) return -ENOENT;

Move this step over to the pre-run function.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

test/dm/test-dm.c | 3 --- test/test-main.c | 7 +++++++ 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index 3b1460327ad..3f9ce3d6fa8 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -92,9 +92,6 @@ static int dm_do_test(struct unit_test_state *uts, struct unit_test *test, ut_assertok(dm_scan_plat(false)); if (test->flags & UT_TESTF_PROBE_TEST) ut_assertok(do_autoprobe(uts)); - if (!CONFIG_IS_ENABLED(OF_PLATDATA) && - (test->flags & UT_TESTF_SCAN_FDT)) - ut_assertok(dm_extended_scan(false));

ut_assertok(test_pre_run(uts, test));

diff --git a/test/test-main.c b/test/test-main.c index 9c600094740..a971fe0e9c8 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -6,7 +6,10 @@

#include <common.h> #include <console.h> +#include <dm.h> +#include <dm/root.h> #include <test/test.h> +#include <test/ut.h>

DECLARE_GLOBAL_DATA_PTR;

@@ -16,6 +19,10 @@ int test_pre_run(struct unit_test_state *uts, struct unit_test *test) if (!(test->flags & UT_TESTF_DM)) uts->start = mallinfo();

+ if (!CONFIG_IS_ENABLED(OF_PLATDATA) && + (test->flags & UT_TESTF_SCAN_FDT)) + ut_assertok(dm_extended_scan(false)); + if (test->flags & UT_TESTF_CONSOLE_REC) { int ret = console_record_reset_enable();

Move this step over to the pre-run function.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

test/dm/test-dm.c | 2 -- test/test-main.c | 3 +++ 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index 3f76a1dc2c2..65b89dbb936 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -73,8 +73,6 @@ static int dm_do_test(struct unit_test_state *uts, struct unit_test *test, ut_assertok(dm_test_init(uts, of_live));

uts->start = mallinfo(); - if (test->flags & UT_TESTF_SCAN_PDATA) - ut_assertok(dm_scan_plat(false));

ut_assertok(test_pre_run(uts, test));

diff --git a/test/test-main.c b/test/test-main.c index bd2f08a2b42..fe96d739dc4 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -34,6 +34,9 @@ int test_pre_run(struct unit_test_state *uts, struct unit_test *test) if (!(test->flags & UT_TESTF_DM)) uts->start = mallinfo();

+ if (test->flags & UT_TESTF_SCAN_PDATA) + ut_assertok(dm_scan_plat(false)); + if (test->flags & UT_TESTF_PROBE_TEST) ut_assertok(do_autoprobe(uts));

We already have a function for silencing the console during tests. Use this from test_pre_run() and drop this code from the driver model tests.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

test/dm/test-dm.c | 4 ---- test/test-main.c | 3 ++- 2 files changed, 2 insertions(+), 5 deletions(-)

diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index 57785e503ff..3ffbc382358 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -65,7 +65,6 @@ static int dm_test_destroy(struct unit_test_state *uts) static int dm_do_test(struct unit_test_state *uts, struct unit_test *test, bool of_live) { - struct sandbox_state *state = state_get_current(); const char *fname = strrchr(test->file, '/') + 1;

printf("Test: %s: %s%s\n", test->name, fname, @@ -74,13 +73,10 @@ static int dm_do_test(struct unit_test_state *uts, struct unit_test *test,

ut_assertok(test_pre_run(uts, test));

- if (!state->show_test_output) - gd->flags |= GD_FLG_SILENT; test->func(uts);

ut_assertok(test_post_run(uts, test));

- gd->flags &= ~(GD_FLG_SILENT | GD_FLG_RECORD); state_set_skip_delays(false);

ut_assertok(dm_test_destroy(uts)); diff --git a/test/test-main.c b/test/test-main.c index db0d82e36c3..e273777b6e2 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -50,13 +50,14 @@ int test_pre_run(struct unit_test_state *uts, struct unit_test *test) return -EAGAIN; } } + ut_silence_console(uts);

return 0; }

int test_post_run(struct unit_test_state *uts, struct unit_test *test) { - gd->flags &= ~GD_FLG_RECORD; + ut_unsilence_console(uts);

return 0; }

For driver model tests we want to reinit the data structures so that everything is in a known state before the test runs. This avoids one test changing something that breaks a subsequent tests.

Move the call for this into test_pre_run().

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

include/test/test.h | 2 ++ include/test/ut.h | 10 ++++++++++ test/dm/test-dm.c | 6 +++--- test/test-main.c | 3 +++ 4 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/include/test/test.h b/include/test/test.h index d282cb2362d..6997568cc07 100644 --- a/include/test/test.h +++ b/include/test/test.h @@ -15,6 +15,7 @@ * @fail_count: Number of tests that failed * @start: Store the starting mallinfo when doing leak test * @priv: A pointer to some other info some suites want to track + * @of_live: true to use livetree if available, false to use flattree * @of_root: Record of the livetree root node (used for setting up tests) * @expect_str: Temporary string used to hold expected string value * @actual_str: Temporary string used to hold actual string value @@ -24,6 +25,7 @@ struct unit_test_state { struct mallinfo start; void *priv; struct device_node *of_root; + bool of_live; char expect_str[256]; char actual_str[256]; }; diff --git a/include/test/ut.h b/include/test/ut.h index e5ec18e60b0..6e56ca99c31 100644 --- a/include/test/ut.h +++ b/include/test/ut.h @@ -387,6 +387,16 @@ int test_pre_run(struct unit_test_state *uts, struct unit_test *test); */ int test_post_run(struct unit_test_state *uts, struct unit_test *test);

+/** + * dm_test_init() - Get ready to run a driver model test + * + * This clears out the driver model data structures. For sandbox it resets the + * state structure. + * + * @uts: Test state + */ +int dm_test_init(struct unit_test_state *uts); + /** * ut_run_tests() - Run a set of tests * diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index 398216b6c76..88feb6537f1 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -23,10 +23,10 @@ DECLARE_GLOBAL_DATA_PTR; struct unit_test_state global_dm_test_state; static struct dm_test_state _global_priv_dm_test_state;

-/* Get ready for testing */ -static int dm_test_init(struct unit_test_state *uts, bool of_live) +int dm_test_init(struct unit_test_state *uts) { struct dm_test_state *dms = uts->priv; + bool of_live = uts->of_live;

memset(dms, '\0', sizeof(*dms)); gd->dm_root = NULL; @@ -69,7 +69,7 @@ static int dm_do_test(struct unit_test_state *uts, struct unit_test *test,

printf("Test: %s: %s%s\n", test->name, fname, !of_live ? " (flat tree)" : ""); - ut_assertok(dm_test_init(uts, of_live)); + uts->of_live = of_live;

ut_assertok(test_pre_run(uts, test));

diff --git a/test/test-main.c b/test/test-main.c index 6f0d32f7e27..f14b7b09f79 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -30,6 +30,9 @@ static int do_autoprobe(struct unit_test_state *uts)

int test_pre_run(struct unit_test_state *uts, struct unit_test *test) { + if (test->flags & UT_TESTF_DM) + ut_assertok(dm_test_init(uts)); + ut_set_skip_delays(uts, false);

uts->start = mallinfo();

Move this function into test-main so that all the init is in one place. Rename it so that its purpose is clearer.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

include/test/ut.h | 9 --------- test/dm/test-dm.c | 22 ---------------------- test/test-main.c | 33 ++++++++++++++++++++++++++++++++- 3 files changed, 32 insertions(+), 32 deletions(-)

diff --git a/include/test/ut.h b/include/test/ut.h index 6e56ca99c31..4e0aba9f700 100644 --- a/include/test/ut.h +++ b/include/test/ut.h @@ -387,15 +387,6 @@ int test_pre_run(struct unit_test_state *uts, struct unit_test *test); */ int test_post_run(struct unit_test_state *uts, struct unit_test *test);

-/** - * dm_test_init() - Get ready to run a driver model test - * - * This clears out the driver model data structures. For sandbox it resets the - * state structure. - * - * @uts: Test state - */ -int dm_test_init(struct unit_test_state *uts);

/** * ut_run_tests() - Run a set of tests diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index ad64f5bdcb3..378c6506303 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -11,7 +11,6 @@ #include <log.h> #include <malloc.h> #include <asm/state.h> -#include <dm/test.h> #include <dm/root.h> #include <dm/uclass-internal.h> #include <test/test.h> @@ -22,27 +21,6 @@ DECLARE_GLOBAL_DATA_PTR;

struct unit_test_state global_dm_test_state;

-int dm_test_init(struct unit_test_state *uts) -{ - bool of_live = uts->of_live; - - uts->root = NULL; - uts->testdev = NULL; - uts->force_fail_alloc = false; - uts->skip_post_probe = false; - gd->dm_root = NULL; - if (!CONFIG_IS_ENABLED(OF_PLATDATA)) - memset(dm_testdrv_op_count, '\0', sizeof(dm_testdrv_op_count)); - state_reset_for_test(state_get_current()); - - /* Determine whether to make the live tree available */ - gd_set_of_root(of_live ? uts->of_root : NULL); - ut_assertok(dm_init(of_live)); - uts->root = dm_root(); - - return 0; -} - static int dm_test_destroy(struct unit_test_state *uts) { int id; diff --git a/test/test-main.c b/test/test-main.c index f14b7b09f79..8b0121bdcec 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -7,12 +7,43 @@ #include <common.h> #include <console.h> #include <dm.h> +#include <asm/state.h> #include <dm/root.h> +#include <dm/test.h> #include <test/test.h> #include <test/ut.h>

DECLARE_GLOBAL_DATA_PTR;

+/** + * dm_test_pre_run() - Get ready to run a driver model test + * + * This clears out the driver model data structures. For sandbox it resets the + * state structure + * + * @uts: Test state + */ +static int dm_test_pre_run(struct unit_test_state *uts) +{ + bool of_live = uts->of_live; + + uts->root = NULL; + uts->testdev = NULL; + uts->force_fail_alloc = false; + uts->skip_post_probe = false; + gd->dm_root = NULL; + if (!CONFIG_IS_ENABLED(OF_PLATDATA)) + memset(dm_testdrv_op_count, '\0', sizeof(dm_testdrv_op_count)); + state_reset_for_test(state_get_current()); + + /* Determine whether to make the live tree available */ + gd_set_of_root(of_live ? uts->of_root : NULL); + ut_assertok(dm_init(of_live)); + uts->root = dm_root(); + + return 0; +} + /* Ensure all the test devices are probed */ static int do_autoprobe(struct unit_test_state *uts) { @@ -31,7 +62,7 @@ static int do_autoprobe(struct unit_test_state *uts) int test_pre_run(struct unit_test_state *uts, struct unit_test *test) { if (test->flags & UT_TESTF_DM) - ut_assertok(dm_test_init(uts)); + ut_assertok(dm_test_pre_run(uts));

ut_set_skip_delays(uts, false);

Add a function to handle the preparation for running a test and the post-test clean-up.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

include/test/ut.h | 16 ++++++++++++++++ test/test-main.c | 32 +++++++++++++++++++++++--------- 2 files changed, 39 insertions(+), 9 deletions(-)

diff --git a/include/test/ut.h b/include/test/ut.h index 4e0aba9f700..98f699cbba2 100644 --- a/include/test/ut.h +++ b/include/test/ut.h @@ -387,6 +387,22 @@ int test_pre_run(struct unit_test_state *uts, struct unit_test *test); */ int test_post_run(struct unit_test_state *uts, struct unit_test *test);

+/** + * ut_run_test() - Run a single test + * + * This runs the test, handling any preparation and clean-up needed. It prints + * the name of each test before running it. + * + * @uts: Test state to update. The caller should ensure that this is zeroed for + * the first call to this function. On exit, @uts->fail_count is + * incremented by the number of failures (0, one hopes) + * @test: Test to run + * @name: Name of test, possibly skipping a prefix that should not be displayed + * @return 0 if all tests passed, -EAGAIN if the test should be skipped, -1 if + * any failed + */ +int ut_run_test(struct unit_test_state *uts, struct unit_test *test, + const char *name);

/** * ut_run_tests() - Run a set of tests diff --git a/test/test-main.c b/test/test-main.c index 3806c2ad89c..dee28d35d82 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -121,6 +121,28 @@ int test_post_run(struct unit_test_state *uts, struct unit_test *test) return 0; }

+int ut_run_test(struct unit_test_state *uts, struct unit_test *test, + const char *test_name) +{ + int ret; + + printf("Test: %s\n", test_name); + + ret = test_pre_run(uts, test); + if (ret == -EAGAIN) + return -EAGAIN; + if (ret) + return ret; + + test->func(uts); + + ret = test_post_run(uts, test); + if (ret) + return ret; + + return 0; +} + int ut_run_tests(struct unit_test_state *uts, const char *prefix, struct unit_test *tests, int count, const char *select_name) { @@ -138,20 +160,12 @@ int ut_run_tests(struct unit_test_state *uts, const char *prefix,

if (select_name && strcmp(select_name, test_name)) continue; - printf("Test: %s\n", test_name); + ret = ut_run_test(uts, test, test_name); found++; - - ret = test_pre_run(uts, test); if (ret == -EAGAIN) continue; if (ret) return ret; - - test->func(uts); - - ret = test_post_run(uts, test); - if (ret) - return ret; } if (select_name && !found) return -ENOENT;

In an effort to make use of a common test runner, use ut_run_test() directly to run driver model tests.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

test/dm/test-dm.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)

diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index 38505c2e7a0..3cb4d5c5035 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -21,15 +21,6 @@ DECLARE_GLOBAL_DATA_PTR;

struct unit_test_state global_dm_test_state;

-static int dm_do_test(struct unit_test_state *uts, struct unit_test *test, - bool of_live) -{ - uts->of_live = of_live; - ut_assertok(ut_run_test(uts, test, test->name)); - - return 0; -} - /** * dm_test_run_on_flattree() - Check if we should run a test with flat DT * @@ -102,7 +93,8 @@ int dm_test_run(const char *test_name) runs = 0; if (CONFIG_IS_ENABLED(OF_LIVE)) { if (!(test->flags & UT_TESTF_FLAT_TREE)) { - ut_assertok(dm_do_test(uts, test, true)); + uts->of_live = true; + ut_assertok(ut_run_test(uts, test, test->name)); runs++; } } @@ -113,7 +105,8 @@ int dm_test_run(const char *test_name) */ if (!(test->flags & UT_TESTF_LIVE_TREE) && (!runs || dm_test_run_on_flattree(test))) { - ut_assertok(dm_do_test(uts, test, false)); + uts->of_live = false; + ut_assertok(ut_run_test(uts, test, test->name)); runs++; } found++;

At present we use a global test state for all driver-model tests. Make use of a local struct like we do with the other tests.

To make this work, add functions to get and set this state. When a test starts, the state is set (so it can be used in the test). When a test finishes, the state is unset, so it cannot be used by mistake.

Signed-off-by: Simon Glass sjg@chromium.org ---

Changes in v3: - Use test_set_state() throughout test-main.c instead of direct assignment

include/test/ut.h | 14 ++++++++++++++ test/dm/test-dm.c | 4 +--- test/dm/test-driver.c | 10 +++++++++- test/dm/test-uclass.c | 7 +++++-- test/test-main.c | 18 ++++++++++++++++++ 5 files changed, 47 insertions(+), 6 deletions(-)

diff --git a/include/test/ut.h b/include/test/ut.h index d06bc5089be..bed0e6eb5f6 100644 --- a/include/test/ut.h +++ b/include/test/ut.h @@ -367,6 +367,20 @@ void ut_unsilence_console(struct unit_test_state *uts); */ void ut_set_skip_delays(struct unit_test_state *uts, bool skip_delays);

+/** + * test_get_state() - Get the active test state + * + * @return the currently active test state, or NULL if none + */ +struct unit_test_state *test_get_state(void); + +/** + * test_set_state() - Set the active test state + * + * @uts: Test state to use as currently active test state, or NULL if none + */ +void test_set_state(struct unit_test_state *uts); + /** * ut_run_test_live_flat() - Run a test with both live and flat tree * diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index 48dca289de3..6f9906a510a 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -19,8 +19,6 @@

DECLARE_GLOBAL_DATA_PTR;

-struct unit_test_state global_dm_test_state; - static bool test_matches(const char *test_name, const char *find_name) { if (!find_name) @@ -43,7 +41,7 @@ int dm_test_run(const char *test_name) { struct unit_test *tests = ll_entry_start(struct unit_test, dm_test); const int n_ents = ll_entry_count(struct unit_test, dm_test); - struct unit_test_state *uts = &global_dm_test_state; + struct unit_test_state uts_s = { .fail_count = 0 }, *uts = &uts_s; struct unit_test *test; int found;

diff --git a/test/dm/test-driver.c b/test/dm/test-driver.c index 49c1c310963..ad2b44c405c 100644 --- a/test/dm/test-driver.c +++ b/test/dm/test-driver.c @@ -18,7 +18,6 @@ #include <test/ut.h>

int dm_testdrv_op_count[DM_TEST_OP_COUNT]; -static struct unit_test_state *uts = &global_dm_test_state;

static int testdrv_ping(struct udevice *dev, int pingval, int *pingret) { @@ -37,6 +36,8 @@ static const struct test_ops test_ops = {

static int test_bind(struct udevice *dev) { + struct unit_test_state *uts = test_get_state(); + /* Private data should not be allocated */ ut_assert(!dev_get_priv(dev));

@@ -46,6 +47,7 @@ static int test_bind(struct udevice *dev)

static int test_probe(struct udevice *dev) { + struct unit_test_state *uts = test_get_state(); struct dm_test_priv *priv = dev_get_priv(dev);

/* Private data should be allocated */ @@ -58,6 +60,8 @@ static int test_probe(struct udevice *dev)

static int test_remove(struct udevice *dev) { + struct unit_test_state *uts = test_get_state(); + /* Private data should still be allocated */ ut_assert(dev_get_priv(dev));

@@ -67,6 +71,8 @@ static int test_remove(struct udevice *dev)

static int test_unbind(struct udevice *dev) { + struct unit_test_state *uts = test_get_state(); + /* Private data should not be allocated */ ut_assert(!dev_get_priv(dev));

@@ -116,6 +122,8 @@ static int test_manual_bind(struct udevice *dev)

static int test_manual_probe(struct udevice *dev) { + struct unit_test_state *uts = test_get_state(); + dm_testdrv_op_count[DM_TEST_OP_PROBE]++; if (!uts->force_fail_alloc) dev_set_priv(dev, calloc(1, sizeof(struct dm_test_priv))); diff --git a/test/dm/test-uclass.c b/test/dm/test-uclass.c index f4b540c9278..067701734a0 100644 --- a/test/dm/test-uclass.c +++ b/test/dm/test-uclass.c @@ -17,8 +17,6 @@ #include <test/test.h> #include <test/ut.h>

-static struct unit_test_state *uts = &global_dm_test_state; - int test_ping(struct udevice *dev, int pingval, int *pingret) { const struct test_ops *ops = device_get_ops(dev); @@ -31,6 +29,7 @@ int test_ping(struct udevice *dev, int pingval, int *pingret)

static int test_post_bind(struct udevice *dev) { + struct unit_test_state *uts = test_get_state(); struct dm_test_perdev_uc_pdata *uc_pdata;

dm_testdrv_op_count[DM_TEST_OP_POST_BIND]++; @@ -56,6 +55,7 @@ static int test_pre_unbind(struct udevice *dev) static int test_pre_probe(struct udevice *dev) { struct dm_test_uclass_perdev_priv *priv = dev_get_uclass_priv(dev); + struct unit_test_state *uts = test_get_state();

dm_testdrv_op_count[DM_TEST_OP_PRE_PROBE]++; ut_assert(priv); @@ -66,6 +66,7 @@ static int test_pre_probe(struct udevice *dev)

static int test_post_probe(struct udevice *dev) { + struct unit_test_state *uts = test_get_state(); struct udevice *prev = list_entry(dev->uclass_node.prev, struct udevice, uclass_node);

@@ -100,6 +101,8 @@ static int test_pre_remove(struct udevice *dev)

static int test_init(struct uclass *uc) { + struct unit_test_state *uts = test_get_state(); + dm_testdrv_op_count[DM_TEST_OP_INIT]++; ut_assert(uclass_get_priv(uc));

diff --git a/test/test-main.c b/test/test-main.c index 4e17c9edb28..139fc1f6f18 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -16,6 +16,19 @@

DECLARE_GLOBAL_DATA_PTR;

+/* This is valid when a test is running, NULL otherwise */ +static struct unit_test_state *cur_test_state; + +struct unit_test_state *test_get_state(void) +{ + return cur_test_state; +} + +void test_set_state(struct unit_test_state *uts) +{ + cur_test_state = uts; +} + /** * dm_test_pre_run() - Get ready to run a driver model test * @@ -180,6 +193,9 @@ static int ut_run_test(struct unit_test_state *uts, struct unit_test *test, note = " (flat tree)"; printf("Test: %s: %s%s\n", test_name, fname, note);

+ /* Allow access to test state from drivers */ + test_set_state(uts); + ret = test_pre_run(uts, test); if (ret == -EAGAIN) return -EAGAIN; @@ -192,6 +208,8 @@ static int ut_run_test(struct unit_test_state *uts, struct unit_test *test, if (ret) return ret;

+ test_set_state( NULL); + return 0; }

Update this function to use the return value of ut_run_list() to check for success/failure, so that they are in sync. Also return a command success code so that the caller gets what it expects.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

test/dm/test-dm.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index 252c3e88328..1df0676915f 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -19,16 +19,13 @@

DECLARE_GLOBAL_DATA_PTR;

-struct unit_test_state global_dm_test_state; - int dm_test_run(const char *test_name) { struct unit_test *tests = ll_entry_start(struct unit_test, dm_test); const int n_ents = ll_entry_count(struct unit_test, dm_test); struct unit_test_state uts_s = { .fail_count = 0 }, *uts = &uts_s; struct device_node *of_root; - - uts->fail_count = 0; + int ret;

if (!CONFIG_IS_ENABLED(OF_PLATDATA)) { /* @@ -38,22 +35,23 @@ int dm_test_run(const char *test_name) if (!gd->fdt_blob || fdt_next_node(gd->fdt_blob, 0, NULL) < 0) { puts("Please run with test device tree:\n" " ./u-boot -d arch/sandbox/dts/test.dtb\n"); - ut_assert(gd->fdt_blob); + return CMD_RET_FAILURE; } }

of_root = gd_of_root(); - ut_run_list("driver model", "dm_test_", tests, n_ents, test_name); + ret = ut_run_list("driver model", "dm_test_", tests, n_ents, test_name);

/* Put everything back to normal so that sandbox works as expected */ gd_set_of_root(of_root); gd->dm_root = NULL; - ut_assertok(dm_init(CONFIG_IS_ENABLED(OF_LIVE))); + if (dm_init(CONFIG_IS_ENABLED(OF_LIVE))) + return CMD_RET_FAILURE; dm_scan_plat(false); if (!CONFIG_IS_ENABLED(OF_PLATDATA)) dm_scan_fdt(false);

- return uts->fail_count ? CMD_RET_FAILURE : 0; + return ret ? CMD_RET_FAILURE : 0; }

int do_ut_dm(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[])

Add this functionality to ut_run_list() so it can be removed from dm_test_run().

At this point all tests are run through ut_run_list().

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

test/dm/test-dm.c | 11 ----------- test/test-main.c | 30 +++++++++++++++++++++++++++++- 2 files changed, 29 insertions(+), 12 deletions(-)

diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index 30046c71509..4f7d5736a53 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -23,21 +23,10 @@ int dm_test_run(const char *test_name) { struct unit_test *tests = ll_entry_start(struct unit_test, dm_test); const int n_ents = ll_entry_count(struct unit_test, dm_test); - struct device_node *of_root; int ret;

- of_root = gd_of_root(); ret = ut_run_list("driver model", "dm_test_", tests, n_ents, test_name);

- /* Put everything back to normal so that sandbox works as expected */ - gd_set_of_root(of_root); - gd->dm_root = NULL; - if (dm_init(CONFIG_IS_ENABLED(OF_LIVE))) - return CMD_RET_FAILURE; - dm_scan_plat(false); - if (!CONFIG_IS_ENABLED(OF_PLATDATA)) - dm_scan_fdt(false); - return ret ? CMD_RET_FAILURE : 0; }

diff --git a/test/test-main.c b/test/test-main.c index 8138fb43875..6edd49f0b61 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -143,7 +143,7 @@ static bool test_matches(const char *prefix, const char *test_name, return false; }

-/* +/** * ut_list_has_dm_tests() - Check if a list of tests has driver model ones * * @tests: List of tests to run @@ -162,6 +162,28 @@ static bool ut_list_has_dm_tests(struct unit_test *tests, int count) return false; }

+/** + * dm_test_restore() Put things back to normal so sandbox works as expected + * + * @of_root: Value to set for of_root + * @return 0 if OK, -ve on error + */ +static int dm_test_restore(struct device_node *of_root) +{ + int ret; + + gd_set_of_root(of_root); + gd->dm_root = NULL; + ret = dm_init(CONFIG_IS_ENABLED(OF_LIVE)); + if (ret) + return ret; + dm_scan_plat(false); + if (!CONFIG_IS_ENABLED(OF_PLATDATA)) + dm_scan_fdt(false); + + return 0; +} + /** * test_pre_run() - Handle any preparation needed to run a test * @@ -359,10 +381,12 @@ int ut_run_list(const char *category, const char *prefix, struct unit_test *tests, int count, const char *select_name) { struct unit_test_state uts = { .fail_count = 0 }; + bool has_dm_tests = false; int ret;

if (!CONFIG_IS_ENABLED(OF_PLATDATA) && ut_list_has_dm_tests(tests, count)) { + has_dm_tests = true; /* * If we have no device tree, or it only has a root node, then * these * tests clearly aren't going to work... @@ -385,5 +409,9 @@ int ut_run_list(const char *category, const char *prefix, else printf("Failures: %d\n", uts.fail_count);

+ /* Best efforts only...ignore errors */ + if (has_dm_tests) + dm_test_restore(uts.of_root); + return ret; }

At present we use the linker list directly. This is not very friendly, so add a helpful macro instead. This will also allow us to change the naming later without updating this code.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

include/test/test.h | 6 ++++++ test/bloblist.c | 5 ++--- test/bootm.c | 4 ++-- test/cmd/mem.c | 4 ++-- test/cmd/setexpr.c | 5 ++--- test/compression.c | 5 ++--- test/dm/test-dm.c | 4 ++-- test/env/cmd_ut_env.c | 4 ++-- test/lib/cmd_ut_lib.c | 4 ++-- test/log/log_ut.c | 4 ++-- test/optee/cmd_ut_optee.c | 5 ++--- test/overlay/cmd_ut_overlay.c | 5 ++--- test/str_ut.c | 5 ++--- test/unicode_ut.c | 4 ++-- 14 files changed, 32 insertions(+), 32 deletions(-)

diff --git a/include/test/test.h b/include/test/test.h index 5eeec35f525..b16c9135f2c 100644 --- a/include/test/test.h +++ b/include/test/test.h @@ -93,6 +93,12 @@ struct unit_test { .func = _name, \ }

+/* Get the start of a list of unit tests for a particular category */ +#define UNIT_TEST_SUITE_START(_suite) \ + ll_entry_start(struct unit_test, _suite) +#define UNIT_TEST_SUITE_COUNT(_suite) \ + ll_entry_count(struct unit_test, _suite) + /* Sizes for devres tests */ enum { TEST_DEVRES_SIZE = 100, diff --git a/test/bloblist.c b/test/bloblist.c index 85a6c39680e..22da080a543 100644 --- a/test/bloblist.c +++ b/test/bloblist.c @@ -386,9 +386,8 @@ BLOBLIST_TEST(bloblist_test_reloc, 0); int do_ut_bloblist(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, - bloblist_test); - const int n_ents = ll_entry_count(struct unit_test, bloblist_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(bloblist_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(bloblist_test);

return cmd_ut_category("bloblist", "bloblist_test_", tests, n_ents, argc, argv); diff --git a/test/bootm.c b/test/bootm.c index 92dc2b6e173..6d9e1b8201b 100644 --- a/test/bootm.c +++ b/test/bootm.c @@ -239,8 +239,8 @@ BOOTM_TEST(bootm_test_subst_both, 0);

int do_ut_bootm(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, bootm_test); - const int n_ents = ll_entry_count(struct unit_test, bootm_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(bootm_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(bootm_test);

return cmd_ut_category("bootm", "bootm_test_", tests, n_ents, argc, argv); diff --git a/test/cmd/mem.c b/test/cmd/mem.c index fbaa8a4b3cf..d76f47cf311 100644 --- a/test/cmd/mem.c +++ b/test/cmd/mem.c @@ -12,8 +12,8 @@

int do_ut_mem(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, mem_test); - const int n_ents = ll_entry_count(struct unit_test, mem_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(mem_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(mem_test);

return cmd_ut_category("cmd_mem", "mem_test_", tests, n_ents, argc, argv); diff --git a/test/cmd/setexpr.c b/test/cmd/setexpr.c index b483069ff0f..27113c083c5 100644 --- a/test/cmd/setexpr.c +++ b/test/cmd/setexpr.c @@ -390,9 +390,8 @@ SETEXPR_TEST(setexpr_test_str_long, UT_TESTF_CONSOLE_REC);

int do_ut_setexpr(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, - setexpr_test); - const int n_ents = ll_entry_count(struct unit_test, setexpr_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(setexpr_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(setexpr_test);

return cmd_ut_category("cmd_setexpr", "setexpr_test_", tests, n_ents, argc, argv); diff --git a/test/compression.c b/test/compression.c index a2a4b9ff9e8..4cd1be564f3 100644 --- a/test/compression.c +++ b/test/compression.c @@ -539,9 +539,8 @@ COMPRESSION_TEST(compression_test_bootm_none, 0); int do_ut_compression(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, - compression_test); - const int n_ents = ll_entry_count(struct unit_test, compression_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(compression_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(compression_test);

return cmd_ut_category("compression", "compression_test_", tests, n_ents, argc, argv); diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index 4f7d5736a53..f1f1b0f0149 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -21,8 +21,8 @@ DECLARE_GLOBAL_DATA_PTR;

int dm_test_run(const char *test_name) { - struct unit_test *tests = ll_entry_start(struct unit_test, dm_test); - const int n_ents = ll_entry_count(struct unit_test, dm_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(dm_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(dm_test); int ret;

ret = ut_run_list("driver model", "dm_test_", tests, n_ents, test_name); diff --git a/test/env/cmd_ut_env.c b/test/env/cmd_ut_env.c index a440b1bfb0e..d65a32179ce 100644 --- a/test/env/cmd_ut_env.c +++ b/test/env/cmd_ut_env.c @@ -12,8 +12,8 @@

int do_ut_env(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, env_test); - const int n_ents = ll_entry_count(struct unit_test, env_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(env_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(env_test);

return cmd_ut_category("environment", "env_test_", tests, n_ents, argc, argv); diff --git a/test/lib/cmd_ut_lib.c b/test/lib/cmd_ut_lib.c index f5c7bf3d3b0..f1ac015b2c8 100644 --- a/test/lib/cmd_ut_lib.c +++ b/test/lib/cmd_ut_lib.c @@ -13,8 +13,8 @@

int do_ut_lib(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, lib_test); - const int n_ents = ll_entry_count(struct unit_test, lib_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(lib_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(lib_test);

return cmd_ut_category("lib", "lib_test_", tests, n_ents, argc, argv); } diff --git a/test/log/log_ut.c b/test/log/log_ut.c index c534add493e..5aa3a184004 100644 --- a/test/log/log_ut.c +++ b/test/log/log_ut.c @@ -13,8 +13,8 @@

int do_ut_log(struct cmd_tbl *cmdtp, int flag, int argc, char * const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, log_test); - const int n_ents = ll_entry_count(struct unit_test, log_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(log_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(log_test);

return cmd_ut_category("log", "log_test_", tests, n_ents, argc, argv); diff --git a/test/optee/cmd_ut_optee.c b/test/optee/cmd_ut_optee.c index 9fa4c91e0dd..c3887ab11d9 100644 --- a/test/optee/cmd_ut_optee.c +++ b/test/optee/cmd_ut_optee.c @@ -94,9 +94,8 @@ OPTEE_TEST(optee_fdt_protected_memory, 0);

int do_ut_optee(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, - optee_test); - const int n_ents = ll_entry_count(struct unit_test, optee_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(optee_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(optee_test); struct unit_test_state *uts; void *fdt_optee = &__dtb_test_optee_optee_begin; void *fdt_no_optee = &__dtb_test_optee_no_optee_begin; diff --git a/test/overlay/cmd_ut_overlay.c b/test/overlay/cmd_ut_overlay.c index c001fb183fe..56a3df17138 100644 --- a/test/overlay/cmd_ut_overlay.c +++ b/test/overlay/cmd_ut_overlay.c @@ -213,9 +213,8 @@ OVERLAY_TEST(fdt_overlay_stacked, 0);

int do_ut_overlay(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, - overlay_test); - const int n_ents = ll_entry_count(struct unit_test, overlay_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(overlay_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(overlay_test); struct unit_test_state *uts; void *fdt_base = &__dtb_test_fdt_base_begin; void *fdt_overlay = &__dtb_test_fdt_overlay_begin; diff --git a/test/str_ut.c b/test/str_ut.c index cd5045516d1..359d7d4ea1f 100644 --- a/test/str_ut.c +++ b/test/str_ut.c @@ -107,9 +107,8 @@ STR_TEST(str_simple_strtoul, 0);

int do_ut_str(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, - str_test); - const int n_ents = ll_entry_count(struct unit_test, str_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(str_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(str_test);

return cmd_ut_category("str", "str_", tests, n_ents, argc, argv); } diff --git a/test/unicode_ut.c b/test/unicode_ut.c index 6130ef0b549..7b9c020eff9 100644 --- a/test/unicode_ut.c +++ b/test/unicode_ut.c @@ -615,8 +615,8 @@ UNICODE_TEST(unicode_test_efi_create_indexed_name);

int do_ut_unicode(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { - struct unit_test *tests = ll_entry_start(struct unit_test, unicode_test); - const int n_ents = ll_entry_count(struct unit_test, unicode_test); + struct unit_test *tests = UNIT_TEST_SUITE_START(unicode_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(unicode_test);

return cmd_ut_category("Unicode", "unicode_test_", tests, n_ents, argc, argv);

At present SPL only runs driver model tests. Update it to run all available tests, i.e. in any test suite.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

arch/sandbox/cpu/spl.c | 7 +++++-- include/test/test.h | 16 +++++----------- test/dm/test-dm.c | 11 ++++++++++- test/test-main.c | 17 +++++++++++++---- 4 files changed, 33 insertions(+), 18 deletions(-)

diff --git a/arch/sandbox/cpu/spl.c b/arch/sandbox/cpu/spl.c index fa5055bb9fc..3ef612ea9c2 100644 --- a/arch/sandbox/cpu/spl.c +++ b/arch/sandbox/cpu/spl.c @@ -12,7 +12,7 @@ #include <spl.h> #include <asm/spl.h> #include <asm/state.h> -#include <test/test.h> +#include <test/ut.h>

DECLARE_GLOBAL_DATA_PTR;

@@ -58,9 +58,12 @@ void spl_board_init(void) preloader_console_init();

if (state->run_unittests) { + struct unit_test *tests = UNIT_TEST_ALL_START(); + const int count = UNIT_TEST_ALL_COUNT(); int ret;

- ret = dm_test_run(state->select_unittests); + ret = ut_run_list("spl", NULL, tests, count, + state->select_unittests); /* continue execution into U-Boot */ } } diff --git a/include/test/test.h b/include/test/test.h index 3330dcc72d3..0b124edd601 100644 --- a/include/test/test.h +++ b/include/test/test.h @@ -99,6 +99,11 @@ struct unit_test { #define UNIT_TEST_SUITE_COUNT(_suite) \ ll_entry_count(struct unit_test, ut_ ## _suite)

+/* Use ! and ~ so that all tests will be sorted between these two values */ +#define UNIT_TEST_ALL_START() ll_entry_start(struct unit_test, ut_!) +#define UNIT_TEST_ALL_END() ll_entry_start(struct unit_test, ut_~) +#define UNIT_TEST_ALL_COUNT() (UNIT_TEST_ALL_END() - UNIT_TEST_ALL_START()) + /* Sizes for devres tests */ enum { TEST_DEVRES_SIZE = 100, @@ -119,15 +124,4 @@ enum { */ struct udevice *testbus_get_clear_removed(void);

-/** - * dm_test_run() - Run driver model tests - * - * Run all the available driver model tests, or a selection - * - * @test_name: Name of single test to run (e.g. "dm_test_fdt_pre_reloc" or just - * "fdt_pre_reloc"), or NULL to run all - * @return 0 if all tests passed, 1 if not - */ -int dm_test_run(const char *test_name); - #endif /* __TEST_TEST_H */ diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index f1f1b0f0149..a23ea232463 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -19,7 +19,16 @@

DECLARE_GLOBAL_DATA_PTR;

-int dm_test_run(const char *test_name) +/** + * dm_test_run() - Run driver model tests + * + * Run all the available driver model tests, or a selection + * + * @test_name: Name of single test to run (e.g. "dm_test_fdt_pre_reloc" or just + * "fdt_pre_reloc"), or NULL to run all + * @return 0 if all tests passed, 1 if not + */ +static int dm_test_run(const char *test_name) { struct unit_test *tests = UNIT_TEST_SUITE_START(dm_test); const int n_ents = UNIT_TEST_SUITE_COUNT(dm_test); diff --git a/test/test-main.c b/test/test-main.c index 6edd49f0b61..e1b49e091ab 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -119,7 +119,8 @@ static bool ut_test_run_on_flattree(struct unit_test *test) * * @prefix: String prefix for the tests. Any tests that have this prefix will be * printed without the prefix, so that it is easier to see the unique part - * of the test name. If NULL, no prefix processing is done + * of the test name. If NULL, any suite name (xxx_test) is considered to be + * a prefix. * @test_name: Name of current test * @select_name: Name of test to run (or NULL for all) * @return true to run this test, false to skip it @@ -133,9 +134,17 @@ static bool test_matches(const char *prefix, const char *test_name, if (!strcmp(test_name, select_name)) return true;

- /* All tests have this prefix */ - if (prefix && !strncmp(test_name, prefix, strlen(prefix))) - test_name += strlen(prefix); + if (!prefix) { + const char *p = strstr(test_name, "_test_"); + + /* convert xxx_test_yyy to yyy, i.e. remove the suite name */ + if (p) + test_name = p + 6; + } else { + /* All tests have this prefix */ + if (!strncmp(test_name, prefix, strlen(prefix))) + test_name += strlen(prefix); + }

if (!strcmp(test_name, select_name)) return true;

At present this casts addresses to pointers so cannot work with sandbox. Update the code to use map_sysmem() instead.

As part of this change, the existing load_ptr is renamed to src_ptr since it is not a pointer to load_addr. It is confusing to use a similar name for something that is not actually related. For the alignment code, ALIGN() is used instead of open-coded alignment. Add a comment to the line that casts away a const.

Use a (new) load_ptr variable to access memory at address load_addr.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

common/spl/spl.c | 3 ++- common/spl/spl_fit.c | 27 +++++++++++++++------------ 2 files changed, 17 insertions(+), 13 deletions(-)

diff --git a/common/spl/spl.c b/common/spl/spl.c index cdd7b05f279..8461293a517 100644 --- a/common/spl/spl.c +++ b/common/spl/spl.c @@ -16,6 +16,7 @@ #include <init.h> #include <irq_func.h> #include <log.h> +#include <mapmem.h> #include <serial.h> #include <spl.h> #include <asm/u-boot.h> @@ -167,7 +168,7 @@ __weak void spl_board_prepare_for_boot(void)

__weak struct image_header *spl_get_load_buffer(ssize_t offset, size_t size) { - return (struct image_header *)(CONFIG_SYS_TEXT_BASE + offset); + return map_sysmem(CONFIG_SYS_TEXT_BASE + offset, 0); }

void spl_set_header_raw_uboot(struct spl_image_info *spl_image) diff --git a/common/spl/spl_fit.c b/common/spl/spl_fit.c index a6ad094e91a..d84e0569cf6 100644 --- a/common/spl/spl_fit.c +++ b/common/spl/spl_fit.c @@ -11,6 +11,7 @@ #include <image.h> #include <log.h> #include <malloc.h> +#include <mapmem.h> #include <spl.h> #include <sysinfo.h> #include <asm/cache.h> @@ -245,11 +246,11 @@ static int spl_load_fit_image(struct spl_load_info *info, ulong sector, size_t length; int len; ulong size; - ulong load_addr, load_ptr; + ulong load_addr; + void *load_ptr; void *src; ulong overhead; int nr_sectors; - int align_len = ARCH_DMA_MINALIGN - 1; uint8_t image_comp = -1, type = -1; const void *data; bool external_data = false; @@ -278,11 +279,13 @@ static int spl_load_fit_image(struct spl_load_info *info, ulong sector, }

if (external_data) { + void *src_ptr; + /* External data */ if (fit_image_get_data_size(fit, node, &len)) return -ENOENT;

- load_ptr = (load_addr + align_len) & ~align_len; + src_ptr = map_sysmem(ALIGN(load_addr, ARCH_DMA_MINALIGN), len); length = len;

overhead = get_aligned_image_overhead(info, offset); @@ -290,12 +293,12 @@ static int spl_load_fit_image(struct spl_load_info *info, ulong sector,

if (info->read(info, sector + get_aligned_image_offset(info, offset), - nr_sectors, (void *)load_ptr) != nr_sectors) + nr_sectors, src_ptr) != nr_sectors) return -EIO;

- debug("External data: dst=%lx, offset=%x, size=%lx\n", - load_ptr, offset, (unsigned long)length); - src = (void *)load_ptr + overhead; + debug("External data: dst=%p, offset=%x, size=%lx\n", + src_ptr, offset, (unsigned long)length); + src = src_ptr + overhead; } else { /* Embedded data */ if (fit_image_get_data(fit, node, &data, &length)) { @@ -304,7 +307,7 @@ static int spl_load_fit_image(struct spl_load_info *info, ulong sector, } debug("Embedded data: dst=%lx, size=%lx\n", load_addr, (unsigned long)length); - src = (void *)data; + src = (void *)data; /* cast away const */ }

#ifdef CONFIG_SPL_FIT_SIGNATURE @@ -320,16 +323,16 @@ static int spl_load_fit_image(struct spl_load_info *info, ulong sector, board_fit_image_post_process(&src, &length); #endif

+ load_ptr = map_sysmem(load_addr, length); if (IS_ENABLED(CONFIG_SPL_GZIP) && image_comp == IH_COMP_GZIP) { size = length; - if (gunzip((void *)load_addr, CONFIG_SYS_BOOTM_LEN, - src, &size)) { + if (gunzip(load_ptr, CONFIG_SYS_BOOTM_LEN, src, &size)) { puts("Uncompressing error\n"); return -EIO; } length = size; } else { - memcpy((void *)load_addr, src, length); + memcpy(load_ptr, src, length); }

if (image_info) { @@ -382,7 +385,7 @@ static int spl_fit_append_fdt(struct spl_image_info *spl_image, }

/* Make the load-address of the FDT available for the SPL framework */ - spl_image->fdt_addr = (void *)image_info.load_addr; + spl_image->fdt_addr = map_sysmem(image_info.load_addr, 0); #if !CONFIG_IS_ENABLED(FIT_IMAGE_TINY) if (CONFIG_IS_ENABLED(LOAD_FIT_APPLY_OVERLAY)) { void *tmpbuffer = NULL;

As an example of an SPL test, add a new test for loading a FIT within SPL. This runs on sandbox_spl. For this to work, the text base is adjusted so that there is plenty of space available.

While we are here, document struct spl_load_info properly, since this is currently ambiguous.

This test only verifies the logic path. It does not actually check that the image is loaded correctly. It is not possible for sandbox's SPL to actually run u-boot.img since it currently includes u-boot.bin rather than u-boot. Further work could expand the test in that direction.

The need for this was noted at:

http://patchwork.ozlabs.org/project/uboot/patch/20201216000944.2832585-3-mr....

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v1)

configs/sandbox_spl_defconfig | 2 +- doc/arch/sandbox.rst | 4 +- include/spl.h | 9 ++++ test/Makefile | 1 + test/image/Makefile | 5 ++ test/image/spl_load.c | 91 +++++++++++++++++++++++++++++++++++ 6 files changed, 110 insertions(+), 2 deletions(-) create mode 100644 test/image/Makefile create mode 100644 test/image/spl_load.c

diff --git a/configs/sandbox_spl_defconfig b/configs/sandbox_spl_defconfig index 61dae34a6a2..041c72ac261 100644 --- a/configs/sandbox_spl_defconfig +++ b/configs/sandbox_spl_defconfig @@ -1,4 +1,4 @@ -CONFIG_SYS_TEXT_BASE=0 +CONFIG_SYS_TEXT_BASE=0x200000 CONFIG_SPL_LIBCOMMON_SUPPORT=y CONFIG_SPL_LIBGENERIC_SUPPORT=y CONFIG_NR_DRAM_BANKS=1 diff --git a/doc/arch/sandbox.rst b/doc/arch/sandbox.rst index 60ee1e0741a..1ccb5344ac6 100644 --- a/doc/arch/sandbox.rst +++ b/doc/arch/sandbox.rst @@ -537,5 +537,7 @@ Addr Config Usage e000 CONFIG_BLOBLIST_ADDR Blob list 10000 CONFIG_MALLOC_F_ADDR Early memory allocation f0000 CONFIG_PRE_CON_BUF_ADDR Pre-console buffer - 100000 CONFIG_TRACE_EARLY_ADDR Early trace buffer (if enabled) + 100000 CONFIG_TRACE_EARLY_ADDR Early trace buffer (if enabled). Also used + as the SPL load buffer in spl_test_load(). + 200000 CONFIG_SYS_TEXT_BASE Load buffer for U-Boot (sandbox_spl only) ======= ======================== =============================== diff --git a/include/spl.h b/include/spl.h index e172500b5f8..652d098994f 100644 --- a/include/spl.h +++ b/include/spl.h @@ -222,6 +222,15 @@ struct spl_load_info { void *priv; int bl_len; const char *filename; + /** + * read() - Read from device + * + * @load: Information about the load state + * @sector: Sector number to read from (each @load->bl_len bytes) + * @count: Number of sectors to read + * @buf: Buffer to read into + * @return number of sectors read, 0 on error + */ ulong (*read)(struct spl_load_info *load, ulong sector, ulong count, void *buf); }; diff --git a/test/Makefile b/test/Makefile index 5cd284e322e..a26e915e050 100644 --- a/test/Makefile +++ b/test/Makefile @@ -3,6 +3,7 @@ # (C) Copyright 2012 The Chromium Authors

obj-y += test-main.o +obj-$(CONFIG_SANDBOX) += image/

ifneq ($(CONFIG_$(SPL_)BLOBLIST),) obj-$(CONFIG_$(SPL_)CMDLINE) += bloblist.o diff --git a/test/image/Makefile b/test/image/Makefile new file mode 100644 index 00000000000..c4039df707f --- /dev/null +++ b/test/image/Makefile @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0+ +# +# Copyright 2021 Google LLC + +obj-$(CONFIG_SPL_BUILD) += spl_load.o diff --git a/test/image/spl_load.c b/test/image/spl_load.c new file mode 100644 index 00000000000..851603ddd75 --- /dev/null +++ b/test/image/spl_load.c @@ -0,0 +1,91 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * Copyright 2021 Google LLC + * Written by Simon Glass sjg@chromium.org + */ + +#include <common.h> +#include <image.h> +#include <mapmem.h> +#include <os.h> +#include <spl.h> +#include <test/ut.h> + +/* Declare a new SPL test */ +#define SPL_TEST(_name, _flags) UNIT_TEST(_name, _flags, spl_test) + +/* Context used for this test */ +struct text_ctx { + int fd; +}; + +static ulong read_fit_image(struct spl_load_info *load, ulong sector, + ulong count, void *buf) +{ + struct text_ctx *text_ctx = load->priv; + off_t offset, ret; + ssize_t res; + + offset = sector * load->bl_len; + ret = os_lseek(text_ctx->fd, offset, OS_SEEK_SET); + if (ret != offset) { + printf("Failed to seek to %zx, got %zx (errno=%d)\n", offset, + ret, errno); + return 0; + } + + res = os_read(text_ctx->fd, buf, count * load->bl_len); + if (res == -1) { + printf("Failed to read %lx bytes, got %ld (errno=%d)\n", + count * load->bl_len, res, errno); + return 0; + } + + return count; +} + +int board_fit_config_name_match(const char *name) +{ + return 0; +} + +struct image_header *spl_get_load_buffer(ssize_t offset, size_t size) +{ + return map_sysmem(0x100000, 0); +} + +static int spl_test_load(struct unit_test_state *uts) +{ + struct spl_image_info image; + struct image_header *header; + struct text_ctx text_ctx; + struct spl_load_info load; + char fname[256]; + int ret; + int fd; + + memset(&load, '\0', sizeof(load)); + load.bl_len = 512; + load.read = read_fit_image; + + ret = os_find_u_boot(fname, sizeof(fname), true); + if (ret) { + printf("(%s not found, error %d)\n", fname, ret); + return ret; + } + load.filename = fname; + + header = spl_get_load_buffer(-sizeof(*header), sizeof(*header)); + + fd = os_open(fname, OS_O_RDONLY); + ut_assert(fd >= 0); + ut_asserteq(512, os_read(fd, header, 512)); + text_ctx.fd = fd; + + load.priv = &text_ctx; + + ut_assertok(spl_load_simple_fit(&image, &load, 0, header)); + + return 0; +} +SPL_TEST(spl_test_load, 0);

Add a second on writing tests, covering when to use Python and C, where to put the tests, etc. Add a link to the existing Python test documentation.

Signed-off-by: Simon Glass sjg@chromium.org ---

(no changes since v2)

Changes in v2: - Add new patches to cover running an SPL test

doc/develop/index.rst | 1 + doc/develop/py_testing.rst | 3 +- doc/develop/testing.rst | 2 + doc/develop/tests_sandbox.rst | 7 + doc/develop/tests_writing.rst | 337 ++++++++++++++++++++++++++++++++++ 5 files changed, 349 insertions(+), 1 deletion(-) create mode 100644 doc/develop/tests_writing.rst

diff --git a/doc/develop/index.rst b/doc/develop/index.rst index a81b0de6884..2c5f3c39d79 100644 --- a/doc/develop/index.rst +++ b/doc/develop/index.rst @@ -31,6 +31,7 @@ Testing

testing py_testing + tests_writing tests_sandbox

Refactoring diff --git a/doc/develop/py_testing.rst b/doc/develop/py_testing.rst index f71e837aa96..c5f5adb7bed 100644 --- a/doc/develop/py_testing.rst +++ b/doc/develop/py_testing.rst @@ -13,7 +13,8 @@ results. Advantages of this approach are: U-Boot; there can be no disconnect. - There is no need to write or embed test-related code into U-Boot itself. It is asserted that writing test-related code in Python is simpler and more - flexible than writing it all in C. + flexible than writing it all in C. But see :doc:`tests_writing` for caveats + and more discussion / analysis. - It is reasonably simple to interact with U-Boot in this way.

Requirements diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst index b181c2e2e41..ced13ac8bb4 100644 --- a/doc/develop/testing.rst +++ b/doc/develop/testing.rst @@ -117,6 +117,8 @@ or is covered sparingly. So here are some suggestions: is much easier to add onto a test - writing a new large test can seem daunting to most contributors.

+See doc:`tests_writing` for how to write tests. +

Future work ----------- diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst index 9ab7e0237c6..dd111b17493 100644 --- a/doc/develop/tests_sandbox.rst +++ b/doc/develop/tests_sandbox.rst @@ -198,3 +198,10 @@ linker_list:: 000000000001f240 D _u_boot_list_2_dm_test_2_dm_test_of_plat_parent 000000000001f260 D _u_boot_list_2_dm_test_2_dm_test_of_plat_phandle 000000000001f280 D _u_boot_list_2_dm_test_2_dm_test_of_plat_props + + +Writing tests +------------- + +See :doc:`tests_writing` for how to write new tests. + diff --git a/doc/develop/tests_writing.rst b/doc/develop/tests_writing.rst new file mode 100644 index 00000000000..a36fd1004fb --- /dev/null +++ b/doc/develop/tests_writing.rst @@ -0,0 +1,337 @@ +.. SPDX-License-Identifier: GPL-2.0+ + +Writing Tests +============= + +This describes how to write tests in U-Boot and describes the possible options. + +Test types +---------- + +There are two basic types of test in U-Boot: + + - Python tests, in test/py/tests + - C tests, in test/ and its subdirectories + +Python tests talk to U-Boot via the command line. They support both sandbox and +real hardware. They typically do not require building test code into U-Boot +itself. They are fairly slow to run, due to the command-line interface and there +being two separate processes. Python tests are fairly easy to write. They can +be a little tricky to debug sometimes due to the voluminous output of pytest. + +C tests are written directly in U-Boot. While they can be used on boards, they +are more commonly used with sandbox, as they obviously add to U-Boot code size. +C tests are easy to write so long as the required facilities exist. Where they +do not it can involve refactoring or adding new features to sandbox. They are +fast to run and easy to debug. + +Regardless of which test type is used, all tests are collected and run by the +pytest framework, so there is typically no need to run them separately. This +means that C tests can be used when it makes sense, and Python tests when it +doesn't. + + +This table shows how to decide whether to write a C or Python test: + +===================== =========================== ============================= +Attribute C test Python test +===================== =========================== ============================= +Fast to run? Yes No (two separate processes) +Easy to write? Yes, if required test Yes + features exist in sandbox +Needs code in U-Boot? Yes No, provided the test can be + executed and the result + determined using the command + line +Easy to debug? Yes Yes, but the amount of output can + sometimes require a bit of digging +Can use gdb? Yes, directly Yes, with --gdbserver +Can run on boards? Some can, but only if Yes + compiled in and not + dependent on sandboxau +===================== =========================== ============================= + + +Python or C +----------- + +Typically in U-Boot we encourage C test using sandbox for all features. This +allows fast testing, easy development and allows contributors to make changes +without needing dozens of boards to test with. + +When a test requires setup or interaction with the running host (such as to +generate images and then running U-Boot to check that they can be loaded), or +cannot be run on sandbox, Python tests should be used. These should typically +NOT rely on running with sandbox, but instead should function correctly on any +board supported by U-Boot. + + +How slow are Python tests? +-------------------------- + +Under the hood, Python tests work by starting a sandbox test and connecting to +it via a pipe. Each interaction with the U-Boot process requires at least a +context switch to handle the pipe interaction. The test sends a command to +U-Boot, which then reacts and shows some output, then the test sees that and +continues. Of course on real hardware, communications delays (e.g. with a serial +console) make this slower. + +For comparison, consider a test that checks the 'md' (memory dump). All times +below are approximate, as measured on an AMD 2950X system. Here is is the test +in Python:: + + @pytest.mark.buildconfigspec('cmd_memory') + def test_md(u_boot_console): + """Test that md reads memory as expected, and that memory can be modified + using the mw command.""" + + ram_base = u_boot_utils.find_ram_base(u_boot_console) + addr = '%08x' % ram_base + val = 'a5f09876' + expected_response = addr + ': ' + val + u_boot_console.run_command('mw ' + addr + ' 0 10') + response = u_boot_console.run_command('md ' + addr + ' 10') + assert(not (expected_response in response)) + u_boot_console.run_command('mw ' + addr + ' ' + val) + response = u_boot_console.run_command('md ' + addr + ' 10') + assert(expected_response in response) + +This runs a few commands and checks the output. Note that it runs a command, +waits for the response and then checks it agains what is expected. If run by +itself it takes around 800ms, including test collection. For 1000 runs it takes +19 seconds, or 19ms per run. Of course 1000 runs it not that useful since we +only want to run it once. + +There is no exactly equivalent C test, but here is a similar one that tests 'ms' +(memory search):: + + /* Test 'ms' command with bytes */ + static int mem_test_ms_b(struct unit_test_state *uts) + { + u8 *buf; + + buf = map_sysmem(0, BUF_SIZE + 1); + memset(buf, '\0', BUF_SIZE); + buf[0x0] = 0x12; + buf[0x31] = 0x12; + buf[0xff] = 0x12; + buf[0x100] = 0x12; + ut_assertok(console_record_reset_enable()); + run_command("ms.b 1 ff 12", 0); + ut_assert_nextline("00000030: 00 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................"); + ut_assert_nextline("--"); + ut_assert_nextline("000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 12 ................"); + ut_assert_nextline("2 matches"); + ut_assert_console_end(); + + ut_asserteq(2, env_get_hex("memmatches", 0)); + ut_asserteq(0xff, env_get_hex("memaddr", 0)); + ut_asserteq(0xfe, env_get_hex("mempos", 0)); + + unmap_sysmem(buf); + + return 0; + } + MEM_TEST(mem_test_ms_b, UT_TESTF_CONSOLE_REC); + +This runs the command directly in U-Boot, then checks the console output, also +directly in U-Boot. If run by itself this takes 100ms. For 1000 runs it takes +660ms, or 0.66ms per run. + +So overall running a C test is perhaps 8 times faster individually and the +interactions are perhaps 25 times faster. + +It should also be noted that the C test is fairly easy to debug. You can set a +breakpoint on do_mem_search(), which is what implements the 'ms' command, +single step to see what might be wrong, etc. That is also possible with the +pytest, but requires two terminals and --gdbserver. + + +Why does speed matter? +---------------------- + +Many development activities rely on running tests: + + - 'git bisect run make qcheck' can be used to find a failing commit + - test-driven development relies on quick iteration of build/test + - U-Boot's continuous integration (CI) systems make use of tests. Running + all sandbox tests typically takes 90 seconds and running each qemu test + takes about 30 seconds. This is currently dwarfed by the time taken to + build all boards + +As U-Boot continues to grow its feature set, fast and reliable tests are a +critical factor factor in developer productivity and happiness. + + +Writing C tests +--------------- + +C tests are arranged into suites which are typically executed by the 'ut' +command. Each suite is in its own file. This section describes how to accomplish +some common test tasks. + + +Add a new driver model test +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use this when adding a test for a new or existing uclass, adding new operations +or features to a uclass, adding new ofnode or dev_read_() functions, or anything +else related to driver model. + +Find a suitable place for your test, perhaps near other test functions in +existing code, or in a new file. Each uclass should have its own test file. + +Declare the test with:: + + /* Test that ... */ + static int dm_test_uclassname_what(struct unit_test_state *uts) + { + /* test code here */ + + return 0; + } + DM_TEST(dm_test_uclassname_what, UT_TESTF_SCAN_FDT); + +Replace 'uclassname' with the name of your uclass, if applicable. Replace 'what' +with what you are testing. + +The flags for DM_TEST() are defined in test/test.h and you typically want +UT_TESTF_SCAN_FDT so that the devicetree is scanned and all devices are bound +and ready for use. The DM_TEST macro adds UT_TESTF_DM automatically so that +the test runner knows it is a driver model test. + +Driver model tests are special in that the entire driver model state is +recreated anew for each test. This ensures that if a previous test deletes a +device, for example, it does not affect subsequent tests. Driver model tests +also run both with livetree and flattree, to ensure that both devicetree +implementations work as expected. + +Example commit: c48cb7ebfb4 ("sandbox: add ADC unit tests") [1] + +[1] https://gitlab.denx.de/u-boot/u-boot/-/commit/c48cb7ebfb4 + + +Add a C test to an existing suite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use this when you are adding to or modifying an existing feature outside driver +model. An example is bloblist. + +Add a new function in the same file as the rest of the suite and register it +with the suite. For example, to add a new mem_search test:: + + /* Test 'ms' command with 32-bit values */ + static int mem_test_ms_new_thing(struct unit_test_state *uts) + { + /* test code here*/ + + return 0; + } + MEM_TEST(mem_test_ms_new_thing, UT_TESTF_CONSOLE_REC); + +Note that the MEM_TEST() macros is defined at the top of the file. + +Example commit: 9fe064646d2 ("bloblist: Support relocating to a larger space") [1] + +[1] https://gitlab.denx.de/u-boot/u-boot/-/commit/9fe064646d2 + + +Add a new test suite +~~~~~~~~~~~~~~~~~~~~ + +Each suite should focus on one feature or subsystem, so if you are writing a +new one of those, you should add a new suite. + +Create a new file in test/ or a subdirectory and define a macro to register the +suite. For example:: + + #include <common.h> + #include <console.h> + #include <mapmem.h> + #include <dm/test.h> + #include <test/ut.h> + + /* Declare a new wibble test */ + #define WIBBLE_TEST(_name, _flags) UNIT_TEST(_name, _flags, wibble_test) + + /* Tetss go here */ + + /* At the bottom of the file: */ + + int do_ut_wibble(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) + { + struct unit_test *tests = UNIT_TEST_SUITE_START(wibble_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(wibble_test); + + return cmd_ut_category("cmd_wibble", "wibble_test_", tests, n_ents, argc, argv); + } + +Then add new tests to it as above. + +Register this new suite in test/cmd_ut.c by adding to cmd_ut_sub[]:: + + /* Within cmd_ut_sub[]... */ + + U_BOOT_CMD_MKENT(wibble, CONFIG_SYS_MAXARGS, 1, do_ut_wibble, "", ""), + +and adding new help to ut_help_text[]:: + + "ut wibble - Test the wibble feature\n" + +If your feature is conditional on a particular Kconfig, then you can use #ifdef +to control that. + +Finally, add the test to the build by adding to the Makefile in the same +directory:: + + obj-$(CONFIG_$(SPL_)CMDLINE) += wibble.o + +Note that CMDLINE is never enabled in SPL, so this test will only be present in +U-Boot proper. See below for how to do SPL tests. + +As before, you can add an extra Kconfig check if needed:: + + ifneq ($(CONFIG_$(SPL_)WIBBLE),) + obj-$(CONFIG_$(SPL_)CMDLINE) += wibble.o + endif + + +Example commit: 919e7a8fb64 ("test: Add a simple test for bloblist") [1] + +[1] https://gitlab.denx.de/u-boot/u-boot/-/commit/919e7a8fb64 + + +Making the test run from pytest +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All C tests must run from pytest. Typically this is automatic, since pytest +scans the U-Boot executable for available tests to run. So long as you have a +'ut' subcommand for your test suite, it will run. The same applies for driver +model tests since they use the 'ut dm' subcommand. + +See test/py/tests/test_ut.py for how unit tests are run. + + +Add a C test for SPL +~~~~~~~~~~~~~~~~~~~~ + +Note: C tests are only available for sandbox_spl at present. There is currently +no mechanism in other boards to existing SPL tests even if they are built into +the image. + +SPL tests cannot be run from the 'ut' command since there are no commands +available in SPL. Instead, sandbox (only) calls ut_run_list() on start-up, when +the -u flag is given. This runs the available unit tests, no matter what suite +they are in. + +To create a new SPL test, follow the same rules as above, either adding to an +existing suite or creating a new one. + +An example SPL test is spl_test_load(). + + +Writing Python tests +-------------------- + +See :doc:`py_testing` for brief notes how to write Python tests. You +should be able to use the existing tests in test/py/tests as examples.