The current method of running unit tests relies on subcommands of the ut command. Only the code in each subcommand knows how to find the tests related to that subcomand.
This is not ideal and we now have quite a few subcommands which do nothing but locate the relevant tests in a linker list, then call a common function to run them.
This series adds a list of test suites, so that these subcommands can be removed.
An issue with 'ut all' is that it doesn't record how many tests failed overall, so it is necessary to examine copious amounts of output to look for failures. This series adds a new 'total' feature allow recording the total number of failed tests.
To help with 'ut all' a new pytest is created which runs it (as well as 'ut info') and makes sure that all is well. Due to the 'ut all' failures this does not pass, so the test is disabled for now. It is here because it provides security against misnaming a test suite and causing it not to run.
Future work may: - get 'ut all' passing - enable test_suite() in CL, to ensure that 'ut all' keeps passing - record duration of each suite - allow running the tests in random order to tease out dependencies - tweak the output to remove common prefixes - getting rid of bootstd, optee and seame 'ut' subcommands
Simon Glass (43): test: Drop unused suite prototypes test: Rename test_get_state() to ut_get_state() test: Add functions to init and uninit the test state test: Pass the test-state into ut_run_list() Improve support for linker lists in data structures test: Add newlines to hush-test messages test: Rename test suites to match their linker-list name test: Update ut info to show suites test/py: Add a test which runs all unit tests test: Drop the _test suffix on linker lists test: Introduce a better array of test suites test: Drop the function for running bdinfo tests test: Drop the function for running cmd tests test: Drop the function for running common tests test: Drop the function for running dm tests test: Drop the function for running env tests test: Drop the function for running exit tests test: Drop the function for running fdt tests test: Drop the function for running font tests test: Drop the function for running lib tests test: Drop the function for running log tests test: Drop the function for running mbr tests test: Drop the function for running mem tests test: Drop the function for running setexpr tests test: Drop the function for running measurement tests test: Drop the function for running bloblist tests test: Drop the function for running bootm tests test: Drop the function for running addrmap tests test: Drop the function for running hush tests test: Drop the function for running loadm tests test: Drop the function for running pci_mps tests test: Drop the function for running seama tests test: Drop the function for running upl tests test: Drop the info test from the list test: Drop conditional compilation for suites test: Pass the test state to cmd_ut_category() test: Move stats into a struct test: Keep a track of the numbers of tests run test: Move stat-printing into its own function test: Record and show the totals for all test runs test: Sort the test suites test: Disable test_suite test: Move help into the suite declaration
arch/sandbox/cpu/spl.c | 6 +- include/dm/test.h | 2 +- include/linker_lists.h | 51 +++++ include/test/cmd.h | 2 +- include/test/common.h | 2 +- include/test/env.h | 2 +- include/test/hush.h | 2 +- include/test/lib.h | 2 +- include/test/log.h | 4 +- include/test/optee.h | 2 +- include/test/overlay.h | 2 +- include/test/suites.h | 53 ++--- include/test/test.h | 23 ++- include/test/ut.h | 45 ++++- test/boot/bootm.c | 11 +- test/boot/bootstd_common.c | 9 +- test/boot/bootstd_common.h | 3 +- test/boot/measurement.c | 12 +- test/boot/upl.c | 11 +- test/cmd/Makefile | 4 +- test/cmd/addrmap.c | 11 +- test/cmd/bdinfo.c | 10 +- test/cmd/cmd_ut_cmd.c | 20 -- test/cmd/exit.c | 11 +- test/cmd/fdt.c | 10 +- test/cmd/font.c | 10 +- test/cmd/loadm.c | 11 +- test/cmd/mbr.c | 10 +- test/cmd/mem.c | 19 -- test/cmd/mem_copy.c | 2 +- test/cmd/mem_search.c | 2 +- test/cmd/pci_mps.c | 12 +- test/cmd/seama.c | 11 +- test/cmd/setexpr.c | 11 +- test/cmd_ut.c | 361 +++++++++++++++++++++------------- test/common/Makefile | 1 - test/common/bloblist.c | 12 +- test/common/cmd_ut_common.c | 21 -- test/dm/Makefile | 2 - test/dm/test-dm.c | 16 -- test/dm/test-driver.c | 10 +- test/dm/test-uclass.c | 8 +- test/env/cmd_ut_env.c | 10 - test/hush/Makefile | 1 - test/hush/cmd_ut_hush.c | 19 -- test/hush/dollar.c | 7 +- test/hush/loop.c | 4 +- test/lib/Makefile | 1 - test/lib/cmd_ut_lib.c | 19 -- test/log/Makefile | 2 - test/log/log_ut.c | 20 -- test/optee/cmd_ut_optee.c | 4 +- test/overlay/cmd_ut_overlay.c | 14 +- test/py/conftest.py | 7 +- test/py/tests/test_spl.py | 2 +- test/py/tests/test_suite.py | 182 +++++++++++++++++ test/py/tests/test_upl.py | 2 +- test/py/tests/test_ut.py | 2 +- test/py/tests/test_vbe.py | 2 +- test/py/tests/test_vpl.py | 2 +- test/test-main.c | 104 ++++++---- test/ut.c | 4 +- 62 files changed, 657 insertions(+), 580 deletions(-) delete mode 100644 test/cmd/cmd_ut_cmd.c delete mode 100644 test/cmd/mem.c delete mode 100644 test/common/cmd_ut_common.c delete mode 100644 test/dm/test-dm.c delete mode 100644 test/hush/cmd_ut_hush.c delete mode 100644 test/lib/cmd_ut_lib.c delete mode 100644 test/log/log_ut.c create mode 100644 test/py/tests/test_suite.py