Hi Simon,
Thank you for the series.
On mer., janv. 15, 2025 at 06:30, Simon Glass sjg@chromium.org wrote:
The current method of running unit tests relies on subcommands of the ut command. Only the code in each subcommand knows how to find the tests related to that subcomand.
This is not ideal and we now have quite a few subcommands which do nothing but locate the relevant tests in a linker list, then call a common function to run them.
This series adds a list of test suites, so that these subcommands can be removed.
An issue with 'ut all' is that it doesn't record how many tests failed overall, so it is necessary to examine copious amounts of output to look for failures. This series adds a new 'total' feature allow recording the total number of failed tests.
To help with 'ut all' a new pytest is created which runs it (as well as 'ut info') and makes sure that all is well. Due to the 'ut all' failures this does not pass, so the test is disabled for now. It is here because it provides security against misnaming a test suite and causing it not to run.
I wanted to test this, but it seems it does not apply on both master or next:
With master: 178f6ecb21fe ("Merge patch series "bootstd: Support recording images"") With next: d6da3dbaef57 ("Merge patch series "cmd: Add support for optee commands."")
When running: $ b4 shazam -s -l --check 20250115133114.590375-1-sjg@chromium.org
I see:
""" [...] snip
Total patches: 43 --- Applying: test: Drop unused suite prototypes Applying: test: Rename test_get_state() to ut_get_state() Applying: test: Add functions to init and uninit the test state Applying: test: Pass the test-state into ut_run_list() Applying: Improve support for linker lists in data structures Applying: test: Add newlines to hush-test messages Applying: test: Rename test suites to match their linker-list name Applying: test: Update ut info to show suites Applying: test/py: Add a test which runs all unit tests Applying: test: Drop the _test suffix on linker lists Applying: test: Introduce a better array of test suites Patch failed at 0011 test: Introduce a better array of test suites error: sha1 information is lacking or useless (test/cmd_ut.c). error: could not build fake ancestor hint: Use 'git am --show-current-patch=diff' to see the failed patch hint: When you have resolved this problem, run "git am --continue". hint: If you prefer to skip this patch, run "git am --skip" instead. hint: To restore the original branch and stop patching, run "git am --abort". """
On what base should this be applied if we want to test this? Sorry if this is a sily question, usually I get dependencies from the cover letter but I could not find any here.
Future work may:
- get 'ut all' passing
- enable test_suite() in CL, to ensure that 'ut all' keeps passing
- record duration of each suite
- allow running the tests in random order to tease out dependencies
- tweak the output to remove common prefixes
- getting rid of bootstd, optee and seame 'ut' subcommands
Simon Glass (43): test: Drop unused suite prototypes test: Rename test_get_state() to ut_get_state() test: Add functions to init and uninit the test state test: Pass the test-state into ut_run_list() Improve support for linker lists in data structures test: Add newlines to hush-test messages test: Rename test suites to match their linker-list name test: Update ut info to show suites test/py: Add a test which runs all unit tests test: Drop the _test suffix on linker lists test: Introduce a better array of test suites test: Drop the function for running bdinfo tests test: Drop the function for running cmd tests test: Drop the function for running common tests test: Drop the function for running dm tests test: Drop the function for running env tests test: Drop the function for running exit tests test: Drop the function for running fdt tests test: Drop the function for running font tests test: Drop the function for running lib tests test: Drop the function for running log tests test: Drop the function for running mbr tests test: Drop the function for running mem tests test: Drop the function for running setexpr tests test: Drop the function for running measurement tests test: Drop the function for running bloblist tests test: Drop the function for running bootm tests test: Drop the function for running addrmap tests test: Drop the function for running hush tests test: Drop the function for running loadm tests test: Drop the function for running pci_mps tests test: Drop the function for running seama tests test: Drop the function for running upl tests test: Drop the info test from the list test: Drop conditional compilation for suites test: Pass the test state to cmd_ut_category() test: Move stats into a struct test: Keep a track of the numbers of tests run test: Move stat-printing into its own function test: Record and show the totals for all test runs test: Sort the test suites test: Disable test_suite test: Move help into the suite declaration
arch/sandbox/cpu/spl.c | 6 +- include/dm/test.h | 2 +- include/linker_lists.h | 51 +++++ include/test/cmd.h | 2 +- include/test/common.h | 2 +- include/test/env.h | 2 +- include/test/hush.h | 2 +- include/test/lib.h | 2 +- include/test/log.h | 4 +- include/test/optee.h | 2 +- include/test/overlay.h | 2 +- include/test/suites.h | 53 ++--- include/test/test.h | 23 ++- include/test/ut.h | 45 ++++- test/boot/bootm.c | 11 +- test/boot/bootstd_common.c | 9 +- test/boot/bootstd_common.h | 3 +- test/boot/measurement.c | 12 +- test/boot/upl.c | 11 +- test/cmd/Makefile | 4 +- test/cmd/addrmap.c | 11 +- test/cmd/bdinfo.c | 10 +- test/cmd/cmd_ut_cmd.c | 20 -- test/cmd/exit.c | 11 +- test/cmd/fdt.c | 10 +- test/cmd/font.c | 10 +- test/cmd/loadm.c | 11 +- test/cmd/mbr.c | 10 +- test/cmd/mem.c | 19 -- test/cmd/mem_copy.c | 2 +- test/cmd/mem_search.c | 2 +- test/cmd/pci_mps.c | 12 +- test/cmd/seama.c | 11 +- test/cmd/setexpr.c | 11 +- test/cmd_ut.c | 361 +++++++++++++++++++++------------- test/common/Makefile | 1 - test/common/bloblist.c | 12 +- test/common/cmd_ut_common.c | 21 -- test/dm/Makefile | 2 - test/dm/test-dm.c | 16 -- test/dm/test-driver.c | 10 +- test/dm/test-uclass.c | 8 +- test/env/cmd_ut_env.c | 10 - test/hush/Makefile | 1 - test/hush/cmd_ut_hush.c | 19 -- test/hush/dollar.c | 7 +- test/hush/loop.c | 4 +- test/lib/Makefile | 1 - test/lib/cmd_ut_lib.c | 19 -- test/log/Makefile | 2 - test/log/log_ut.c | 20 -- test/optee/cmd_ut_optee.c | 4 +- test/overlay/cmd_ut_overlay.c | 14 +- test/py/conftest.py | 7 +- test/py/tests/test_spl.py | 2 +- test/py/tests/test_suite.py | 182 +++++++++++++++++ test/py/tests/test_upl.py | 2 +- test/py/tests/test_ut.py | 2 +- test/py/tests/test_vbe.py | 2 +- test/py/tests/test_vpl.py | 2 +- test/test-main.c | 104 ++++++---- test/ut.c | 4 +- 62 files changed, 657 insertions(+), 580 deletions(-) delete mode 100644 test/cmd/cmd_ut_cmd.c delete mode 100644 test/cmd/mem.c delete mode 100644 test/common/cmd_ut_common.c delete mode 100644 test/dm/test-dm.c delete mode 100644 test/hush/cmd_ut_hush.c delete mode 100644 test/lib/cmd_ut_lib.c delete mode 100644 test/log/log_ut.c create mode 100644 test/py/tests/test_suite.py
-- 2.34.1