Buildman's test coverage is inadequate, particularly in the area of the core builder threads and logic. As a result it is harder to make changes than it should be, since verifying correctness manually is tedious.
The existing output test relies on the user to verify that things look OK. This is getting harder with more output options available, so this series turns it into a unit test.
A new functional test is provided which tests buildman from the parser options down to the logic that issues git and make commands. This runs in a few seconds and provides coverage of the builder logic and threads, plus most build-related options.
Output formatting is already tested in test.py, and there is also a test there which checks that errors and warnings are correctly detected by the build system and reported in the summary.
So overall, with this series, test coverage is now considerably better.
This test also fixes a few bugs, some reported by Steve Rae: - The -H options hang - An incorrect 'failure' count value in some cases - Correct GetMetaDataForList() action - Permit '/' in branch names - Ignore conflicting tags (allows building multiple series together)
Changes in v3: - Add new patch to permit branch names with an embedded '/' - Add new patch to ignore conflicting tags in buildman
Changes in v2: - Add a function to print out the terminal output recorded - Add a comment to _HandleCommandGit - Make sure the test temporary directory is removed - Add patch to expand output test to cover directory prefixes
Simon Glass (18): patman: Add a way of recording terminal output for testing buildman: Send builder output through a function for testing buildman: Enhance basic test to check summary output patman: RunPipe() should not pipe stdout/stderr unless asked buildman: Move the command line code into its own file buildman: Move full help code into the control module patman: Provide a way to intercept commands for testing buildman: Add a functional test buildman: Set up bsettings outside the control module buildman: Avoid looking at config file or toolchains in tests buildman: Allow tests to have their own boards buildman: Correct counting of build failures on retry buildman: Provide an internal option to clean the outpur dir patman: Start with a clean series when needed buildman: Add additional functional tests buildman: Expand output test to cover directory prefixes buildman: Permit branch names with an embedded '/' buildman: Ignore conflicting tags
tools/buildman/bsettings.py | 15 +- tools/buildman/builder.py | 58 ++--- tools/buildman/builderthread.py | 15 +- tools/buildman/buildman.py | 98 +------- tools/buildman/cmdline.py | 85 +++++++ tools/buildman/control.py | 73 ++++-- tools/buildman/func_test.py | 519 ++++++++++++++++++++++++++++++++++++++++ tools/buildman/test.py | 153 +++++++++++- tools/buildman/toolchain.py | 4 +- tools/patman/command.py | 22 ++ tools/patman/patchstream.py | 6 +- tools/patman/terminal.py | 72 ++++++ 12 files changed, 955 insertions(+), 165 deletions(-) create mode 100644 tools/buildman/cmdline.py create mode 100644 tools/buildman/func_test.py