[U-Boot] [PATCH v2 0/7] test: Include Python tools in test coverage

At present there are a number of tools in U-Boot which have tests which are not widely used:
patman - prepare, check and send patches buildman - multi-threaded multi-commit builder dtoc - convert device tree file to C binman - produce firmware images
In fact many tests are broken and this has not been noticed.
This series fixes up the tests and updates 'make tests' to run them. This should help ensure that the tests stay current.
Note: This series is available at u-boot-dm/binman-working. It requires Masahiro's pylibfdt series.
Changes in v2: - Add new patch to correct operation when tests pass - Adjust PYTHONPATH for new locations - Adjust PYTHONPATH for new locations - Drop patches already applied - Run tests with travis-ci also - Drop RFC tag from patch for binman code-coverage tests
Simon Glass (7): test: Correct operation when tests pass test: Run binman tests test: Run patman tests test: Run buildman tests test: Run dtoc tests travis.yml: Run tests for tools binman: Run code coverage tests
.travis.yml | 15 +++++++++++++++ test/run | 16 +++++++++++++++- 2 files changed, 30 insertions(+), 1 deletion(-)

When tests pass an error message is printed because of a variable that is not initialised. Fix this.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v2: - Add new patch to correct operation when tests pass
test/run | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/test/run b/test/run index caee4f83f2c..56160977a3f 100755 --- a/test/run +++ b/test/run @@ -3,9 +3,10 @@ run_test() { $@ [ $? -ne 0 ] && result=$((result+1)) - echo "result $result" }
+result=0 + # Run all tests that the standard sandbox build can support run_test ./test/py/test.py --bd sandbox --build

On Sun, Nov 26, 2017 at 08:25:55PM -0700, Simon Glass wrote:
When tests pass an error message is printed because of a variable that is not initialised. Fix this.
Signed-off-by: Simon Glass sjg@chromium.org
Applied to u-boot/master, thanks!

Update the test script to run the binman tests also.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v2: - Adjust PYTHONPATH for new locations
test/run | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/test/run b/test/run index 56160977a3f..a12ea31bf23 100755 --- a/test/run +++ b/test/run @@ -17,6 +17,9 @@ run_test ./test/py/test.py --bd sandbox_spl --build -k \ # Run tests for the flat DT version of sandbox ./test/py/test.py --bd sandbox_flattree --build
+PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \ + ./tools/binman/binman -t + if [ $result == 0 ]; then echo "Tests passed!" else

On 26 November 2017 at 20:25, Simon Glass sjg@chromium.org wrote:
Update the test script to run the binman tests also.
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v2:
- Adjust PYTHONPATH for new locations
test/run | 3 +++ 1 file changed, 3 insertions(+)
Applied to u-boot-dm

Update the test script to run the patman tests also.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v2: None
test/run | 1 + 1 file changed, 1 insertion(+)
diff --git a/test/run b/test/run index a12ea31bf23..8f72a1f7b29 100755 --- a/test/run +++ b/test/run @@ -19,6 +19,7 @@ run_test ./test/py/test.py --bd sandbox_spl --build -k \
PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \ ./tools/binman/binman -t +run_test ./tools/patman/patman --test
if [ $result == 0 ]; then echo "Tests passed!"

On 26 November 2017 at 20:25, Simon Glass sjg@chromium.org wrote:
Update the test script to run the patman tests also.
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v2: None
test/run | 1 + 1 file changed, 1 insertion(+)
Applied to u-boot-dm

Update the test script to run the buildman tests also.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v2: None
test/run | 1 + 1 file changed, 1 insertion(+)
diff --git a/test/run b/test/run index 8f72a1f7b29..a53383bb8e8 100755 --- a/test/run +++ b/test/run @@ -20,6 +20,7 @@ run_test ./test/py/test.py --bd sandbox_spl --build -k \ PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \ ./tools/binman/binman -t run_test ./tools/patman/patman --test +run_test ./tools/buildman/buildman -t
if [ $result == 0 ]; then echo "Tests passed!"

On 26 November 2017 at 20:25, Simon Glass sjg@chromium.org wrote:
Update the test script to run the buildman tests also.
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v2: None
test/run | 1 + 1 file changed, 1 insertion(+)
Applied to u-boot-dm

Update the test script to run the dtoc tests also.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v2: - Adjust PYTHONPATH for new locations
test/run | 1 + 1 file changed, 1 insertion(+)
diff --git a/test/run b/test/run index a53383bb8e8..af1219fdda9 100755 --- a/test/run +++ b/test/run @@ -21,6 +21,7 @@ PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \ ./tools/binman/binman -t run_test ./tools/patman/patman --test run_test ./tools/buildman/buildman -t +PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test ./tools/dtoc/dtoc -t
if [ $result == 0 ]; then echo "Tests passed!"

On 26 November 2017 at 20:25, Simon Glass sjg@chromium.org wrote:
Update the test script to run the dtoc tests also.
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v2:
- Adjust PYTHONPATH for new locations
test/run | 1 + 1 file changed, 1 insertion(+)
Applied to u-boot-dm

Run tests for the Python tools used by U-Boot.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v2: None
.travis.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
diff --git a/.travis.yml b/.travis.yml index 0b7a0622bfd..5afed98bf6b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -113,6 +113,20 @@ script: ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID} -k "${TEST_PY_TEST_SPEC:-not a_test_which_does_not_exist}" --build-dir "$UBOOT_TRAVIS_BUILD_DIR"; + ret=$?; + if [[ $ret -ne 0 ]]; then + exit $ret; + fi; + fi; + if [[ -n "${TEST_PY_TOOLS}" ]]; then + PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt" + PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}" + ./tools/binman/binman -t && + ./tools/patman/patman --test && + ./tools/buildman/buildman -t && + PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt" + PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}" + ./tools/dtoc/dtoc -t; fi
matrix: @@ -283,6 +297,7 @@ matrix: TEST_PY_TEST_SPEC="test_ofplatdata" BUILDMAN="^sandbox$" TOOLCHAIN="x86_64" + TEST_PY_TOOLS="yes" - env: - TEST_PY_BD="sandbox_flattree" BUILDMAN="^sandbox_flattree$"

Run tests for the Python tools used by U-Boot.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v2: None
.travis.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
Applied to u-boot-dm thanks!

Binman has 100% test coverage for the code as it is at present. To encourage it to stay that way, run the code-coverage test as part of the normal U-Boot tests.
This is RFC because it requires the Python code coverage tools to be available.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v2: - Drop patches already applied - Run tests with travis-ci also - Drop RFC tag from patch for binman code-coverage tests
test/run | 7 +++++++ 1 file changed, 7 insertions(+)
diff --git a/test/run b/test/run index af1219fdda9..d5631b802af 100755 --- a/test/run +++ b/test/run @@ -23,6 +23,13 @@ run_test ./tools/patman/patman --test run_test ./tools/buildman/buildman -t PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test ./tools/dtoc/dtoc -t
+# This needs you to set up Python test coverage tools. +# To enable Python test coverage on Debian-type distributions (e.g. Ubuntu): +# $ sudo apt-get install python-pip python-pytest +# $ sudo pip install coverage +PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \ + ./tools/binman/binman -T + if [ $result == 0 ]; then echo "Tests passed!" else

Binman has 100% test coverage for the code as it is at present. To encourage it to stay that way, run the code-coverage test as part of the normal U-Boot tests.
This is RFC because it requires the Python code coverage tools to be available.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v2: - Drop patches already applied - Run tests with travis-ci also - Drop RFC tag from patch for binman code-coverage tests
test/run | 7 +++++++ 1 file changed, 7 insertions(+)
Applied to u-boot-dm thanks!
participants (3)
-
Simon Glass
-
sjg@google.com
-
Tom Rini