Hi Simon,
On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass sjg@chromium.org wrote:
Add a few notes about how testing works in U-Boot.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Teddy Reed teddy.reed@gmail.com
test/README | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 test/README
diff --git a/test/README b/test/README new file mode 100644 index 0000000..dfd83d6 --- /dev/null +++ b/test/README @@ -0,0 +1,82 @@ +Testing in U-Boot +=================
+U-Boot has a large amount of code. This file describes how this code is +tested and what tests you should write when adding a new feature.
+Sandbox +------- +U-Boot can be built as a user-space application (e.g. for Linux). This +allows test to be executed without needing target hardware. The 'sandbox' +target provides this feature and it is widely used in tests.
+Pytest Suite +------------
+Many tests are available using the pytest suite, in test/py. This can run +either on sandbox or on real hardware. It relies on the U-Boot console to +inject test commands and check the result. It is slower to run than C code, +but provides the ability to unify lots of test and summarise their results.
lots of tests
+You can run the tests on sandbox with:
./test/py/test.py --bd sandbox --build+This will produce HTML output in build-sandbox/test-log.html
+See test/py/README.md for more information about the pytest suite.
+tbot +----
+Tbot provides a way to execute tests on target hardware. It is intended for +trying out both U-Boot and Linux (and potentially other software) on a +number of boards automatically. It can be used to create a continuous test +environment. See tools/tbot/README for more information.
+Ad-hoc tests +------------
+There are several ad-hoc tests which run outside the pytest environment:
- test/fs - File system test (shell script)
- test/image - FIT and lagacy image tests (shell script and Python)
s/lagacy/legacy/
- test/stdint - A test that stdint.h can be used in U-Boot (shell script)
- trace - Test for the tracing feature (shell script)
- vboot - Test for verified boot (shell script)
+The above should be converted to run as part of the pytest suite.
Is this a NOTE or a TODO?
+When to write tests +-------------------
+If you add code to U-Boot without a test you are taking a risk. Even if you +perform thorough manual testing at the time of submission, it may break when +future changes are made to U-Boot. It may even break when applied to mainline, +if other changes interact with it. A good mindset is that untested code +probably doesn't work and should be deleted.
+You can assume that the Pytest suite will be run before patches are accepted +to mainline, so this provides protection against future breakage.
+On the other hand there is quite a bit of code that is not covered with tests, +or is covered sparingly. So here are some suggestions:
+- If you are adding a new uclass, add a sandbox driver and a test that uses it +- If you are modifying code covered by an existing test, add a new test case
- to cover your changes
+- If the code you are modifying has not tests, consider writing one. Even a
- very basic test is useful, and may be picked up and enhanced by others. It
- is much easier to add onto a test - writing a new large test can seem
- daunting to most contributors.
+Future work +-----------
+Converting existing shell scripts into pytest tests.
2.8.0.rc3.226.g39d4020
U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Thank for these docs! I wonder if a clone/mirror of the repo on Github can be set up to run the sandbox/Pytest tests in TravisCI with minimum effort? :)
-Teddy