On Sat, 2 Apr 2022 at 11:06, Alper Nebi Yasak alpernebiyasak@gmail.com wrote:
While running tests for a python tool, the tests' outputs get printed in whatever order they happen to run, without any indication as to which output belongs to which test. Unittest supports capturing these outputs and printing them as part of the test summaries, but when a failure or error occurs it switches back to printing as the tests run. Testtools and subunit tests can do the same as their parts inherit from unittest, but they don't outright expose this functionality.
On the unittest side, enable output buffering for the custom test result class. Try to avoid ugly outputs by not printing stdout/stderr before the test summary for low verbosity levels and for successful tests.
On the subunit side, implement a custom TestProtocolClient that enables the same underlying functionality and injects the captured streams as additional test details. This causes them to be merged into their test's error traceback message, which is later rebuilt into an exception and passed to our unittest report class.
Signed-off-by: Alper Nebi Yasak alpernebiyasak@gmail.com
The way I had to do this for concurrencytest feels hacky, so I am ambivalent towards this patch. I'm actually fine with re-running a failing binman test alone with --debug -v6 to get better, isolated output from it.
Yes that's what I tend to do, but I think we should try this patch. We can always drop it in a future release if it causes problems.
tools/concurrencytest/concurrencytest.py | 83 +++++++++++++++++++++++- tools/patman/test_util.py | 33 +++++++++- 2 files changed, 112 insertions(+), 4 deletions(-)
Reviewed-by: Simon Glass sjg@chromium.org
Applied to u-boot-dm, thanks!