On Sun, Jan 26, 2025 at 05:39:22PM -0700, Simon Glass wrote:
Execution time varies widely with the existing tests. Provides a way to produce a summary of the time taken for each test, along with a histogram.
This is enabled with the --timing flag.
Enable it for sandbox in CI.
Example:
Duration : Number of tests ======== : ======================================== <1ms : 1 <8ms : 1 <20ms : # 20 <30ms : ######## 127 <50ms : ######################################## 582 <75ms : ####### 102 <100ms : ## 39 <200ms : ##### 86 <300ms : # 29 <500ms : ## 42 <750ms : # 16 <1.0s : # 15 <2.0s : # 23 <3.0s : 13 <5.0s : 9 <7.5s : 1 <10.0s : 6 <20.0s : 12
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v2:
- Only show the report if the --timing flag is provided
- Use --timing flag to show the report in CI for just the sandbox tests
.azure-pipelines.yml | 3 +- .gitlab-ci.yml | 3 +- doc/develop/py_testing.rst | 33 +++++++++++++++ test/py/conftest.py | 87 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 124 insertions(+), 2 deletions(-)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index df3f82074af..1a5d15a8cdc 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -295,7 +295,7 @@ stages: export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:${PATH} export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci # "${var:+"-k $var"}" expands to "" if $var is empty, "-k $var" if not
./test/py/test.py -ra -o cache_dir="\$UBOOT_TRAVIS_BUILD_DIR"/.pytest_cache --bd \${TEST_PY_BD} \${TEST_PY_ID} \${TEST_PY_TEST_SPEC:+"-k \${TEST_PY_TEST_SPEC}"} --build-dir "\$UBOOT_TRAVIS_BUILD_DIR" --report-dir "\$UBOOT_TRAVIS_BUILD_DIR" --junitxml=\$(System.DefaultWorkingDirectory)/results.xml
./test/py/test.py -ra -o cache_dir="\$UBOOT_TRAVIS_BUILD_DIR"/.pytest_cache --bd \${TEST_PY_BD} \${TEST_PY_ID} \${TEST_TIMING} \${TEST_PY_TEST_SPEC:+"-k \${TEST_PY_TEST_SPEC}"} --build-dir "\$UBOOT_TRAVIS_BUILD_DIR" --report-dir "\$UBOOT_TRAVIS_BUILD_DIR" --junitxml=\$(System.DefaultWorkingDirectory)/results.xml
Lets add a variable to pass further arguments to pytest directly and then use that to pass --timing. That would let us do --capture=tee-sys when needed to really debug a pytest failure in CI, in CI. Just thought of that, sorry.