On 4/20/20 1:38 AM, Simon Glass wrote:
On Sat, 18 Apr 2020 at 06:09, Heinrich Schuchardt xypron.glpk@gmx.de wrote:
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1] test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1] test/py/tests/test_efi_loader.py:100: No static network configuration is defined SKIPPED [2] test/py/tests/test_efi_loader.py:115: Network not initialized
Signed-off-by: Heinrich Schuchardt xypron.glpk@gmx.de
test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.py sssssssssssssssssssssssssssssssssssssss [ 0%] test/py/tests/test_fs/test_ext.py ssssssssssssssssssssss [ 0%] test/py/tests/test_fs/test_mkdir.py ssssssssssss [ 0%] test/py/tests/test_fs/test_symlink.py ssss [ 0%] test/py/tests/test_fs/test_unlink.py ssssssssssssss [ 0%]
This new line has actionable information:
SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for filesystem: fat16
Next step is to change this line to provide a more useful output, e.g.
- except CalledProcessError: - pytest.skip('Setup failed for filesystem: ' + fs_type) + except CalledProcessError as err: + pytest.skip('Setup failed for filesystem: ' + fs_type + \ + ', {}'.format(err))
SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.
Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using /usr/sbin/mkfs.vfat or /sbin/mkfs.vfat and we can fix it.
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
Best regards
Heinrich