On 4/20/20 8:32 PM, Tom Rini wrote:
On Mon, Apr 20, 2020 at 08:23:08PM +0200, Heinrich Schuchardt wrote:
On 4/20/20 1:38 AM, Simon Glass wrote:
On Sat, 18 Apr 2020 at 06:09, Heinrich Schuchardt xypron.glpk@gmx.de wrote:
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1] test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1] test/py/tests/test_efi_loader.py:100: No static network configuration is defined SKIPPED [2] test/py/tests/test_efi_loader.py:115: Network not initialized
Signed-off-by: Heinrich Schuchardt xypron.glpk@gmx.de
test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.py sssssssssssssssssssssssssssssssssssssss [ 0%] test/py/tests/test_fs/test_ext.py ssssssssssssssssssssss [ 0%] test/py/tests/test_fs/test_mkdir.py ssssssssssss [ 0%] test/py/tests/test_fs/test_symlink.py ssss [ 0%] test/py/tests/test_fs/test_unlink.py ssssssssssssss [ 0%]
This new line has actionable information:
SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for filesystem: fat16
Next step is to change this line to provide a more useful output, e.g.
- except CalledProcessError:
pytest.skip('Setup failed for filesystem: ' + fs_type)
- except CalledProcessError as err:
pytest.skip('Setup failed for filesystem: ' + fs_type + \', {}'.format(err))SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.
Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using /usr/sbin/mkfs.vfat or /sbin/mkfs.vfat and we can fix it.
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
OK, so I ran this through "qcheck" and I got a bit more output, but some of which indicates we should figure out how to enable these tests for sandbox out of the box.
The follow-up patch I really want to see is passing -ra in the travis/gitlab/azure file as I very much agree we should get everything running in CI or have a good reason why not (ie the MMC tests might be hard to enable on all platforms via qemu but we should make a good try at it).
And giving better error messages too would be very helpful so it's clearer why we aren't (and why some people would not want to) run the FS tests for example as we really haven't figured out a good 100% non-root-user method to do those tests.
The tests run fine on my Debian system with https://github.com/xypron2/u-boot/commit/f465d52a5a539761d9e5602331280ce07a1... where I assume mkfs.* and fsck.ext4 are in /sbin. The same should work on Ubuntu.
On current distributions there are also symlinks in /usr/sbin/ but https://askubuntu.com/questions/674847/sudo-usr-sbin-mkfs-ext4-command-not-f... indicates that this does not hold true for elder distributions.
I will submit the test once tested in Travis CI and Gitlab CI.
Best regards
Heinrich