Hi Tom,
On 1 June 2018 at 09:36, Tom Rini trini@konsulko.com wrote:
On Fri, Jun 01, 2018 at 08:27:03AM -0600, Simon Glass wrote:
Hi Tom,
On 26 May 2018 at 11:06, Tom Rini trini@konsulko.com wrote:
The most portable way to get access to coverage is to invoke it as 'python-coverage'.
Cc: Simon Glass sjg@chromium.org Signed-off-by: Tom Rini trini@konsulko.com
So, here's the set of problems I have. First, running 'pip install coverage' doesn't get me a 'coverage' binary, so I've just never made good use of 'make tests' and so never have these run. It also means they aren't strictly in travis-ci, so they're never run automatically. With this patch, make tests executes for me, and now fails noting that a lot of things don't have coverage. I'm honestly not sure what the best path forward here is.
tools/binman/README | 3 +-- tools/binman/binman.py | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-)
Reviewed-by: Simon Glass sjg@chromium.org
For me this patch works fine. The -T option works correctly.
Typing 'python-coverage html' produces this report:
Note that your patch needs to update this line too:
print "Type 'coverage html' to get a report in htmlcov/index.html"Can you send me the output you see?
So, I need to amend this a bit. On my Docker-based ubuntu 16.04 setup, I see as non-fatal (and not coverage related, but binman test): <unittest.result.TestResult run=79 errors=0 failures=0> <unittest.result.TestResult run=11 errors=0 failures=4> Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 218, in testGood self.assertEqual(result.ok, True) AssertionError: False != True
Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 252, in testIndent self.assertEqual(len(result.problems), 1) AssertionError: 2 != 1
Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 230, in testNoSignoff self.assertEqual(len(result.problems), 1) AssertionError: 2 != 1
Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 241, in testSpaces self.assertEqual(len(result.problems), 3) AssertionError: 4 != 3
This is new to the SPDX change in checkpatch. I'll send a pull request with a fix for this soon.
Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/7.3.0/ <unittest.result.TestResult run=35 errors=0 failures=1> Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/buildman/test.py", line 426, in testToolchainDownload self.toolchains.LocateArchUrl('arm')) AssertionError: 'https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.9.0/x86_64-gcc...' != 'https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/7.3.0/x86_64-gcc...'
This is due to the version number changes. It needs a fix to the test.
<unittest.result.TestResult run=12 errors=0 failures=0> <unittest.result.TestResult run=79 errors=0 failures=0>
And then the coverage part works fine. With the changes in this RFC patch so that my desktop itself could run coverage stuff I also see: Missing tests for u_boot_with_ucode_ptr.py, u_boot_spl_bss_pad.py, entry.py, u_boot_spl.py, u_boot_spl_with_ucode_ptr.py, u_boot_spl_dtb.py, u_boot_img.py, intel_vbt.py, u_boot_dtb.py, intel_vga.py, intel_descriptor.py, u_boot_dtb_with_ucode.py, intel_cmc.py, x86_start16.py, u_boot_nodtb.py, u_boot.py, intel_fsp.py, blob.py, intel_mrc.py, u_boot_spl_nodtb.py, u_boot_ucode.py, x86_start16_spl.py, intel_me.py ValueError: Test coverage failure Tests FAILED Makefile:1710: recipe for target 'tests' failed make: *** [tests] Error 1
This looks like it is not getting any coverage info at all. Perhaps there is a complete test failure? Can you try:
binman -t
and see what tests fail?
So I guess there's something wrong with trying to use coverage this way, rather than via pip.
Maybe, but it works OK for me. Given that it works fine on travis (right?) I think we should apply your patch, with the additional change I mentioned in my previous email.
Regards, Simon