[U-Boot] [RFC][DO NOT APPLY] binman: Switch to 'python-coverage'
The most portable way to get access to coverage is to invoke it as 'python-coverage'.
Cc: Simon Glass sjg@chromium.org Signed-off-by: Tom Rini trini@konsulko.com --- So, here's the set of problems I have. First, running 'pip install coverage' doesn't get me a 'coverage' binary, so I've just never made good use of 'make tests' and so never have these run. It also means they aren't strictly in travis-ci, so they're never run automatically. With this patch, make tests executes for me, and now fails noting that a lot of things don't have coverage. I'm honestly not sure what the best path forward here is. --- tools/binman/README | 3 +-- tools/binman/binman.py | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-)
diff --git a/tools/binman/README b/tools/binman/README index b20098177ee9..5d55ed68e3da 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -506,8 +506,7 @@ implementations target 100% test coverage. Run 'binman -T' to check this.
To enable Python test coverage on Debian-type distributions (e.g. Ubuntu):
- $ sudo apt-get install python-pip python-pytest - $ sudo pip install coverage + $ sudo apt-get install python-coverage python-pytest
Advanced Features / Technical docs diff --git a/tools/binman/binman.py b/tools/binman/binman.py index fa2f551f5542..447062dca167 100755 --- a/tools/binman/binman.py +++ b/tools/binman/binman.py @@ -71,11 +71,11 @@ def RunTests(debug): def RunTestCoverage(): """Run the tests and check that we get 100% coverage""" # This uses the build output from sandbox_spl to get _libfdt.so - cmd = ('PYTHONPATH=$PYTHONPATH:%s/sandbox_spl/tools coverage run ' + cmd = ('PYTHONPATH=$PYTHONPATH:%s/sandbox_spl/tools python-coverage run ' '--include "tools/binman/*.py" --omit "*test*,*binman.py" ' 'tools/binman/binman.py -t' % options.build_dir) os.system(cmd) - stdout = command.Output('coverage', 'report') + stdout = command.Output('python-coverage', 'report') lines = stdout.splitlines()
test_set= set([os.path.basename(line.split()[0])
Hi Tom,
On 26 May 2018 at 11:06, Tom Rini trini@konsulko.com wrote:
The most portable way to get access to coverage is to invoke it as 'python-coverage'.
Cc: Simon Glass sjg@chromium.org Signed-off-by: Tom Rini trini@konsulko.com
So, here's the set of problems I have. First, running 'pip install coverage' doesn't get me a 'coverage' binary, so I've just never made good use of 'make tests' and so never have these run. It also means they aren't strictly in travis-ci, so they're never run automatically. With this patch, make tests executes for me, and now fails noting that a lot of things don't have coverage. I'm honestly not sure what the best path forward here is.
tools/binman/README | 3 +-- tools/binman/binman.py | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-)
Reviewed-by: Simon Glass sjg@chromium.org
For me this patch works fine. The -T option works correctly.
Typing 'python-coverage html' produces this report:
Note that your patch needs to update this line too:
print "Type 'coverage html' to get a report in htmlcov/index.html"
Can you send me the output you see?
Regards, Simon
On Fri, Jun 01, 2018 at 08:27:03AM -0600, Simon Glass wrote:
Hi Tom,
On 26 May 2018 at 11:06, Tom Rini trini@konsulko.com wrote:
The most portable way to get access to coverage is to invoke it as 'python-coverage'.
Cc: Simon Glass sjg@chromium.org Signed-off-by: Tom Rini trini@konsulko.com
So, here's the set of problems I have. First, running 'pip install coverage' doesn't get me a 'coverage' binary, so I've just never made good use of 'make tests' and so never have these run. It also means they aren't strictly in travis-ci, so they're never run automatically. With this patch, make tests executes for me, and now fails noting that a lot of things don't have coverage. I'm honestly not sure what the best path forward here is.
tools/binman/README | 3 +-- tools/binman/binman.py | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-)
Reviewed-by: Simon Glass sjg@chromium.org
For me this patch works fine. The -T option works correctly.
Typing 'python-coverage html' produces this report:
Note that your patch needs to update this line too:
print "Type 'coverage html' to get a report in htmlcov/index.html"Can you send me the output you see?
So, I need to amend this a bit. On my Docker-based ubuntu 16.04 setup, I see as non-fatal (and not coverage related, but binman test): <unittest.result.TestResult run=79 errors=0 failures=0> <unittest.result.TestResult run=11 errors=0 failures=4> Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 218, in testGood self.assertEqual(result.ok, True) AssertionError: False != True
Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 252, in testIndent self.assertEqual(len(result.problems), 1) AssertionError: 2 != 1
Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 230, in testNoSignoff self.assertEqual(len(result.problems), 1) AssertionError: 2 != 1
Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 241, in testSpaces self.assertEqual(len(result.problems), 3) AssertionError: 4 != 3
Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/7.3.0/ <unittest.result.TestResult run=35 errors=0 failures=1> Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/buildman/test.py", line 426, in testToolchainDownload self.toolchains.LocateArchUrl('arm')) AssertionError: 'https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.9.0/x86_64-gcc...' != 'https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/7.3.0/x86_64-gcc...'
<unittest.result.TestResult run=12 errors=0 failures=0> <unittest.result.TestResult run=79 errors=0 failures=0>
And then the coverage part works fine. With the changes in this RFC patch so that my desktop itself could run coverage stuff I also see: Missing tests for u_boot_with_ucode_ptr.py, u_boot_spl_bss_pad.py, entry.py, u_boot_spl.py, u_boot_spl_with_ucode_ptr.py, u_boot_spl_dtb.py, u_boot_img.py, intel_vbt.py, u_boot_dtb.py, intel_vga.py, intel_descriptor.py, u_boot_dtb_with_ucode.py, intel_cmc.py, x86_start16.py, u_boot_nodtb.py, u_boot.py, intel_fsp.py, blob.py, intel_mrc.py, u_boot_spl_nodtb.py, u_boot_ucode.py, x86_start16_spl.py, intel_me.py ValueError: Test coverage failure Tests FAILED Makefile:1710: recipe for target 'tests' failed make: *** [tests] Error 1
So I guess there's something wrong with trying to use coverage this way, rather than via pip.
Hi Tom,
On 1 June 2018 at 09:36, Tom Rini trini@konsulko.com wrote:
On Fri, Jun 01, 2018 at 08:27:03AM -0600, Simon Glass wrote:
Hi Tom,
On 26 May 2018 at 11:06, Tom Rini trini@konsulko.com wrote:
The most portable way to get access to coverage is to invoke it as 'python-coverage'.
Cc: Simon Glass sjg@chromium.org Signed-off-by: Tom Rini trini@konsulko.com
So, here's the set of problems I have. First, running 'pip install coverage' doesn't get me a 'coverage' binary, so I've just never made good use of 'make tests' and so never have these run. It also means they aren't strictly in travis-ci, so they're never run automatically. With this patch, make tests executes for me, and now fails noting that a lot of things don't have coverage. I'm honestly not sure what the best path forward here is.
tools/binman/README | 3 +-- tools/binman/binman.py | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-)
Reviewed-by: Simon Glass sjg@chromium.org
For me this patch works fine. The -T option works correctly.
Typing 'python-coverage html' produces this report:
Note that your patch needs to update this line too:
print "Type 'coverage html' to get a report in htmlcov/index.html"Can you send me the output you see?
So, I need to amend this a bit. On my Docker-based ubuntu 16.04 setup, I see as non-fatal (and not coverage related, but binman test): <unittest.result.TestResult run=79 errors=0 failures=0> <unittest.result.TestResult run=11 errors=0 failures=4> Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 218, in testGood self.assertEqual(result.ok, True) AssertionError: False != True
Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 252, in testIndent self.assertEqual(len(result.problems), 1) AssertionError: 2 != 1
Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 230, in testNoSignoff self.assertEqual(len(result.problems), 1) AssertionError: 2 != 1
Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/patman/test.py", line 241, in testSpaces self.assertEqual(len(result.problems), 3) AssertionError: 4 != 3
This is new to the SPDX change in checkpatch. I'll send a pull request with a fix for this soon.
Checking: https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/7.3.0/ <unittest.result.TestResult run=35 errors=0 failures=1> Traceback (most recent call last): File "/home/trini/work/u-boot/u-boot/tools/buildman/test.py", line 426, in testToolchainDownload self.toolchains.LocateArchUrl('arm')) AssertionError: 'https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/4.9.0/x86_64-gcc...' != 'https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/7.3.0/x86_64-gcc...'
This is due to the version number changes. It needs a fix to the test.
<unittest.result.TestResult run=12 errors=0 failures=0> <unittest.result.TestResult run=79 errors=0 failures=0>
And then the coverage part works fine. With the changes in this RFC patch so that my desktop itself could run coverage stuff I also see: Missing tests for u_boot_with_ucode_ptr.py, u_boot_spl_bss_pad.py, entry.py, u_boot_spl.py, u_boot_spl_with_ucode_ptr.py, u_boot_spl_dtb.py, u_boot_img.py, intel_vbt.py, u_boot_dtb.py, intel_vga.py, intel_descriptor.py, u_boot_dtb_with_ucode.py, intel_cmc.py, x86_start16.py, u_boot_nodtb.py, u_boot.py, intel_fsp.py, blob.py, intel_mrc.py, u_boot_spl_nodtb.py, u_boot_ucode.py, x86_start16_spl.py, intel_me.py ValueError: Test coverage failure Tests FAILED Makefile:1710: recipe for target 'tests' failed make: *** [tests] Error 1
This looks like it is not getting any coverage info at all. Perhaps there is a complete test failure? Can you try:
binman -t
and see what tests fail?
So I guess there's something wrong with trying to use coverage this way, rather than via pip.
Maybe, but it works OK for me. Given that it works fine on travis (right?) I think we should apply your patch, with the additional change I mentioned in my previous email.
Regards, Simon
participants (2)
-
Simon Glass -
Tom Rini