Add this shortcut for running tests. Unfortunately 'make test' cannot be used as it is an existing directory.

Signed-off-by: Simon Glass sjg@chromium.org Suggested-by: Teddy Reed teddy.reed@gmail.com ---

Makefile | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile index 99cc8cf..91634d5 100644 --- a/Makefile +++ b/Makefile @@ -425,7 +425,7 @@ timestamp_h := include/generated/timestamp_autogenerated.h

no-dot-config-targets := clean clobber mrproper distclean \ help %docs check% coccicheck \ - ubootversion backup + ubootversion backup tests

config-targets := 0 mixed-targets := 0 @@ -1489,6 +1489,7 @@ help: @echo '' @echo 'Other generic targets:' @echo ' all - Build all necessary images depending on configuration' + @echo ' tests - Build U-Boot for sandbox and run tests' @echo '* u-boot - Build the bare u-boot' @echo ' dir/ - Build all files in dir and below' @echo ' dir/file.[oisS] - Build specified target only' @@ -1521,6 +1522,8 @@ help: @echo 'Execute "make" or "make all" to build all targets marked with [*] ' @echo 'For further info see the ./README file'

+tests: + $(srctree)/test/run

# Documentation targets # ---------------------------------------------------------------------------

Fix review comments that were missed at the time. Also explain why we need to regenerate the device tree for each test.

Reported-by: Teddy Reed teddy.reed@gmail.com Suggested-by: Stephen Warren swarren@nvidia.com Signed-off-by: Simon Glass sjg@chromium.org Fixes: f6349c3c (test: Add a README) ---

test/README | 8 ++++---- test/py/tests/test_vboot.py | 6 ++++-- 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/test/README b/test/README index ee55972..1142e9c 100644 --- a/test/README +++ b/test/README @@ -13,7 +13,7 @@ To run most tests on sandbox, type this: test/run

in the U-Boot directory. Note that only the pytest suite is run using this -comment. +command.

Sandbox @@ -29,7 +29,7 @@ Pytest Suite Many tests are available using the pytest suite, in test/py. This can run either on sandbox or on real hardware. It relies on the U-Boot console to inject test commands and check the result. It is slower to run than C code, -but provides the ability to unify lots of test and summarise their results. +but provides the ability to unify lots of tests and summarise their results.

You can run the tests on sandbox with:

@@ -55,11 +55,11 @@ Ad-hoc tests There are several ad-hoc tests which run outside the pytest environment:

test/fs - File system test (shell script) - test/image - FIT and lagacy image tests (shell script and Python) + test/image - FIT and legacy image tests (shell script and Python) test/stdint - A test that stdint.h can be used in U-Boot (shell script) trace - Test for the tracing feature (shell script)

-The above should be converted to run as part of the pytest suite. +TODO: Move these into pytest.

When to write tests diff --git a/test/py/tests/test_vboot.py b/test/py/tests/test_vboot.py index 14ec85b..46552fc 100644 --- a/test/py/tests/test_vboot.py +++ b/test/py/tests/test_vboot.py @@ -106,7 +106,9 @@ def test_vboot(u_boot_console):

algo = sha

- # Compile our device tree files for kernel and U-Boot + # Compile our device tree files for kernel and U-Boot. These are + # regenerated here since mkimage will modify them (by adding a + # public key) below. dtc('sandbox-kernel.dts') dtc('sandbox-u-boot.dts')

@@ -139,7 +141,7 @@ def test_vboot(u_boot_console): sig = util.cmd(cons, 'fdtget -t bx %s %s value' % (fit, sig_node)) byte_list = sig.split() byte = int(byte_list[0], 16) - byte_list = ['%x' % (byte + 1)] + byte_list[1:] + byte_list[0] = '%x' % (byte + 1) sig = ' '.join(byte_list) util.cmd(cons, 'fdtput -t bx %s %s value %s' % (fit, sig_node, sig))

This check was missed. Add it and make the message more verbose.

Signed-off-by: Simon Glass sjg@chromium.org Reported-by: Tom Rini trini@konsulko.com Fixes: 9e17b034 (test/py: Provide a way to check that a command fails) ---

test/py/multiplexed_log.py | 2 ++ test/py/u_boot_utils.py | 4 +++- 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py index 35a32fb..bf926c3 100644 --- a/test/py/multiplexed_log.py +++ b/test/py/multiplexed_log.py @@ -102,6 +102,7 @@ class RunAndLog(object): self.name = name self.chained_file = chained_file self.output = None + self.exit_status = None

def close(self): """Clean up any resources managed by this object.""" @@ -166,6 +167,7 @@ class RunAndLog(object):

# Store the output so it can be accessed if we raise an exception. self.output = output + self.exit_status = exit_status if exception: raise exception return output diff --git a/test/py/u_boot_utils.py b/test/py/u_boot_utils.py index e358c58..d71348f 100644 --- a/test/py/u_boot_utils.py +++ b/test/py/u_boot_utils.py @@ -201,9 +201,11 @@ def run_and_log_expect_exception(u_boot_console, cmd, retcode, msg): runner = u_boot_console.log.get_runner(cmd[0], sys.stdout) runner.run(cmd) except Exception as e: + assert(retcode == runner.exit_status) assert(msg in runner.output) else: - raise Exception('Expected exception, but not raised') + raise Exception("Expected an exception with retcode %d message '%s'," + "but it was not raised" % (retcode, msg)) finally: runner.close()

Fix some typos in various files introduced with the vboot test conversion.

Reported-by: Teddy Reed teddy.reed@gmail.com

Signed-off-by: Simon Glass sjg@chromium.org ---

test/py/tests/test_vboot.py | 12 ++++++------ test/py/u_boot_console_base.py | 4 ++-- test/py/u_boot_utils.py | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/test/py/tests/test_vboot.py b/test/py/tests/test_vboot.py index 46552fc..8d10710 100644 --- a/test/py/tests/test_vboot.py +++ b/test/py/tests/test_vboot.py @@ -1,4 +1,4 @@ -# Copyright (c) 2013, Google Inc. +# Copyright (c) 2016, Google Inc. # # SPDX-License-Identifier: GPL-2.0+ # @@ -16,7 +16,7 @@ For image verification: For configuration verification: - Corrupt signature and check for failure - Create FIT (with unsigned configuration) with mkimage -- Check that image veriication works +- Check that image verification works - Sign the FIT and mark the key as 'required' for verification - Check that image verification works - Corrupt the signature @@ -41,7 +41,7 @@ def test_vboot(u_boot_console): key-generation process is quite slow and we want to avoid doing it twice. """ def dtc(dts): - """Run the device-tree compiler to compile a .dts file + """Run the device tree compiler to compile a .dts file

The output file will be the same as the input file but with a .dtb extension. @@ -73,12 +73,12 @@ def test_vboot(u_boot_console): assert(expect_string in output)

def make_fit(its): - """Make a new FIT from the .its source file + """Make a new FIT from the .its source file.

This runs 'mkimage -f' to create a new FIT.

Args: - its: Filename containing .its source + its: Filename containing .its source. """ util.run_and_log(cons, [mkimage, '-D', dtc_args, '-f', '%s%s' % (datadir, its), fit]) @@ -94,7 +94,7 @@ def test_vboot(u_boot_console): '-r', fit])

def test_with_algo(sha): - """Test verified boot with the given hash algorithm + """Test verified boot with the given hash algorithm.

This is the main part of the test code. The same procedure is followed for both hashing algorithms. diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 4606ad4..356cf80 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -223,9 +223,9 @@ class ConsoleBase(object): for each command in a list.

Args: - cmd: List of commands (each a string) + cmd: List of commands (each a string). Returns: - Combined output of all commands, as a string + Combined output of all commands, as a string. """ output = '' for cmd in cmds: diff --git a/test/py/u_boot_utils.py b/test/py/u_boot_utils.py index d71348f..e74e342 100644 --- a/test/py/u_boot_utils.py +++ b/test/py/u_boot_utils.py @@ -186,7 +186,7 @@ def cmd(u_boot_console, cmd_str): return run_and_log(u_boot_console, cmd_str.split())

def run_and_log_expect_exception(u_boot_console, cmd, retcode, msg): - """Run a command which is expected to fail. + """Run a command that is expected to fail.

This runs a command and checks that it fails with the expected return code and exception method. If not, an exception is raised. @@ -195,7 +195,7 @@ def run_and_log_expect_exception(u_boot_console, cmd, retcode, msg): u_boot_console: A console connection to U-Boot. cmd: The command to run, as an array of argv[]. retcode: Expected non-zero return code from the command. - msg: String which should be contained within the command's output. + msg: String that should be contained within the command's output. """ try: runner = u_boot_console.log.get_runner(cmd[0], sys.stdout)

Instead of this, use the existing run_and_log() function, enhanced to support a command string as well as a list of arguments.

Suggested-by: Stephen Warren swarren@nvidia.com

Signed-off-by: Simon Glass sjg@chromium.org ---

test/py/tests/test_vboot.py | 22 ++++++++++++---------- test/py/u_boot_utils.py | 19 +++++-------------- 2 files changed, 17 insertions(+), 24 deletions(-)

diff --git a/test/py/tests/test_vboot.py b/test/py/tests/test_vboot.py index 8d10710..7935fae 100644 --- a/test/py/tests/test_vboot.py +++ b/test/py/tests/test_vboot.py @@ -50,8 +50,8 @@ def test_vboot(u_boot_console): dts: Device tree file to compile. """ dtb = dts.replace('.dts', '.dtb') - util.cmd(cons, 'dtc %s %s%s -O dtb ' - '-o %s%s' % (dtc_args, datadir, dts, tmpdir, dtb)) + util.run_and_log(cons, 'dtc %s %s%s -O dtb ' + '-o %s%s' % (dtc_args, datadir, dts, tmpdir, dtb))

def run_bootm(test_type, expect_string): """Run a 'bootm' command U-Boot. @@ -138,12 +138,14 @@ def test_vboot(u_boot_console): '-k', dtb])

# Increment the first byte of the signature, which should cause failure - sig = util.cmd(cons, 'fdtget -t bx %s %s value' % (fit, sig_node)) + sig = util.run_and_log(cons, 'fdtget -t bx %s %s value' % + (fit, sig_node)) byte_list = sig.split() byte = int(byte_list[0], 16) byte_list[0] = '%x' % (byte + 1) sig = ' '.join(byte_list) - util.cmd(cons, 'fdtput -t bx %s %s value %s' % (fit, sig_node, sig)) + util.run_and_log(cons, 'fdtput -t bx %s %s value %s' % + (fit, sig_node, sig))

run_bootm('Signed config with bad hash', 'Bad Data Hash')

@@ -164,14 +166,14 @@ def test_vboot(u_boot_console):

# Create an RSA key pair public_exponent = 65537 - util.cmd(cons, 'openssl genpkey -algorithm RSA -out %sdev.key ' - '-pkeyopt rsa_keygen_bits:2048 ' - '-pkeyopt rsa_keygen_pubexp:%d ' - '2>/dev/null' % (tmpdir, public_exponent)) + util.run_and_log(cons, 'openssl genpkey -algorithm RSA -out %sdev.key ' + '-pkeyopt rsa_keygen_bits:2048 ' + '-pkeyopt rsa_keygen_pubexp:%d ' + '2>/dev/null' % (tmpdir, public_exponent))

# Create a certificate containing the public key - util.cmd(cons, 'openssl req -batch -new -x509 -key %sdev.key -out ' - '%sdev.crt' % (tmpdir, tmpdir)) + util.run_and_log(cons, 'openssl req -batch -new -x509 -key %sdev.key -out ' + '%sdev.crt' % (tmpdir, tmpdir))

# Create a number kernel image with zeroes with open('%stest-kernel.bin' % tmpdir, 'w') as fd: diff --git a/test/py/u_boot_utils.py b/test/py/u_boot_utils.py index e74e342..2ba4bae 100644 --- a/test/py/u_boot_utils.py +++ b/test/py/u_boot_utils.py @@ -158,7 +158,9 @@ def run_and_log(u_boot_console, cmd, ignore_errors=False):

Args: u_boot_console: A console connection to U-Boot. - cmd: The command to run, as an array of argv[]. + cmd: The command to run, as an array of argv[], or a string. + If a string, note that it is split up so that quoted spaces + will not be preserved. E.g. "fred and" becomes ['"fred', 'and"'] ignore_errors: Indicate whether to ignore errors. If True, the function will simply return if the command cannot be executed or exits with an error code, otherwise an exception will be raised if such @@ -167,24 +169,13 @@ def run_and_log(u_boot_console, cmd, ignore_errors=False): Returns: The output as a string. """ - + if isinstance(cmd, str): + cmd = cmd.split() runner = u_boot_console.log.get_runner(cmd[0], sys.stdout) output = runner.run(cmd, ignore_errors=ignore_errors) runner.close() return output

-def cmd(u_boot_console, cmd_str): - """Run a single command string and log its output. - - Args: - u_boot_console: A console connection to U-Boot. - cmd: The command to run, as a string. - - Returns: - The output as a string. - """ - return run_and_log(u_boot_console, cmd_str.split()) - def run_and_log_expect_exception(u_boot_console, cmd, retcode, msg): """Run a command that is expected to fail.

Add a proper function for this rather than using internal functions. Use it in the single call site.

Also, do a restart at the end of the vboot test to reset to the normal device tree.

Signed-off-by: Simon Glass sjg@chromium.org Suggested-by: Stephen Warren swarren@nvidia.com ---

test/py/tests/test_vboot.py | 5 +++-- test/py/u_boot_console_base.py | 5 +++++ 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/test/py/tests/test_vboot.py b/test/py/tests/test_vboot.py index 25c3a53..d7ab439 100644 --- a/test/py/tests/test_vboot.py +++ b/test/py/tests/test_vboot.py @@ -65,8 +65,7 @@ def test_vboot(u_boot_console): sha_algo: Either 'sha1' or 'sha256', to select the algorithm to use. """ - cons.cleanup_spawn() - cons.ensure_spawned() + cons.restart_uboot() with cons.log.section('Verified boot %s %s' % (sha_algo, test_type)): output = cons.run_command_list( ['sb load hostfs - 100 %stest.fit' % tmpdir, @@ -190,4 +189,6 @@ def test_vboot(u_boot_console): test_with_algo('sha1') test_with_algo('sha256') finally: + # Go back to the original U-Boot with the correct dtb. cons.config.dtb = old_dtb + cons.restart_uboot() diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 356cf80..b855b10 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -393,6 +393,11 @@ class ConsoleBase(object): pass self.p = None

+ def restart_uboot(self): + """Shut down and restart U-Boot.""" + self.cleanup_spawn() + self.ensure_spawned() + def get_spawn_output(self): """Return the start-up output from U-Boot

Return one string for each command that was executed. This seems cleaner.

Suggested-by: Teddy Reed teddy.reed@gmail.com Signed-off-by: Simon Glass sjg@chromium.org ---

test/py/tests/test_vboot.py | 2 +- test/py/u_boot_console_base.py | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/test/py/tests/test_vboot.py b/test/py/tests/test_vboot.py index d7ab439..021892b 100644 --- a/test/py/tests/test_vboot.py +++ b/test/py/tests/test_vboot.py @@ -71,7 +71,7 @@ def test_vboot(u_boot_console): ['sb load hostfs - 100 %stest.fit' % tmpdir, 'fdt addr 100', 'bootm 100']) - assert(expect_string in output) + assert(expect_string in ''.join(output))

def make_fit(its): """Make a new FIT from the .its source file. diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index b855b10..ee9b928 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -225,11 +225,12 @@ class ConsoleBase(object): Args: cmd: List of commands (each a string). Returns: - Combined output of all commands, as a string. + A list of output strings from each command, one element for each + command. """ - output = '' + output = [] for cmd in cmds: - output += self.run_command(cmd) + output.append(self.run_command(cmd)) return output

def ctrlc(self):