+Roger, for Travis-CI

Hi Teddy,

On 3 July 2016 at 13:17, Teddy Reed teddy.reed@gmail.com wrote:

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass sjg@chromium.org wrote:

...

Add a few notes about how testing works in U-Boot.

Signed-off-by: Simon Glass sjg@chromium.org

Reviewed-by: Teddy Reed teddy.reed@gmail.com

Thanks for the comments. Will tidy up.

...

---

test/README | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 test/README

diff --git a/test/README b/test/README new file mode 100644 index 0000000..dfd83d6 --- /dev/null +++ b/test/README @@ -0,0 +1,82 @@ +Testing in U-Boot +=================

+U-Boot has a large amount of code. This file describes how this code is +tested and what tests you should write when adding a new feature.

+Sandbox +------- +U-Boot can be built as a user-space application (e.g. for Linux). This +allows test to be executed without needing target hardware. The 'sandbox' +target provides this feature and it is widely used in tests.

+Pytest Suite +------------

+Many tests are available using the pytest suite, in test/py. This can run +either on sandbox or on real hardware. It relies on the U-Boot console to +inject test commands and check the result. It is slower to run than C code, +but provides the ability to unify lots of test and summarise their results.

lots of tests

...

+You can run the tests on sandbox with:

•   ./test/py/test.py --bd sandbox --build
  

+This will produce HTML output in build-sandbox/test-log.html

+See test/py/README.md for more information about the pytest suite.

+tbot +----

+Tbot provides a way to execute tests on target hardware. It is intended for +trying out both U-Boot and Linux (and potentially other software) on a +number of boards automatically. It can be used to create a continuous test +environment. See tools/tbot/README for more information.

+Ad-hoc tests +------------

+There are several ad-hoc tests which run outside the pytest environment:

• test/fs - File system test (shell script)
• test/image - FIT and lagacy image tests (shell script and Python)

s/lagacy/legacy/

...

• test/stdint - A test that stdint.h can be used in U-Boot (shell script)
• trace - Test for the tracing feature (shell script)
• vboot - Test for verified boot (shell script)

+The above should be converted to run as part of the pytest suite.

Is this a NOTE or a TODO?

TODO - I'll reword it.

...

+When to write tests +-------------------

+If you add code to U-Boot without a test you are taking a risk. Even if you +perform thorough manual testing at the time of submission, it may break when +future changes are made to U-Boot. It may even break when applied to mainline, +if other changes interact with it. A good mindset is that untested code +probably doesn't work and should be deleted.

+You can assume that the Pytest suite will be run before patches are accepted +to mainline, so this provides protection against future breakage.

+On the other hand there is quite a bit of code that is not covered with tests, +or is covered sparingly. So here are some suggestions:

+- If you are adding a new uclass, add a sandbox driver and a test that uses it +- If you are modifying code covered by an existing test, add a new test case

• to cover your changes

+- If the code you are modifying has not tests, consider writing one. Even a

• very basic test is useful, and may be picked up and enhanced by others. It
• is much easier to add onto a test - writing a new large test can seem
• daunting to most contributors.

+Future work +-----------

+Converting existing shell scripts into pytest tests.

2.8.0.rc3.226.g39d4020

---

U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot

Thank for these docs! I wonder if a clone/mirror of the repo on Github can be set up to run the sandbox/Pytest tests in TravisCI with minimum effort? :)

There is a .travis.yml file - I have not tried this but I believe people are already doing it somewhere.

Regards. Simon

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass sjg@chromium.org wrote:

At present sandbox exits when the 'bootm' command completes, since it is not actually able to run the OS that is loaded. Normally 'bootm' failure is considered a fatal error in U-Boot.

However this is annoying for tests, which may want to examine the state after a test is complete. In any case there is a 'reset' command which can be used to exit, if required.

Change the behaviour to return normally from the 'bootm' command on sandbox.

Signed-off-by: Simon Glass sjg@chromium.org

Reviewed-by: Teddy Reed teddy.reed@gmail.com

---

arch/sandbox/lib/bootm.c | 2 +- common/bootm_os.c | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/sandbox/lib/bootm.c b/arch/sandbox/lib/bootm.c index d49c927..0c9a797 100644 --- a/arch/sandbox/lib/bootm.c +++ b/arch/sandbox/lib/bootm.c @@ -56,7 +56,7 @@ int do_bootm_linux(int flag, int argc, char *argv[], bootm_headers_t *images) bootstage_mark(BOOTSTAGE_ID_RUN_OS); printf("## Transferring control to Linux (at address %08lx)...\n", images->ep);

•           reset_cpu(0);
  

•           printf("sandbox: continuing, as we cannot run Linux\n");
    }
  
    return 0;
  

diff --git a/common/bootm_os.c b/common/bootm_os.c index 9ec84bd..e3f5a46 100644 --- a/common/bootm_os.c +++ b/common/bootm_os.c @@ -481,6 +481,7 @@ int boot_selected_os(int argc, char * const argv[], int state,

    /* Stand-alone may return when 'autostart' is 'no' */
    if (images->os.type == IH_TYPE_STANDALONE ||

•       IS_ENABLED(CONFIG_SANDBOX) ||
        state == BOOTM_STATE_OS_FAKE_GO) /* We expect to return */
            return 0;
    bootstage_error(BOOTSTAGE_ID_BOOT_OS_RETURNED);
  

-- 2.8.0.rc3.226.g39d4020

---

U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot

This will help a lot with vboot testing, thanks!

This may affect others with existing sandbox tests set up in a CI/CB. But I feel this is the right approach since it now allows fallback (corrupted recovery) / secondary boot media testing too.

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass sjg@chromium.org wrote:

Tests may want to look at the output from running a command. Return it so that this is possible.

Signed-off-by: Simon Glass sjg@chromium.org

Reviewed-by: Teddy Reed teddy.reed@gmail.com

---

test/py/multiplexed_log.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py index 68917eb..02c44df 100644 --- a/test/py/multiplexed_log.py +++ b/test/py/multiplexed_log.py @@ -119,7 +119,7 @@ class RunAndLog(object): raised if such problems occur.

     Returns:

•        Nothing.
  

•        The output as a string.
     """
  
     msg = '+' + ' '.join(cmd) + '\n'
  

@@ -161,6 +161,7 @@ class RunAndLog(object): self.chained_file.write(output) if exception: raise exception

•    return output
  
  

class SectionCtxMgr(object): """A context manager for Python's "with" statement, which allows a certain -- 2.8.0.rc3.226.g39d4020

---

U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass sjg@chromium.org wrote:

It is useful to be able to obtain the output from a command. Return it from this function.

Signed-off-by: Simon Glass sjg@chromium.org

Reviewed-by: Teddy Reed teddy.reed@gmail.com

---

test/py/u_boot_utils.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/test/py/u_boot_utils.py b/test/py/u_boot_utils.py index 6a6b2ec..5dc0f71 100644 --- a/test/py/u_boot_utils.py +++ b/test/py/u_boot_utils.py @@ -165,12 +165,13 @@ def run_and_log(u_boot_console, cmd, ignore_errors=False): problems occur.

 Returns:

•    Nothing.
  

•    The output as a string.
  

  """

  runner = u_boot_console.log.get_runner(cmd[0], sys.stdout)

• runner.run(cmd, ignore_errors=ignore_errors)

• output = runner.run(cmd, ignore_errors=ignore_errors) runner.close()
• return output

ram_base = None def find_ram_base(u_boot_console): -- 2.8.0.rc3.226.g39d4020

---

U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass sjg@chromium.org wrote:

Sometimes we want to run a command and check that it fails. Add a function to handle this. It can check the return code and also make sure that the output contains a given error message.

Signed-off-by: Simon Glass sjg@chromium.org

test/py/u_boot_utils.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)

diff --git a/test/py/u_boot_utils.py b/test/py/u_boot_utils.py index 5d638d9..c834e7b 100644 --- a/test/py/u_boot_utils.py +++ b/test/py/u_boot_utils.py @@ -185,6 +185,28 @@ def cmd(u_boot_console, cmd_str): """ return run_and_log(u_boot_console, cmd_str.split())

+def run_and_log_expect_exception(u_boot_console, cmd, retcode, msg):

• """Run a command which is expected to fail.

nit, in the restrictive case, s/which/that/

• This runs a command and checks that it fails with the expected return code
• and exception method. If not, an exception is raised.
• Args:

•    u_boot_console: A console connection to U-Boot.
  

•    cmd: The command to run, as an array of argv[].
  

•    retcode: Expected non-zero return code from the command.
  

•    msg: String which should be contained within the command's output.
  

nit, same as above

• """
• try:

•    runner = u_boot_console.log.get_runner(cmd[0], sys.stdout)
  

•    runner.run(cmd)
  

• except Exception as e:

•    assert(msg in runner.output)
  

I understand why this is a partial match, as requiring a complete match would result in changing several test cases every time debug output was altered. Not fun.

That said, it seems dangerous. When performing failure tests there are often several failing side-effects. For example, if you are testing hash-mismatches in vboot several code paths may complain about comparison failures, partial output matching may not be able to isolate the failing logic. You may encounter a false positive if any side-effect emits the same expected output.

Would optional parameters make sense? They could allow a tester to restrict matching to the last line of output, or request a complete string match?

• else:

•    raise Exception('Expected exception, but not raised')
  

This is a bit vague, does it make sense to include the expected (failing) retcode too?

• finally:

•    runner.close()
  

ram_base = None def find_ram_base(u_boot_console): """Find the running U-Boot's RAM location. -- 2.8.0.rc3.226.g39d4020

---

U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot

Let me know if my comments about output matching are too pedantic. Since there are few existing test cases, the last thing I want to do is over-engineer some helpful testing facilities. :)

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass sjg@chromium.org wrote:

Some tests want to execute a sequence of commands. Add a helper for this.

Signed-off-by: Simon Glass sjg@chromium.org

Reviewed-by: Teddy Reed teddy.reed@gmail.com

---

test/py/u_boot_console_base.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)

diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 815fa64..b5aad7c 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -216,6 +216,22 @@ class ConsoleBase(object): self.cleanup_spawn() raise

• def run_command_list(self, cmds):

•    """Run a list of commands.
  

•    This is a helper function to call run_command() with default arguments
  

•    for each command in a list.
  

•    Args:
  

•        cmd: List of commands (each a string)
  

nit, in most other docstrings, these Args and Return lines have punctuation.

•    Returns:
  

•        Combined output of all commands, as a string
  

•    """
  

•    output = ''
  

•    for cmd in cmds:
  

•        output += self.run_command(cmd)
  

Although this fits the prototypes for other run_* functions, a list of output strings is much easier to test and reason about. :/

•    return output
  

• def ctrlc(self): """Send a CTRL-C character to U-Boot.

-- 2.8.0.rc3.226.g39d4020

---

U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass sjg@chromium.org wrote:

The error code may provide useful information for debugging. Add it to the error string.

Signed-off-by: Simon Glass sjg@chromium.org

Reviewed-by: Teddy Reed teddy.reed@gmail.com

---

tools/fit_image.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/fit_image.c b/tools/fit_image.c index 58aa8e2..f471982 100644 --- a/tools/fit_image.c +++ b/tools/fit_image.c @@ -650,8 +650,8 @@ static int fit_handle_file(struct image_tool_params *params) }

    if (ret) {

•           fprintf(stderr, "%s Can't add hashes to FIT blob\n",
  

•                   params->cmdname);
  

•           fprintf(stderr, "%s Can't add hashes to FIT blob: %d\n",
  

•                   params->cmdname, ret);
            goto err_system;
    }
  
  

-- 2.8.0.rc3.226.g39d4020

Simple change, should help debugging/dev! -Teddy

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass sjg@chromium.org wrote:

At present all the hush tests are skipped on sandbox because the test thinks that this option is disabled. In fact it has just been renamed.

It might be better to use the full CONFIG_xxx name in tests with @pytest.mark.buildconfigspec(), since at present it is not really clear that the options are related.

Fixes: f1f9d4fa (hush: complete renaming CONFIG_SYS_HUSH_PARSER to CONFIG_HUSH_PARSER)

Signed-off-by: Simon Glass sjg@chromium.org

Reviewed-by: Teddy Reed teddy.reed@gmail.com

---

test/py/tests/test_hush_if_test.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/test/py/tests/test_hush_if_test.py b/test/py/tests/test_hush_if_test.py index 1eeaa5b..b572538 100644 --- a/test/py/tests/test_hush_if_test.py +++ b/test/py/tests/test_hush_if_test.py @@ -109,27 +109,27 @@ def exec_hush_if(u_boot_console, expr, result): response = u_boot_console.run_command(cmd) assert response.strip() == str(result).lower()

-@pytest.mark.buildconfigspec('sys_hush_parser') +@pytest.mark.buildconfigspec('hush_parser') def test_hush_if_test_setup(u_boot_console): """Set up environment variables used during the "if" tests."""

 u_boot_console.run_command('setenv ut_var_nonexistent')
 u_boot_console.run_command('setenv ut_var_exists 1')

-@pytest.mark.buildconfigspec('sys_hush_parser') +@pytest.mark.buildconfigspec('hush_parser') @pytest.mark.parametrize('expr,result', subtests) def test_hush_if_test(u_boot_console, expr, result): """Test a single "if test" condition."""

 exec_hush_if(u_boot_console, expr, result)

-@pytest.mark.buildconfigspec('sys_hush_parser') +@pytest.mark.buildconfigspec('hush_parser') def test_hush_if_test_teardown(u_boot_console): """Clean up environment variables used during the "if" tests."""

 u_boot_console.run_command('setenv ut_var_exists')

-@pytest.mark.buildconfigspec('sys_hush_parser') +@pytest.mark.buildconfigspec('hush_parser') # We might test this on real filesystems via UMS, DFU, 'save', etc. # Of those, only UMS currently allows file removal though. @pytest.mark.boardspec('sandbox') -- 2.8.0.rc3.226.g39d4020

---

U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot

Awesome, thanks!

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass sjg@chromium.org wrote:

Now that we have a suitable test framework we should move all tests into it. The vboot test is a suitable candidate. Rewrite it in Python and move the data files into an appropriate directory.

Signed-off-by: Simon Glass sjg@chromium.org

test/README | 1 - test/py/tests/test_vboot.py | 185 ++++++++++++++++++++++ test/{ => py/tests}/vboot/sandbox-kernel.dts | 0 test/{ => py/tests}/vboot/sandbox-u-boot.dts | 0 test/{ => py/tests}/vboot/sign-configs-sha1.its | 0 test/{ => py/tests}/vboot/sign-configs-sha256.its | 0 test/{ => py/tests}/vboot/sign-images-sha1.its | 0 test/{ => py/tests}/vboot/sign-images-sha256.its | 0 test/vboot/.gitignore | 3 - test/vboot/vboot_test.sh | 151 ------------------ 10 files changed, 185 insertions(+), 155 deletions(-) create mode 100644 test/py/tests/test_vboot.py rename test/{ => py/tests}/vboot/sandbox-kernel.dts (100%) rename test/{ => py/tests}/vboot/sandbox-u-boot.dts (100%) rename test/{ => py/tests}/vboot/sign-configs-sha1.its (100%) rename test/{ => py/tests}/vboot/sign-configs-sha256.its (100%) rename test/{ => py/tests}/vboot/sign-images-sha1.its (100%) rename test/{ => py/tests}/vboot/sign-images-sha256.its (100%) delete mode 100644 test/vboot/.gitignore delete mode 100755 test/vboot/vboot_test.sh

diff --git a/test/README b/test/README index 6cfee05..ee55972 100644 --- a/test/README +++ b/test/README @@ -58,7 +58,6 @@ There are several ad-hoc tests which run outside the pytest environment: test/image - FIT and lagacy image tests (shell script and Python) test/stdint - A test that stdint.h can be used in U-Boot (shell script) trace - Test for the tracing feature (shell script)

• vboot - Test for verified boot (shell script)

The above should be converted to run as part of the pytest suite.

diff --git a/test/py/tests/test_vboot.py b/test/py/tests/test_vboot.py new file mode 100644 index 0000000..c779895 --- /dev/null +++ b/test/py/tests/test_vboot.py @@ -0,0 +1,185 @@ +# Copyright (c) 2013, Google Inc. +# +# SPDX-License-Identifier: GPL-2.0+ +# +# U-Boot Verified Boot Test

+""" +This tests verified boot in the following ways:

+For image verification: +- Create FIT (unsigned) with mkimage +- Check that verification shows that no keys are verified +- Sign image +- Check that verification shows that a key is now verified

+For configuration verification: +- Corrupt signature and check for failure +- Create FIT (with unsigned configuration) with mkimage +- Check that image veriication works

verification

+- Sign the FIT and mark the key as 'required' for verification +- Check that image verification works +- Corrupt the signature +- Check that image verification no-longer works

+Tests run with both SHA1 and SHA256 hashing. +"""

+import pytest +import sys +import u_boot_utils as util

+@pytest.mark.buildconfigspec('fit_signature') +def test_vboot(u_boot_console):

• """Test verified boot signing with mkimage and verification with 'bootm'.
• This works using sandbox only as it needs to update the device tree used
• by U-Boot to hold public keys from the signing process.
• The SHA1 and SHA256 tests are combined into a single test since the
• key-generation process is quite slow and we want to avoid doing it twice.
• """
• def dtc(dts):

•    """Run the device-tree compiler to compile a .dts file
  

In a few other places it reads: "device tree" without a dash. ;)

•    The output file will be the same as the input file but with a .dtb
  

•    extension.
  

•    Args:
  

•        dts: Device tree file to compile.
  

•    """
  

•    dtb = dts.replace('.dts', '.dtb')
  

•    util.cmd(cons, 'dtc %s %s%s -O dtb '
  

•                   '-o %s%s' % (dtc_args, datadir, dts, tmpdir, dtb))
  

• def run_bootm(test_type, expect_string):

•    """Run a 'bootm' command U-Boot.
  

•    This always starts a fresh U-Boot instance since the device tree may
  

•    contain a new public key.
  

•    Args:
  

•        test_type: A string identifying the test type
  

•        expect_string: A string which is expected in the output
  

nit, punctuation.

•    """
  

•    cons.cleanup_spawn()
  

•    cons.ensure_spawned()
  

•    cons.log.action('%s: Test Verified Boot Run: %s' % (algo, test_type))
  

•    output = cons.run_command_list(
  

•        ['sb load hostfs - 100 %stest.fit' % tmpdir,
  

•         'fdt addr 100',
  

•         'bootm 100'])
  

•    assert(expect_string in output)
  

• def make_fit(its):

•    """Make a new FIT from the .its source file
  

•    This runs 'mkimage -f' to create a new FIT.
  

•    Args:
  

•        its: Filename containing .its source
  

nit, same

•    """
  

•    util.run_and_log(cons, [mkimage, '-D', dtc_args, '-f',
  

•                            '%s%s' % (datadir, its), fit])
  

• def sign_fit():

•    """Sign the FIT
  

•    Signs the FIT and writes the signature into it. It also writes the
  

•    public key into the dtb.
  

•    """
  

•    cons.log.action('%s: Sign images' % algo)
  

•    util.run_and_log(cons, [mkimage, '-F', '-k', tmpdir, '-K', dtb,
  

•                            '-r', fit])
  

• def test_with_algo(sha):

nit, maybe sha_algo?

•    """Test verified boot with the given hash algorithm
  

nit, same

•    This is the main part of the test code. The same procedure is followed
  

•    for both hashing algorithms.
  

•    Args:
  

•        sha: Either 'sha1' or 'sha256', to select the algorithm to use
  

•    """
  

•    global algo
  

•    algo = sha
  

•    # Compile our device tree files for kernel and U-Boot
  

•    dtc('sandbox-kernel.dts')
  

•    dtc('sandbox-u-boot.dts')
  

•    # Build the FIT, but don't sign anything yet
  

•    cons.log.action('%s: Test FIT with signed images' % algo)
  

•    make_fit('sign-images-%s.its' % algo)
  

•    run_bootm('unsigned images', 'dev-')
  

•    # Sign images with our dev keys
  

•    sign_fit()
  

•    run_bootm('signed images', 'dev+')
  

•    # Create a fresh .dtb without the public keys
  

•    dtc('sandbox-u-boot.dts')
  

•    cons.log.action('%s: Test FIT with signed configuration' % algo)
  

•    make_fit('sign-configs-%s.its' % algo)
  

•    run_bootm('unsigned config', '%s+ OK' % algo)
  

•    # Sign images with our dev keys
  

•    sign_fit()
  

•    run_bootm('signed config', 'dev+')
  

•    cons.log.action('%s: Check signed config on the host' % algo)
  

•    util.run_and_log(cons, [fit_check_sign, '-f', fit, '-k', tmpdir,
  

•                            '-k', dtb])
  

•    # Increment the first byte of the signature, which should cause failure
  

•    sig = util.cmd(cons, 'fdtget -t bx %s %s value' % (fit, sig_node))
  

•    byte_list = sig.split()
  

•    byte = int(byte_list[0], 16)
  

•    byte_list = ['%x' % (byte + 1)] + byte_list[1:]
  

•    sig = ' '.join(byte_list)
  

•    util.cmd(cons, 'fdtput -t bx %s %s value %s' % (fit, sig_node, sig))
  

•    run_bootm('Signed config with bad hash', 'Bad Data Hash')
  

•    cons.log.action('%s: Check bad config on the host' % algo)
  

•    util.run_and_log_expect_exception(cons, [fit_check_sign, '-f', fit,
  

•            '-k', dtb], 1, 'Failed to verify required signature')
  

• cons = u_boot_console
• tmpdir = cons.config.result_dir + '/'
• tmp = tmpdir + 'vboot.tmp'

Is there a need to clean these up afterward?

Python's tempfile might be helpful, you can supply the result_dir as the directory prefix.

• datadir = 'test/py/tests/vboot/'
• fit = '%stest.fit' % tmpdir
• mkimage = cons.config.build_dir + '/tools/mkimage'
• fit_check_sign = cons.config.build_dir + '/tools/fit_check_sign'
• dtc_args = '-I dts -O dtb -i %s' % tmpdir
• dtb = '%ssandbox-u-boot.dtb' % tmpdir
• sig_node = '/configurations/conf@1/signature@1'

If these variables are used throughout the tests like globals, should they be DATADIR, MKIMAGE, etc?

• # Create an RSA key pair
• public_exponent = 65537
• util.cmd(cons, 'openssl genpkey -algorithm RSA -out %sdev.key '

•               '-pkeyopt rsa_keygen_bits:2048 '
  

•               '-pkeyopt rsa_keygen_pubexp:%d '
  

•               '2>/dev/null'  % (tmpdir, public_exponent))
  

• # Create a certificate containing the public key
• util.cmd(cons, 'openssl req -batch -new -x509 -key %sdev.key -out '

•               '%sdev.crt' % (tmpdir, tmpdir))
  

• # Create a number kernel image with zeroes
• with open('%stest-kernel.bin' % tmpdir, 'w') as fd:

•    fd.write(5000 * chr(0))
  

Any need to clean these up afterward or place them into test-run-unique directories? Is the expectation that subsequent tests will overwrite existing test data (also that vboot tests cannot run concurrently)?

• try:

•    # We need to use our own device tree file. Remember to restore it
  

•    # afterwards.
  

•    old_dtb = cons.config.dtb
  

•    cons.config.dtb = dtb
  

•    test_with_algo('sha1')
  

•    test_with_algo('sha256')
  

• finally:

•    cons.config.dtb = old_dtb
  

diff --git a/test/vboot/sandbox-kernel.dts b/test/py/tests/vboot/sandbox-kernel.dts similarity index 100% rename from test/vboot/sandbox-kernel.dts rename to test/py/tests/vboot/sandbox-kernel.dts diff --git a/test/vboot/sandbox-u-boot.dts b/test/py/tests/vboot/sandbox-u-boot.dts similarity index 100% rename from test/vboot/sandbox-u-boot.dts rename to test/py/tests/vboot/sandbox-u-boot.dts diff --git a/test/vboot/sign-configs-sha1.its b/test/py/tests/vboot/sign-configs-sha1.its similarity index 100% rename from test/vboot/sign-configs-sha1.its rename to test/py/tests/vboot/sign-configs-sha1.its diff --git a/test/vboot/sign-configs-sha256.its b/test/py/tests/vboot/sign-configs-sha256.its similarity index 100% rename from test/vboot/sign-configs-sha256.its rename to test/py/tests/vboot/sign-configs-sha256.its diff --git a/test/vboot/sign-images-sha1.its b/test/py/tests/vboot/sign-images-sha1.its similarity index 100% rename from test/vboot/sign-images-sha1.its rename to test/py/tests/vboot/sign-images-sha1.its diff --git a/test/vboot/sign-images-sha256.its b/test/py/tests/vboot/sign-images-sha256.its similarity index 100% rename from test/vboot/sign-images-sha256.its rename to test/py/tests/vboot/sign-images-sha256.its diff --git a/test/vboot/.gitignore b/test/vboot/.gitignore deleted file mode 100644 index 4631242..0000000 --- a/test/vboot/.gitignore +++ /dev/null @@ -1,3 +0,0 @@ -/*.dtb -/test.fit -/dev-keys diff --git a/test/vboot/vboot_test.sh b/test/vboot/vboot_test.sh deleted file mode 100755 index 6d7abb8..0000000 --- a/test/vboot/vboot_test.sh +++ /dev/null @@ -1,151 +0,0 @@ -#!/bin/bash -# -# Copyright (c) 2013, Google Inc. -# -# Simple Verified Boot Test Script -# -# SPDX-License-Identifier: GPL-2.0+

-set -e

-# Run U-Boot and report the result -# Args: -# $1: Test message -run_uboot() {

•   echo -n "Test Verified Boot Run: $1: "
  

•   ${uboot} -d sandbox-u-boot.dtb >${tmp} -c '
  

-sb load hostfs - 100 test.fit; -fdt addr 100; -bootm 100; -reset'

•   if ! grep -q "$2" ${tmp}; then
  

•           echo
  

•           echo "Verified boot key check failed, output follows:"
  

•           cat ${tmp}
  

•           false
  

•   else
  

•           echo "OK"
  

•   fi
  

-}

-echo "Simple Verified Boot Test" -echo "=========================" -echo -echo "Please see doc/uImage.FIT/verified-boot.txt for more information" -echo

-err=0 -tmp=/tmp/vboot_test.$$

-dir=$(dirname $0)

-if [ -z ${O} ]; then

•   O=.
  

-fi -O=$(readlink -f ${O})

-dtc="-I dts -O dtb -p 2000" -uboot="${O}/u-boot" -mkimage="${O}/tools/mkimage" -fit_check_sign="${O}/tools/fit_check_sign" -keys="${dir}/dev-keys" -echo ${mkimage} -D "${dtc}"

-echo "Build keys" -mkdir -p ${keys}

-PUBLIC_EXPONENT=${1}

-if [ -z "${PUBLIC_EXPONENT}" ]; then

•   PUBLIC_EXPONENT=65537
  

-fi

-# Create an RSA key pair -openssl genpkey -algorithm RSA -out ${keys}/dev.key \

• -pkeyopt rsa_keygen_bits:2048 \
• -pkeyopt rsa_keygen_pubexp:${PUBLIC_EXPONENT} 2>/dev/null

-# Create a certificate containing the public key -openssl req -batch -new -x509 -key ${keys}/dev.key -out ${keys}/dev.crt

-pushd ${dir} >/dev/null

-function do_test {

•   echo do $sha test
  

•   # Compile our device tree files for kernel and U-Boot
  

•   dtc -p 0x1000 sandbox-kernel.dts -O dtb -o sandbox-kernel.dtb
  

•   dtc -p 0x1000 sandbox-u-boot.dts -O dtb -o sandbox-u-boot.dtb
  

•   # Create a number kernel image with zeroes
  

•   head -c 5000 /dev/zero >test-kernel.bin
  

•   # Build the FIT, but don't sign anything yet
  

•   echo Build FIT with signed images
  

•   ${mkimage} -D "${dtc}" -f sign-images-$sha.its test.fit >${tmp}
  

•   run_uboot "unsigned signatures:" "dev-"
  

•   # Sign images with our dev keys
  

•   echo Sign images
  

•   ${mkimage} -D "${dtc}" -F -k dev-keys -K sandbox-u-boot.dtb \
  

•           -r test.fit >${tmp}
  

•   run_uboot "signed images" "dev+"
  

•   # Create a fresh .dtb without the public keys
  

•   dtc -p 0x1000 sandbox-u-boot.dts -O dtb -o sandbox-u-boot.dtb
  

•   echo Build FIT with signed configuration
  

•   ${mkimage} -D "${dtc}" -f sign-configs-$sha.its test.fit >${tmp}
  

•   run_uboot "unsigned config" $sha"+ OK"
  

•   # Sign images with our dev keys
  

•   echo Sign images
  

•   ${mkimage} -D "${dtc}" -F -k dev-keys -K sandbox-u-boot.dtb \
  

•           -r test.fit >${tmp}
  

•   run_uboot "signed config" "dev+"
  

•   echo check signed config on the host
  

•   if ! ${fit_check_sign} -f test.fit -k sandbox-u-boot.dtb >${tmp}; then
  

•           echo
  

•           echo "Verified boot key check on host failed, output follows:"
  

•           cat ${tmp}
  

•           false
  

•   else
  

•           if ! grep -q "dev+" ${tmp}; then
  

•                   echo
  

•                   echo "Verified boot key check failed, output follows:"
  

•                   cat ${tmp}
  

•                   false
  

•           else
  

•                   echo "OK"
  

•           fi
  

•   fi
  

•   run_uboot "signed config" "dev+"
  

•   # Increment the first byte of the signature, which should cause failure
  

•   sig=$(fdtget -t bx test.fit /configurations/conf@1/signature@1 value)
  

•   newbyte=$(printf %x $((0x${sig:0:2} + 1)))
  

•   sig="${newbyte} ${sig:2}"
  

•   fdtput -t bx test.fit /configurations/conf@1/signature@1 value ${sig}
  

•   run_uboot "signed config with bad hash" "Bad Data Hash"
  

-}

-sha=sha1 -do_test -sha=sha256 -do_test

-popd >/dev/null

-echo -if ${ok}; then

•   echo "Test passed"
  

-else

•   echo "Test failed"
  

-fi

2.8.0.rc3.226.g39d4020

---

U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot

Thanks for this refactor! If the comments related to the sh to Python are too nit-picky we can certainly change and expand the test harness within additional patches later.