[U-Boot] [PATCH 1/8] test/py: fix timeout to be absolute

From: Stephen Warren swarren@nvidia.com
Currently, Spawn.expect() imposes its timeout solely upon receipt of new data, not on its overall operation. In theory, this could cause the timeout not to fire if U-Boot continually generated output that did not match the expected patterns.
Fix the code to additionally impose a timeout on overall operation, which is the intended mode of operation.
Signed-off-by: Stephen Warren swarren@nvidia.com --- test/py/u_boot_spawn.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py index 1baee63df25c..df4c67597cab 100644 --- a/test/py/u_boot_spawn.py +++ b/test/py/u_boot_spawn.py @@ -122,6 +122,7 @@ class Spawn(object): if type(patterns[pi]) == type(''): patterns[pi] = re.compile(patterns[pi])
+ tstart_s = time.time() try: while True: earliest_m = None @@ -142,7 +143,11 @@ class Spawn(object): self.after = self.buf[pos:posafter] self.buf = self.buf[posafter:] return earliest_pi - events = self.poll.poll(self.timeout) + tnow_s = time.time() + tdelta_ms = (tnow_s - tstart_s) * 1000 + if tdelta_ms > self.timeout: + raise Timeout() + events = self.poll.poll(self.timeout - tdelta_ms) if not events: raise Timeout() c = os.read(self.fd, 1024)

From: Stephen Warren swarren@nvidia.com
Prior to this change, U-Boot was lazilly (re-)spawned if/when a test attempted to interact with it, and no active connection existed. This approach was simple, yet had the disadvantage that U-Boot might be spawned in the middle of a test function, e.g. after the test had already performed actions such as creating data files, etc. In that case, this could cause the log to contain the sequence (1) some test logs, (2) U-Boot's boot process, (3) the rest of that test's logs. This isn't optimally readable. This issue will affect the upcoming DFU and enhanced UMS tests.
This change converts u_boot_console to be a function-scoped fixture, so that pytest attempts to re-create the object for each test invocation. This allows the fixture factory function to ensure that U-Boot is spawned prior to every test. In practice, the same object is returned each time so there is essentially no additional overhead due to this change.
This allows us to remove:
- The explicit ensure_spawned() call from test_sleep, since the core now ensures that the spawn happens before the test code is executed.
- The laxy calls to ensure_spawned() in the u_boot_console_* implementations.
The one downside is that test_env's "state_ttest_env" fixture must be converted to a function-scoped fixture too, since a module-scoped fixture cannot use a function-scoped fixture. To avoid overhead, we use the same trick of returning the same object each time.
Signed-off-by: Stephen Warren swarren@nvidia.com --- test/py/conftest.py | 3 ++- test/py/tests/test_env.py | 8 ++++++-- test/py/tests/test_sandbox_exit.py | 2 -- test/py/tests/test_sleep.py | 4 ---- test/py/u_boot_console_base.py | 2 -- test/py/u_boot_console_sandbox.py | 1 - 6 files changed, 8 insertions(+), 12 deletions(-)
diff --git a/test/py/conftest.py b/test/py/conftest.py index e1674dfce053..38aa3f922a86 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -227,7 +227,7 @@ def pytest_generate_tests(metafunc): vals = subconfig.get(fn + 's', []) metafunc.parametrize(fn, vals)
-@pytest.fixture(scope='session') +@pytest.fixture(scope='function') def u_boot_console(request): '''Generate the value of a test's u_boot_console fixture.
@@ -238,6 +238,7 @@ def u_boot_console(request): The fixture value. '''
+ console.ensure_spawned() return console
tests_not_run = set() diff --git a/test/py/tests/test_env.py b/test/py/tests/test_env.py index a3e8dd30330b..557c3afe5c0a 100644 --- a/test/py/tests/test_env.py +++ b/test/py/tests/test_env.py @@ -77,11 +77,15 @@ class StateTestEnv(object): return var n += 1
-@pytest.fixture(scope='module') +ste = None +@pytest.fixture(scope='function') def state_test_env(u_boot_console): '''pytest fixture to provide a StateTestEnv object to tests.'''
- return StateTestEnv(u_boot_console) + global ste + if not ste: + ste = StateTestEnv(u_boot_console) + return ste
def unset_var(state_test_env, var): '''Unset an environment variable. diff --git a/test/py/tests/test_sandbox_exit.py b/test/py/tests/test_sandbox_exit.py index 2aa8eb4abc68..1ec3607eb28a 100644 --- a/test/py/tests/test_sandbox_exit.py +++ b/test/py/tests/test_sandbox_exit.py @@ -13,7 +13,6 @@ def test_reset(u_boot_console):
u_boot_console.run_command('reset', wait_for_prompt=False) assert(u_boot_console.validate_exited()) - u_boot_console.ensure_spawned()
@pytest.mark.boardspec('sandbox') def test_ctrl_c(u_boot_console): @@ -21,4 +20,3 @@ def test_ctrl_c(u_boot_console):
u_boot_console.kill(signal.SIGINT) assert(u_boot_console.validate_exited()) - u_boot_console.ensure_spawned() diff --git a/test/py/tests/test_sleep.py b/test/py/tests/test_sleep.py index 64f1ddf9a09f..437b6bb9fea9 100644 --- a/test/py/tests/test_sleep.py +++ b/test/py/tests/test_sleep.py @@ -9,10 +9,6 @@ def test_sleep(u_boot_console): '''Test the sleep command, and validate that it sleeps for approximately the correct amount of time.'''
- # Do this before we time anything, to make sure U-Boot is already running. - # Otherwise, the system boot time is included in the time measurement. - u_boot_console.ensure_spawned() - # 3s isn't too long, but is enough to cross a few second boundaries. sleep_time = 3 tstart = time.time() diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 520f9a9e9f31..10fe3dbdd372 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -144,8 +144,6 @@ class ConsoleBase(object): command string and emitted the subsequent command prompts. '''
- self.ensure_spawned() - if self.at_prompt and \ self.at_prompt_logevt != self.logstream.logfile.cur_evt: self.logstream.write(self.prompt, implicit=True) diff --git a/test/py/u_boot_console_sandbox.py b/test/py/u_boot_console_sandbox.py index 88b137e8c3a1..eb84150a1e44 100644 --- a/test/py/u_boot_console_sandbox.py +++ b/test/py/u_boot_console_sandbox.py @@ -51,7 +51,6 @@ class ConsoleSandbox(ConsoleBase): Nothing. '''
- self.ensure_spawned() self.log.action('kill %d' % sig) self.p.kill(sig)

On 20 January 2016 at 15:15, Stephen Warren swarren@wwwdotorg.org wrote:
From: Stephen Warren swarren@nvidia.com
Prior to this change, U-Boot was lazilly (re-)spawned if/when a test attempted to interact with it, and no active connection existed. This approach was simple, yet had the disadvantage that U-Boot might be spawned in the middle of a test function, e.g. after the test had already performed actions such as creating data files, etc. In that case, this could cause the log to contain the sequence (1) some test logs, (2) U-Boot's boot process, (3) the rest of that test's logs. This isn't optimally readable. This issue will affect the upcoming DFU and enhanced UMS tests.
This change converts u_boot_console to be a function-scoped fixture, so that pytest attempts to re-create the object for each test invocation. This allows the fixture factory function to ensure that U-Boot is spawned prior to every test. In practice, the same object is returned each time so there is essentially no additional overhead due to this change.
This allows us to remove:
- The explicit ensure_spawned() call from test_sleep, since the core now
ensures that the spawn happens before the test code is executed.
- The laxy calls to ensure_spawned() in the u_boot_console_*
implementations.
The one downside is that test_env's "state_ttest_env" fixture must be converted to a function-scoped fixture too, since a module-scoped fixture cannot use a function-scoped fixture. To avoid overhead, we use the same trick of returning the same object each time.
Signed-off-by: Stephen Warren swarren@nvidia.com
test/py/conftest.py | 3 ++- test/py/tests/test_env.py | 8 ++++++-- test/py/tests/test_sandbox_exit.py | 2 -- test/py/tests/test_sleep.py | 4 ---- test/py/u_boot_console_base.py | 2 -- test/py/u_boot_console_sandbox.py | 1 - 6 files changed, 8 insertions(+), 12 deletions(-)
Acked-by: Simon Glass sjg@chromium.org

From: Stephen Warren swarren@nvidia.com
Tests may fail for a number of reasons, and in particular for reasons other than a timeout waiting for U-Boot to print expected data. If the last operation that a failed test performs is not waiting for U-Boot to print something, then any trailing output from U-Boot during that test's operation will not be logged as part of that test, but rather either along with the next test, or even thrown away, potentiall hiding clues re: the test failure reason.
Solve this by explicitly draining (and hence logging) the U-Boot output in the case of failed tests.
Signed-off-by: Stephen Warren swarren@nvidia.com --- test/py/conftest.py | 1 + test/py/u_boot_console_base.py | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
diff --git a/test/py/conftest.py b/test/py/conftest.py index 38aa3f922a86..c1f19cee65a3 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -386,6 +386,7 @@ def pytest_runtest_protocol(item, nextitem): skipped = report
if failed: + console.drain_console() tests_failed.add(item.name) elif skipped: tests_skipped.add(item.name) diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 10fe3dbdd372..418a26bb8e40 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -14,6 +14,7 @@ import os import pytest import re import sys +import u_boot_spawn
# Regexes for text we expect U-Boot to send to the console. pattern_u_boot_spl_signon = re.compile('(U-Boot SPL \d{4}\.\d{2}-[^\r\n]*)') @@ -213,6 +214,43 @@ class ConsoleBase(object):
self.run_command(chr(3), wait_for_echo=False, send_nl=False)
+ def drain_console(self): + '''Read from and log the U-Boot console for a short time. + + U-Boot's console output is only logged when the test code actively + waits for U-Boot to emit specific data. There are cases where tests + can fail without doing this. For example, if a test asks U-Boot to + enable USB device mode, then polls until a host-side device node + exists. In such a case, it is useful to log U-Boot's console output + in case U-Boot printed clues as to why the host-side even did not + occur. This function will do that. + + Args: + None. + + Returns: + Nothing. + ''' + + # If we are already not connected to U-Boot, there's nothing to drain. + # This should only happen when a previous call to run_command() or + # wait_for() failed (and hence the output has already been logged), or + # the system is shutting down. + if not self.p: + return + + orig_timeout = self.p.timeout + try: + # Drain the log for a relatively short time. + self.p.timeout = 1000 + # Wait for something U-Boot will likely never send. This will + # cause the console output to be read and logged. + self.p.expect(['This should never match U-Boot output']) + except u_boot_spawn.Timeout: + pass + finally: + self.p.timeout = orig_timeout + def ensure_spawned(self): '''Ensure a connection to a correctly running U-Boot instance.

On 20 January 2016 at 15:15, Stephen Warren swarren@wwwdotorg.org wrote:
From: Stephen Warren swarren@nvidia.com
Tests may fail for a number of reasons, and in particular for reasons other than a timeout waiting for U-Boot to print expected data. If the last operation that a failed test performs is not waiting for U-Boot to print something, then any trailing output from U-Boot during that test's operation will not be logged as part of that test, but rather either along with the next test, or even thrown away, potentiall hiding clues re: the test failure reason.
Solve this by explicitly draining (and hence logging) the U-Boot output in the case of failed tests.
Signed-off-by: Stephen Warren swarren@nvidia.com
test/py/conftest.py | 1 + test/py/u_boot_console_base.py | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
Acked-by: Simon Glass sjg@chromium.org

From: Stephen Warren swarren@nvidia.com
Write a note to the log file when a test sends CTRL-C to U-Boot. This makes it easier to follow what's happening in the logs, especially since U-Boot doesn't echo the character back to its output, so there's no other signal of what's going on.
Signed-off-by: Stephen Warren swarren@nvidia.com --- test/py/u_boot_console_base.py | 1 + 1 file changed, 1 insertion(+)
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 418a26bb8e40..433bec6e9fdd 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -212,6 +212,7 @@ class ConsoleBase(object): Nothing. '''
+ self.log.action('Sending Ctrl-C') self.run_command(chr(3), wait_for_echo=False, send_nl=False)
def drain_console(self):

On 20 January 2016 at 15:15, Stephen Warren swarren@wwwdotorg.org wrote:
From: Stephen Warren swarren@nvidia.com
Write a note to the log file when a test sends CTRL-C to U-Boot. This makes it easier to follow what's happening in the logs, especially since U-Boot doesn't echo the character back to its output, so there's no other signal of what's going on.
Signed-off-by: Stephen Warren swarren@nvidia.com
test/py/u_boot_console_base.py | 1 + 1 file changed, 1 insertion(+)
Acked-by: Simon Glass sjg@chromium.org

From: Stephen Warren swarren@nvidia.com
Sometimes it's useful to run shell commands and ignore any errors. One example might be cleanup logic; if a test-case experiences an error, the cleanup logic might experience an error too, and we don't want that error to mask the original error, so we want to ignore the subsequent error.
Signed-off-by: Stephen Warren swarren@nvidia.com --- test/py/multiplexed_log.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py index 48f2b51de15f..5059bbfb99c3 100644 --- a/test/py/multiplexed_log.py +++ b/test/py/multiplexed_log.py @@ -106,13 +106,17 @@ class RunAndLog(object): '''Clean up any resources managed by this object.''' pass
- def run(self, cmd, cwd=None): + def run(self, cmd, cwd=None, ignore_errors=False): '''Run a command as a sub-process, and log the results.
Args: cmd: The command to execute. cwd: The directory to run the command in. Can be None to use the current directory. + ignore_errors: Indicate whether to ignore errors. If True, the + function will simply return if the command cannot be executed + or exits with an error code, otherwise an exception will be + raised if such problems occur.
Returns: Nothing. @@ -148,7 +152,7 @@ class RunAndLog(object): exception = e if output and not output.endswith('\n'): output += '\n' - if exit_status and not exception: + if exit_status and not exception and not ignore_errors: exception = Exception('Exit code: ' + str(exit_status)) if exception: output += str(exception) + '\n'

On 20 January 2016 at 15:15, Stephen Warren swarren@wwwdotorg.org wrote:
From: Stephen Warren swarren@nvidia.com
Sometimes it's useful to run shell commands and ignore any errors. One example might be cleanup logic; if a test-case experiences an error, the cleanup logic might experience an error too, and we don't want that error to mask the original error, so we want to ignore the subsequent error.
Signed-off-by: Stephen Warren swarren@nvidia.com
test/py/multiplexed_log.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
Acked-by: Simon Glass sjg@chromium.org

From: Stephen Warren swarren@nvidia.com
Add various common utility functions. These will be used by a forthcoming re-written UMS test, and a brand-new DFU test.
Signed-off-by: Stephen Warren swarren@nvidia.com --- test/py/u_boot_console_base.py | 19 +++++ test/py/u_boot_utils.py | 171 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 190 insertions(+) create mode 100644 test/py/u_boot_utils.py
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 433bec6e9fdd..06f61f987180 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -215,6 +215,25 @@ class ConsoleBase(object): self.log.action('Sending Ctrl-C') self.run_command(chr(3), wait_for_echo=False, send_nl=False)
+ def wait_for(self, text): + '''Wait for a pattern to be emitted by U-Boot. + + This is useful when a long-running command such as "dfu" is executing, + and it periodically emits some text that should show up at a specific + location in the log file. + + Args: + text: The text to wait for; either a string (containing raw text, + not a regular expression) or an re object. + + Returns: + Nothing. + ''' + + if type(text) == type(''): + text = re.escape(text) + self.p.expect([text]) + def drain_console(self): '''Read from and log the U-Boot console for a short time.
diff --git a/test/py/u_boot_utils.py b/test/py/u_boot_utils.py new file mode 100644 index 000000000000..539af618dbf2 --- /dev/null +++ b/test/py/u_boot_utils.py @@ -0,0 +1,171 @@ +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +# Utility code shared across multiple tests. + +import hashlib +import os +import os.path +import sys +import time + +def md5sum_data(data): + '''Calculate the MD5 hash of some data. + + Args: + data: The data to hash. + + Returns: + The hash of the data, as a binary string. + ''' + + h = hashlib.md5() + h.update(data) + return h.digest() + +def md5sum_file(fn, max_length=None): + '''Calculate the MD5 hash of the contents of a file. + + Args: + fn: The filename of the file to hash. + max_length: The number of bytes to hash. If the file has more + bytes than this, they will be ignored. If None or omitted, the + entire file will be hashed. + + Returns: + The hash of the file content, as a binary string. + ''' + + with open(fn, 'rb') as fh: + if max_length: + params = [max_length] + else: + params = [] + data = fh.read(*params) + return md5sum_data(data) + +class PersistentRandomFile(object): + '''Generate and store information about a persistent file containing + random data.''' + + def __init__(self, u_boot_console, fn, size): + '''Create or process the persistent file. + + If the file does not exist, it is generated. + + If the file does exist, its content is hashed for later comparison. + + These files are always located in the "persistent data directory" of + the current test run. + + Args: + u_boot_console: A console connection to U-Boot. + fn: The filename (without path) to create. + size: The desired size of the file in bytes. + + Returns: + Nothing. + ''' + + self.fn = fn + + self.abs_fn = u_boot_console.config.persistent_data_dir + '/' + fn + + if os.path.exists(self.abs_fn): + u_boot_console.log.action('Persistent data file ' + self.abs_fn + + ' already exists') + self.content_hash = md5sum_file(self.abs_fn) + else: + u_boot_console.log.action('Generating ' + self.abs_fn + + ' (random, persistent, %d bytes)' % size) + data = os.urandom(size) + with open(self.abs_fn, 'wb') as fh: + fh.write(data) + self.content_hash = md5sum_data(data) + +def attempt_to_open_file(fn): + '''Attempt to open a file, without throwing exceptions. + + Any errors (exceptions) that occur during the attempt to open the file + are ignored. This is useful in order to test whether a file (in + particular, a device node) exists and can be successfully opened, in order + to poll for e.g. USB enumeration completion. + + Args: + fn: The filename to attempt to open. + + Returns: + An open file handle to the file, or None if the file could not be + opened. + ''' + + try: + return open(fn, 'rb') + except: + return None + +def wait_until_open_succeeds(fn): + '''Poll until a file can be opened, or a timeout occurs. + + Continually attempt to open a file, and return when this succeeds, or + raise an exception after a timeout. + + Args: + fn: The filename to attempt to open. + + Returns: + An open file handle to the file. + ''' + + for i in xrange(100): + fh = attempt_to_open_file(fn) + if fh: + return fh + time.sleep(0.1) + raise Exception('File could not be opened') + +def wait_until_file_open_fails(fn, ignore_errors): + '''Poll until a file cannot be opened, or a timeout occurs. + + Continually attempt to open a file, and return when this fails, or + raise an exception after a timeout. + + Args: + fn: The filename to attempt to open. + ignore_errors: Indicate whether to ignore timeout errors. If True, the + function will simply return if a timeout occurs, otherwise an + exception will be raised. + + Returns: + Nothing. + ''' + + for i in xrange(100): + fh = attempt_to_open_file(fn) + if not fh: + return + fh.close() + time.sleep(0.1) + if ignore_errors: + return + raise Exception('File can still be opened') + +def run_and_log(u_boot_console, cmd, ignore_errors=False): + '''Run a command and log its output. + + Args: + u_boot_console: A console connection to U-Boot. + cmd: The command to run, as an array of argv[]. + ignore_errors: Indicate whether to ignore errors. If True, the function + will simply return if the command cannot be executed or exits with + an error code, otherwise an exception will be raised if such + problems occur. + + Returns: + Nothing. + ''' + + runner = u_boot_console.log.get_runner(cmd[0], sys.stdout) + runner.run(cmd, ignore_errors=ignore_errors) + runner.close()

Hi Stephen,
On 20 January 2016 at 15:15, Stephen Warren swarren@wwwdotorg.org wrote:
From: Stephen Warren swarren@nvidia.com
Add various common utility functions. These will be used by a forthcoming re-written UMS test, and a brand-new DFU test.
Signed-off-by: Stephen Warren swarren@nvidia.com
test/py/u_boot_console_base.py | 19 +++++ test/py/u_boot_utils.py | 171 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 190 insertions(+) create mode 100644 test/py/u_boot_utils.py
Acked-by: Simon Glass sjg@chromium.org
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 433bec6e9fdd..06f61f987180 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -215,6 +215,25 @@ class ConsoleBase(object): self.log.action('Sending Ctrl-C') self.run_command(chr(3), wait_for_echo=False, send_nl=False)
- def wait_for(self, text):
'''Wait for a pattern to be emitted by U-Boot.
I meant to say we should use """ for function comments to keep it consistent with the rest of U-Boot. Maybe could adjust this in a follow-on patch?
This is useful when a long-running command such as "dfu" is executing,
and it periodically emits some text that should show up at a specific
location in the log file.
Args:
text: The text to wait for; either a string (containing raw text,
not a regular expression) or an re object.
Returns:
Nothing.
'''
if type(text) == type(''):
text = re.escape(text)
self.p.expect([text])
Does this potentially wait forever?
- def drain_console(self): '''Read from and log the U-Boot console for a short time.
diff --git a/test/py/u_boot_utils.py b/test/py/u_boot_utils.py new file mode 100644 index 000000000000..539af618dbf2 --- /dev/null +++ b/test/py/u_boot_utils.py @@ -0,0 +1,171 @@ +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0
+# Utility code shared across multiple tests.
+import hashlib +import os +import os.path +import sys +import time
+def md5sum_data(data):
- '''Calculate the MD5 hash of some data.
- Args:
data: The data to hash.
- Returns:
The hash of the data, as a binary string.
- '''
- h = hashlib.md5()
- h.update(data)
- return h.digest()
Or just:
return hashlib.md5().update(data).digest()
Regards, Simon

On 01/21/2016 08:36 PM, Simon Glass wrote:
Hi Stephen,
On 20 January 2016 at 15:15, Stephen Warren swarren@wwwdotorg.org wrote:
From: Stephen Warren swarren@nvidia.com
Add various common utility functions. These will be used by a forthcoming re-written UMS test, and a brand-new DFU test.
Signed-off-by: Stephen Warren swarren@nvidia.com
test/py/u_boot_console_base.py | 19 +++++ test/py/u_boot_utils.py | 171 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 190 insertions(+) create mode 100644 test/py/u_boot_utils.py
Acked-by: Simon Glass sjg@chromium.org
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 433bec6e9fdd..06f61f987180 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -215,6 +215,25 @@ class ConsoleBase(object): self.log.action('Sending Ctrl-C') self.run_command(chr(3), wait_for_echo=False, send_nl=False)
- def wait_for(self, text):
'''Wait for a pattern to be emitted by U-Boot.
I meant to say we should use """ for function comments to keep it consistent with the rest of U-Boot. Maybe could adjust this in a follow-on patch?
That feels inconsistent with using ' for strings everywhere else. I don't see a good reason why the docstrings should use a different quote character. Should the existing Python code be made consistent instead?
This is useful when a long-running command such as "dfu" is executing,
and it periodically emits some text that should show up at a specific
location in the log file.
Args:
text: The text to wait for; either a string (containing raw text,
not a regular expression) or an re object.
Returns:
Nothing.
'''
if type(text) == type(''):
text = re.escape(text)
self.p.expect([text])
Does this potentially wait forever?
The expect() function throws a Timeout exception if none of the strings/regexs passed to it match within the defined timeout (which is stored in self.p.timeout).

On 01/21/2016 08:36 PM, Simon Glass wrote:
Hi Stephen,
On 20 January 2016 at 15:15, Stephen Warren swarren@wwwdotorg.org wrote:
From: Stephen Warren swarren@nvidia.com
Add various common utility functions. These will be used by a forthcoming re-written UMS test, and a brand-new DFU test.
diff --git a/test/py/u_boot_utils.py b/test/py/u_boot_utils.py
+def md5sum_data(data):
- '''Calculate the MD5 hash of some data.
- Args:
data: The data to hash.
- Returns:
The hash of the data, as a binary string.
- '''
- h = hashlib.md5()
- h.update(data)
- return h.digest()
Or just:
return hashlib.md5().update(data).digest()
I was going to do that, but it doesn't look like update() returns the hash object, so I can't chain at least the .digest() call together, so I may as well leave it as-is.

From: Stephen Warren swarren@nvidia.com
Enhance the UMS test to optionally mount a partition and read/write a file to it, validating that the content written and read back are identical.
This enhancement is backwards-compatible; old boardenv contents that don't define the new configuration data will cause the test code to perform as before.
test/ums/ is deleted since the Python test now performs the same testing that it did.
The code is also re-written to make use of the recently added utility module, and split it up into nested functions so the overall logic of the test process can be followed more easily without the details cluttering the code.
Cc: Lukasz Majewski l.majewski@samsung.com Signed-off-by: Stephen Warren swarren@nvidia.com --- test/py/tests/test_ums.py | 212 +++++++++++++++++++++++++++++++++++--------- test/ums/README | 30 ------- test/ums/ums_gadget_test.sh | 183 -------------------------------------- 3 files changed, 169 insertions(+), 256 deletions(-) delete mode 100644 test/ums/README delete mode 100755 test/ums/ums_gadget_test.sh
diff --git a/test/py/tests/test_ums.py b/test/py/tests/test_ums.py index a137221c7a5b..cb6e5ef8c20b 100644 --- a/test/py/tests/test_ums.py +++ b/test/py/tests/test_ums.py @@ -7,8 +7,11 @@ # should be enhanced to validate disk IO.
import os +import os.path import pytest +import re import time +import u_boot_utils
''' Note: This test relies on: @@ -17,13 +20,36 @@ a) boardenv_* to contain configuration values to define which USB ports are available for testing. Without this, this test will be automatically skipped. For example:
+# Leave this list empty if you have no block_devs below with writable +# partitions defined. +env__mount_points = ( + "/mnt/ubtest-mnt-p2371-2180-na", +) + env__usb_dev_ports = ( - {'tgt_usb_ctlr': '0', 'host_ums_dev_node': '/dev/disk/by-path/pci-0000:00:14.0-usb-0:13:1.0-scsi-0:0:0:0'}, + { + "tgt_usb_ctlr": "0", + "host_ums_dev_node": "/dev/disk/by-path/pci-0000:00:14.0-usb-0:13:1.0-scsi-0:0:0:0", + }, )
env__block_devs = ( - {'type': 'mmc', 'id': '0'}, # eMMC; always present - {'type': 'mmc', 'id': '1'}, # SD card; present since I plugged one in + # eMMC; always present + { + "type": "mmc", + "id": "0", + # The following two properties are optional. + # If present, the partition will be mounted and a file written-to and + # read-from it. If missing, only a simple block read test will be + # performed. + "writable_fs_partition": 1, + "writable_fs_subdir": "tmp/", + }, + # SD card; present since I plugged one in + { + "type": "mmc", + "id": "1" + }, )
b) udev rules to set permissions on devices nodes, so that sudo is not @@ -34,41 +60,15 @@ ACTION=="add", SUBSYSTEM=="block", SUBSYSTEMS=="usb", KERNELS=="3-13", MODE:="66 (You may wish to change the group ID instead of setting the permissions wide open. All that matters is that the user ID running the test can access the device.) -'''
-def open_ums_device(host_ums_dev_node): - '''Attempt to open a device node, returning either the opened file handle, - or None on any error.''' +c) /etc/fstab entries to allow the block device to be mounted without requiring +root permissions. For example:
- try: - return open(host_ums_dev_node, 'rb') - except: - return None - -def wait_for_ums_device(host_ums_dev_node): - '''Continually attempt to open the device node exported by the "ums" - command, and either return the opened file handle, or raise an exception - after a timeout.''' - - for i in xrange(100): - fh = open_ums_device(host_ums_dev_node) - if fh: - return fh - time.sleep(0.1) - raise Exception('UMS device did not appear') - -def wait_for_ums_device_gone(host_ums_dev_node): - '''Continually attempt to open the device node exported by the "ums" - command, and either return once the device has disappeared, or raise an - exception if it does not before a timeout occurs.''' - - for i in xrange(100): - fh = open_ums_device(host_ums_dev_node) - if not fh: - return - fh.close() - time.sleep(0.1) - raise Exception('UMS device did not disappear') +/dev/disk/by-path/pci-0000:00:14.0-usb-0:13:1.0-scsi-0:0:0:0-part1 /mnt/ubtest-mnt-p2371-2180-na ext4 noauto,user,nosuid,nodev + +This entry is only needed if any block_devs above contain a +writable_fs_partition value. +'''
@pytest.mark.buildconfigspec('cmd_usb_mass_storage') def test_ums(u_boot_console, env__usb_dev_port, env__block_devs): @@ -76,6 +76,14 @@ def test_ums(u_boot_console, env__usb_dev_port, env__block_devs): device when "ums" is running, and this device must disappear when "ums" is aborted.'''
+ have_writable_fs_partition = 'writable_fs_partition' in env__block_devs[0] + if not have_writable_fs_partition: + # If 'writable_fs_subdir' is missing, we'll skip all parts of the + # testing which mount filesystems. + u_boot_console.log.warning( + 'boardenv missing "writable_fs_partition"; ' + + 'UMS testing will be limited.') + tgt_usb_ctlr = env__usb_dev_port['tgt_usb_ctlr'] host_ums_dev_node = env__usb_dev_port['host_ums_dev_node']
@@ -84,11 +92,129 @@ def test_ums(u_boot_console, env__usb_dev_port, env__block_devs): # device list here. We'll test each block device somewhere else. tgt_dev_type = env__block_devs[0]['type'] tgt_dev_id = env__block_devs[0]['id'] + if have_writable_fs_partition: + mount_point = u_boot_console.config.env['env__mount_points'][0] + mount_subdir = env__block_devs[0]['writable_fs_subdir'] + part_num = env__block_devs[0]['writable_fs_partition'] + host_ums_part_node = '%s-part%d' % (host_ums_dev_node, part_num) + else: + host_ums_part_node = host_ums_dev_node + + test_f = u_boot_utils.PersistentRandomFile(u_boot_console, 'ums.bin', + 1024 * 1024); + if have_writable_fs_partition: + mounted_test_fn = mount_point + '/' + mount_subdir + test_f.fn + + def start_ums(): + '''Start U-Boot's ums shell command. + + This also waits for the host-side USB enumeration process to complete. + + Args: + None. + + Returns: + Nothing. + ''' + + u_boot_console.log.action( + 'Starting long-running U-Boot ums shell command') + cmd = 'ums %s %s %s' % (tgt_usb_ctlr, tgt_dev_type, tgt_dev_id) + u_boot_console.run_command(cmd, wait_for_prompt=False) + u_boot_console.wait_for(re.compile('UMS: LUN.*[\r\n]')) + fh = u_boot_utils.wait_until_open_succeeds(host_ums_part_node) + u_boot_console.log.action('Reading raw data from UMS device') + fh.read(4096) + fh.close() + + def mount(): + '''Mount the block device that U-Boot exports. + + Args: + None. + + Returns: + Nothing. + '''
- cmd = 'ums %s %s %s' % (tgt_usb_ctlr, tgt_dev_type, tgt_dev_id) - u_boot_console.run_command('ums 0 mmc 0', wait_for_prompt=False) - fh = wait_for_ums_device(host_ums_dev_node) - fh.read(4096) - fh.close() - u_boot_console.ctrlc() - wait_for_ums_device_gone(host_ums_dev_node) + u_boot_console.log.action('Mounting exported UMS device') + cmd = ('/bin/mount', host_ums_part_node) + u_boot_utils.run_and_log(u_boot_console, cmd) + + def umount(ignore_errors): + '''Unmount the block device that U-Boot exports. + + Args: + ignore_errors: Ignore any errors. This is useful if an error has + already been detected, and the code is performing best-effort + cleanup. In this case, we do not want to mask the original + error by "honoring" any new errors. + + Returns: + Nothing. + ''' + + u_boot_console.log.action('Unmounting UMS device') + cmd = ('/bin/umount', host_ums_part_node) + u_boot_utils.run_and_log(u_boot_console, cmd, ignore_errors) + + def stop_ums(ignore_errors): + '''Stop U-Boot's ums shell command from executing. + + This also waits for the host-side USB de-enumeration process to + complete. + + Args: + ignore_errors: Ignore any errors. This is useful if an error has + already been detected, and the code is performing best-effort + cleanup. In this case, we do not want to mask the original + error by "honoring" any new errors. + + Returns: + Nothing. + ''' + + u_boot_console.log.action( + 'Stopping long-running U-Boot ums shell command') + u_boot_console.ctrlc() + u_boot_utils.wait_until_file_open_fails(host_ums_part_node, + ignore_errors) + + ignore_cleanup_errors = True + try: + start_ums() + if not have_writable_fs_partition: + # Skip filesystem-based testing if not configured + return + try: + mount() + u_boot_console.log.action('Writing test file via UMS') + cmd = ('rm', '-f', mounted_test_fn) + u_boot_utils.run_and_log(u_boot_console, cmd) + if os.path.exists(mounted_test_fn): + raise Exception('Could not rm target UMS test file') + cmd = ('cp', test_f.abs_fn, mounted_test_fn) + u_boot_utils.run_and_log(u_boot_console, cmd) + ignore_cleanup_errors = False + finally: + umount(ignore_errors=ignore_cleanup_errors) + finally: + stop_ums(ignore_errors=ignore_cleanup_errors) + + ignore_cleanup_errors = True + try: + start_ums() + try: + mount() + u_boot_console.log.action('Reading test file back via UMS') + read_back_hash = u_boot_utils.md5sum_file(mounted_test_fn) + cmd = ('rm', '-f', mounted_test_fn) + u_boot_utils.run_and_log(u_boot_console, cmd) + ignore_cleanup_errors = False + finally: + umount(ignore_errors=ignore_cleanup_errors) + finally: + stop_ums(ignore_errors=ignore_cleanup_errors) + + written_hash = test_f.content_hash + assert(written_hash == read_back_hash) diff --git a/test/ums/README b/test/ums/README deleted file mode 100644 index c80fbfefbf52..000000000000 --- a/test/ums/README +++ /dev/null @@ -1,30 +0,0 @@ -UMS test script. - -ums_gadget_test.sh -================== - -Example usage: -1. On the target: - create UMS exportable partitions (with e.g. gpt write), or specify a - partition number (PART_NUM) as "-" to use the entire device - ums 0 mmc 0 -2. On the host: - sudo test/ums/ums_gadget_test.sh VID PID PART_NUM [-f FILE_SYSTEM] [test_file] - e.g. sudo test/ums/ums_gadget_test.sh 0525 a4a5 6 -f vfat ./dat_14M.img - -... where: - VID - UMS device USB Vendor ID - PID - UMS device USB Product ID - PART_NUM - is the partition number on which UMS operates or "-" to use the - whole device - -Information about available partitions on the target one can read with using -the 'mmc part' or 'part list' commands. - -The partition num (PART_NUM) can be specified as '-' for using the whole device. - -The [-f FILE_SYSTEM] optional switch allows for formatting target partition to -FILE_SYSTEM. - -The last, optional [test_file] parameter is for specifying the exact test file -to use. diff --git a/test/ums/ums_gadget_test.sh b/test/ums/ums_gadget_test.sh deleted file mode 100755 index 9da486b266ce..000000000000 --- a/test/ums/ums_gadget_test.sh +++ /dev/null @@ -1,183 +0,0 @@ -#! /bin/bash - -# Copyright (C) 2014 Samsung Electronics -# Lukasz Majewski l.majewski@samsung.com -# -# UMS operation test script -# -# SPDX-License-Identifier: GPL-2.0+ - -clear - -COLOUR_RED="\33[31m" -COLOUR_GREEN="\33[32m" -COLOUR_ORANGE="\33[33m" -COLOUR_DEFAULT="\33[0m" - -DIR=./ -SUFFIX=img -RCV_DIR=rcv/ -LOG_FILE=./log/log-`date +%d-%m-%Y_%H-%M-%S` - -cd `dirname $0` -../dfu/dfu_gadget_test_init.sh 33M 97M - -cleanup () { - rm -rf $RCV_DIR $MNT_DIR -} - -control_c() -# run if user hits control-c -{ - echo -en "\n*** CTRL+C ***\n" - umount $MNT_DIR - cleanup - exit 0 -} - -# trap keyboard interrupt (control-c) -trap control_c SIGINT - -die () { - printf " $COLOUR_RED FAILED $COLOUR_DEFAULT \n" - cleanup - exit 1 -} - -calculate_md5sum () { - MD5SUM=`md5sum $1` - MD5SUM=`echo $MD5SUM | cut -d ' ' -f1` - echo "md5sum:"$MD5SUM -} - -ums_test_file () { - printf "$COLOUR_GREEN========================================================================================= $COLOUR_DEFAULT\n" - printf "File:$COLOUR_GREEN %s $COLOUR_DEFAULT\n" $1 - - mount /dev/$MEM_DEV $MNT_DIR - if [ -f $MNT_DIR/dat_* ]; then - rm $MNT_DIR/dat_* - fi - - cp ./$1 $MNT_DIR - - while true; do - umount $MNT_DIR > /dev/null 2>&1 - if [ $? -eq 0 ]; then - break - fi - printf "$COLOUR_ORANGE\tSleeping to wait for umount...$COLOUR_DEFAULT\n" - sleep 1 - done - - echo -n "TX: " - calculate_md5sum $1 - - MD5_TX=$MD5SUM - sleep 1 - N_FILE=$DIR$RCV_DIR${1:2}"_rcv" - - mount /dev/$MEM_DEV $MNT_DIR - cp $MNT_DIR/$1 $N_FILE || die $? - rm $MNT_DIR/$1 - umount $MNT_DIR - - echo -n "RX: " - calculate_md5sum $N_FILE - MD5_RX=$MD5SUM - - if [ "$MD5_TX" == "$MD5_RX" ]; then - printf " $COLOUR_GREEN -------> OK $COLOUR_DEFAULT \n" - else - printf " $COLOUR_RED -------> FAILED $COLOUR_DEFAULT \n" - cleanup - exit 1 - fi -} - -printf "$COLOUR_GREEN========================================================================================= $COLOUR_DEFAULT\n" -echo "U-boot UMS test program" - -if [ $EUID -ne 0 ]; then - echo "You must be root to do this." 1>&2 - exit 100 -fi - -if [ $# -lt 3 ]; then - echo "Wrong number of arguments" - echo "Example:" - echo "sudo ./ums_gadget_test.sh VID PID PART_NUM [-f ext4] [test_file]" - die -fi - -MNT_DIR="/mnt/tmp-ums-test" - -VID=$1; shift -PID=$1; shift -PART_NUM=$1; shift - -if [ "$1" == "-f" ]; then - shift - FS_TO_FORMAT=$1; shift -fi - -TEST_FILE=$1 - -for f in `find /sys -type f -name idProduct`; do - d=`dirname ${f}` - if [ `cat ${d}/idVendor` != "${VID}" ]; then - continue - fi - if [ `cat ${d}/idProduct` != "${PID}" ]; then - continue - fi - USB_DEV=${d} - break -done - -if [ -z "${USB_DEV}" ]; then - echo "Connect target" - echo "e.g. ums 0 mmc 0" - exit 1 -fi - -MEM_DEV=`find $USB_DEV -type d -name "sd[a-z]" | awk -F/ '{print $(NF)}' -` - -mkdir -p $RCV_DIR -if [ ! -d $MNT_DIR ]; then - mkdir -p $MNT_DIR -fi - -if [ "$PART_NUM" == "-" ]; then - PART_NUM="" -fi -MEM_DEV=$MEM_DEV$PART_NUM - -if [ -n "$FS_TO_FORMAT" ]; then - echo -n "Formatting partition /dev/$MEM_DEV to $FS_TO_FORMAT" - mkfs -t $FS_TO_FORMAT /dev/$MEM_DEV > /dev/null 2>&1 - if [ $? -eq 0 ]; then - printf " $COLOUR_GREEN DONE $COLOUR_DEFAULT \n" - else - die - fi -fi - -printf "Mount: /dev/$MEM_DEV \n" - -if [ -n "$TEST_FILE" ]; then - if [ ! -e $TEST_FILE ]; then - echo "No file: $TEST_FILE" - die - fi - ums_test_file $TEST_FILE -else - for file in $DIR*.$SUFFIX - do - ums_test_file $file - done -fi - -cleanup - -exit 0

Hi Stephen,
From: Stephen Warren swarren@nvidia.com
Enhance the UMS test to optionally mount a partition and read/write a file to it, validating that the content written and read back are identical.
This enhancement is backwards-compatible; old boardenv contents that don't define the new configuration data will cause the test code to perform as before.
test/ums/ is deleted since the Python test now performs the same testing that it did.
The code is also re-written to make use of the recently added utility module, and split it up into nested functions so the overall logic of the test process can be followed more easily without the details cluttering the code.
Cc: Lukasz Majewski l.majewski@samsung.com Signed-off-by: Stephen Warren swarren@nvidia.com
test/py/tests/test_ums.py | 212 +++++++++++++++++++++++++++++++++++--------- test/ums/README | 30 ------- test/ums/ums_gadget_test.sh | 183 -------------------------------------- 3 files changed, 169 insertions(+), 256 deletions(-) delete mode 100644 test/ums/README delete mode 100755 test/ums/ums_gadget_test.sh
diff --git a/test/py/tests/test_ums.py b/test/py/tests/test_ums.py index a137221c7a5b..cb6e5ef8c20b 100644 --- a/test/py/tests/test_ums.py +++ b/test/py/tests/test_ums.py @@ -7,8 +7,11 @@ # should be enhanced to validate disk IO.
import os +import os.path import pytest +import re import time +import u_boot_utils
''' Note: This test relies on: @@ -17,13 +20,36 @@ a) boardenv_* to contain configuration values to define which USB ports are available for testing. Without this, this test will be automatically skipped. For example:
+# Leave this list empty if you have no block_devs below with writable +# partitions defined. +env__mount_points = (
- "/mnt/ubtest-mnt-p2371-2180-na",
+)
env__usb_dev_ports = (
- {'tgt_usb_ctlr': '0', 'host_ums_dev_node':
'/dev/disk/by-path/pci-0000:00:14.0-usb-0:13:1.0-scsi-0:0:0:0'},
- {
"tgt_usb_ctlr": "0",
"host_ums_dev_node":
"/dev/disk/by-path/pci-0000:00:14.0-usb-0:13:1.0-scsi-0:0:0:0",
- },
)
env__block_devs = (
- {'type': 'mmc', 'id': '0'}, # eMMC; always present
- {'type': 'mmc', 'id': '1'}, # SD card; present since I plugged
one in
- # eMMC; always present
- {
"type": "mmc",
"id": "0",
# The following two properties are optional.
# If present, the partition will be mounted and a file
written-to and
# read-from it. If missing, only a simple block read test
will be
# performed.
"writable_fs_partition": 1,
"writable_fs_subdir": "tmp/",
- },
- # SD card; present since I plugged one in
- {
"type": "mmc",
"id": "1"
- },
)
b) udev rules to set permissions on devices nodes, so that sudo is not @@ -34,41 +60,15 @@ ACTION=="add", SUBSYSTEM=="block", SUBSYSTEMS=="usb", KERNELS=="3-13", MODE:="66 (You may wish to change the group ID instead of setting the permissions wide open. All that matters is that the user ID running the test can access the device.) -'''
-def open_ums_device(host_ums_dev_node):
- '''Attempt to open a device node, returning either the opened
file handle,
- or None on any error.'''
+c) /etc/fstab entries to allow the block device to be mounted without requiring +root permissions. For example:
- try:
return open(host_ums_dev_node, 'rb')
- except:
return None
-def wait_for_ums_device(host_ums_dev_node):
- '''Continually attempt to open the device node exported by the
"ums"
- command, and either return the opened file handle, or raise an
exception
- after a timeout.'''
- for i in xrange(100):
fh = open_ums_device(host_ums_dev_node)
if fh:
return fh
time.sleep(0.1)
- raise Exception('UMS device did not appear')
-def wait_for_ums_device_gone(host_ums_dev_node):
- '''Continually attempt to open the device node exported by the
"ums"
- command, and either return once the device has disappeared, or
raise an
- exception if it does not before a timeout occurs.'''
- for i in xrange(100):
fh = open_ums_device(host_ums_dev_node)
if not fh:
return
fh.close()
time.sleep(0.1)
- raise Exception('UMS device did not disappear')
+/dev/disk/by-path/pci-0000:00:14.0-usb-0:13:1.0-scsi-0:0:0:0-part1 /mnt/ubtest-mnt-p2371-2180-na ext4 noauto,user,nosuid,nodev + +This entry is only needed if any block_devs above contain a +writable_fs_partition value. +'''
@pytest.mark.buildconfigspec('cmd_usb_mass_storage') def test_ums(u_boot_console, env__usb_dev_port, env__block_devs): @@ -76,6 +76,14 @@ def test_ums(u_boot_console, env__usb_dev_port, env__block_devs): device when "ums" is running, and this device must disappear when "ums" is aborted.'''
- have_writable_fs_partition = 'writable_fs_partition' in
env__block_devs[0]
- if not have_writable_fs_partition:
# If 'writable_fs_subdir' is missing, we'll skip all parts
of the
# testing which mount filesystems.
u_boot_console.log.warning(
'boardenv missing "writable_fs_partition"; ' +
'UMS testing will be limited.')
- tgt_usb_ctlr = env__usb_dev_port['tgt_usb_ctlr'] host_ums_dev_node = env__usb_dev_port['host_ums_dev_node']
@@ -84,11 +92,129 @@ def test_ums(u_boot_console, env__usb_dev_port, env__block_devs): # device list here. We'll test each block device somewhere else. tgt_dev_type = env__block_devs[0]['type'] tgt_dev_id = env__block_devs[0]['id']
- if have_writable_fs_partition:
mount_point =
u_boot_console.config.env['env__mount_points'][0]
mount_subdir = env__block_devs[0]['writable_fs_subdir']
part_num = env__block_devs[0]['writable_fs_partition']
host_ums_part_node = '%s-part%d' % (host_ums_dev_node,
part_num)
- else:
host_ums_part_node = host_ums_dev_node
- test_f = u_boot_utils.PersistentRandomFile(u_boot_console,
'ums.bin',
1024 * 1024);
- if have_writable_fs_partition:
mounted_test_fn = mount_point + '/' + mount_subdir +
test_f.fn +
- def start_ums():
'''Start U-Boot's ums shell command.
This also waits for the host-side USB enumeration process to
complete. +
Args:
None.
Returns:
Nothing.
'''
u_boot_console.log.action(
'Starting long-running U-Boot ums shell command')
cmd = 'ums %s %s %s' % (tgt_usb_ctlr, tgt_dev_type,
tgt_dev_id)
u_boot_console.run_command(cmd, wait_for_prompt=False)
u_boot_console.wait_for(re.compile('UMS: LUN.*[\r\n]'))
fh =
u_boot_utils.wait_until_open_succeeds(host_ums_part_node)
u_boot_console.log.action('Reading raw data from UMS device')
fh.read(4096)
fh.close()
- def mount():
'''Mount the block device that U-Boot exports.
Args:
None.
Returns:
Nothing.
'''
- cmd = 'ums %s %s %s' % (tgt_usb_ctlr, tgt_dev_type, tgt_dev_id)
- u_boot_console.run_command('ums 0 mmc 0', wait_for_prompt=False)
- fh = wait_for_ums_device(host_ums_dev_node)
- fh.read(4096)
- fh.close()
- u_boot_console.ctrlc()
- wait_for_ums_device_gone(host_ums_dev_node)
u_boot_console.log.action('Mounting exported UMS device')
cmd = ('/bin/mount', host_ums_part_node)
u_boot_utils.run_and_log(u_boot_console, cmd)
- def umount(ignore_errors):
'''Unmount the block device that U-Boot exports.
Args:
ignore_errors: Ignore any errors. This is useful if an
error has
already been detected, and the code is performing
best-effort
cleanup. In this case, we do not want to mask the
original
error by "honoring" any new errors.
Returns:
Nothing.
'''
u_boot_console.log.action('Unmounting UMS device')
cmd = ('/bin/umount', host_ums_part_node)
u_boot_utils.run_and_log(u_boot_console, cmd, ignore_errors)
- def stop_ums(ignore_errors):
'''Stop U-Boot's ums shell command from executing.
This also waits for the host-side USB de-enumeration process
to
complete.
Args:
ignore_errors: Ignore any errors. This is useful if an
error has
already been detected, and the code is performing
best-effort
cleanup. In this case, we do not want to mask the
original
error by "honoring" any new errors.
Returns:
Nothing.
'''
u_boot_console.log.action(
'Stopping long-running U-Boot ums shell command')
u_boot_console.ctrlc()
u_boot_utils.wait_until_file_open_fails(host_ums_part_node,
ignore_errors)
- ignore_cleanup_errors = True
- try:
start_ums()
if not have_writable_fs_partition:
# Skip filesystem-based testing if not configured
return
try:
mount()
u_boot_console.log.action('Writing test file via UMS')
cmd = ('rm', '-f', mounted_test_fn)
u_boot_utils.run_and_log(u_boot_console, cmd)
if os.path.exists(mounted_test_fn):
raise Exception('Could not rm target UMS test file')
cmd = ('cp', test_f.abs_fn, mounted_test_fn)
u_boot_utils.run_and_log(u_boot_console, cmd)
ignore_cleanup_errors = False
finally:
umount(ignore_errors=ignore_cleanup_errors)
- finally:
stop_ums(ignore_errors=ignore_cleanup_errors)
- ignore_cleanup_errors = True
- try:
start_ums()
try:
mount()
u_boot_console.log.action('Reading test file back via
UMS')
read_back_hash =
u_boot_utils.md5sum_file(mounted_test_fn)
cmd = ('rm', '-f', mounted_test_fn)
u_boot_utils.run_and_log(u_boot_console, cmd)
ignore_cleanup_errors = False
finally:
umount(ignore_errors=ignore_cleanup_errors)
- finally:
stop_ums(ignore_errors=ignore_cleanup_errors)
- written_hash = test_f.content_hash
- assert(written_hash == read_back_hash)
diff --git a/test/ums/README b/test/ums/README deleted file mode 100644 index c80fbfefbf52..000000000000 --- a/test/ums/README +++ /dev/null @@ -1,30 +0,0 @@ -UMS test script.
-ums_gadget_test.sh
-Example usage: -1. On the target:
- create UMS exportable partitions (with e.g. gpt write), or
specify a
- partition number (PART_NUM) as "-" to use the entire device
- ums 0 mmc 0
-2. On the host:
- sudo test/ums/ums_gadget_test.sh VID PID PART_NUM [-f
FILE_SYSTEM] [test_file]
- e.g. sudo test/ums/ums_gadget_test.sh 0525 a4a5 6 -f
vfat ./dat_14M.img - -... where:
- VID - UMS device USB Vendor ID
- PID - UMS device USB Product ID
- PART_NUM - is the partition number on which UMS operates or "-"
to use the
whole device
-Information about available partitions on the target one can read with using -the 'mmc part' or 'part list' commands.
-The partition num (PART_NUM) can be specified as '-' for using the whole device. - -The [-f FILE_SYSTEM] optional switch allows for formatting target partition to -FILE_SYSTEM.
-The last, optional [test_file] parameter is for specifying the exact test file -to use. diff --git a/test/ums/ums_gadget_test.sh b/test/ums/ums_gadget_test.sh deleted file mode 100755 index 9da486b266ce..000000000000 --- a/test/ums/ums_gadget_test.sh +++ /dev/null @@ -1,183 +0,0 @@ -#! /bin/bash
-# Copyright (C) 2014 Samsung Electronics -# Lukasz Majewski l.majewski@samsung.com -# -# UMS operation test script -# -# SPDX-License-Identifier: GPL-2.0+
-clear
-COLOUR_RED="\33[31m" -COLOUR_GREEN="\33[32m" -COLOUR_ORANGE="\33[33m" -COLOUR_DEFAULT="\33[0m"
-DIR=./ -SUFFIX=img -RCV_DIR=rcv/ -LOG_FILE=./log/log-`date +%d-%m-%Y_%H-%M-%S`
-cd `dirname $0` -../dfu/dfu_gadget_test_init.sh 33M 97M
-cleanup () {
- rm -rf $RCV_DIR $MNT_DIR
-}
-control_c() -# run if user hits control-c -{
- echo -en "\n*** CTRL+C ***\n"
- umount $MNT_DIR
- cleanup
- exit 0
-}
-# trap keyboard interrupt (control-c) -trap control_c SIGINT
-die () {
- printf " $COLOUR_RED FAILED $COLOUR_DEFAULT \n"
- cleanup
- exit 1
-}
-calculate_md5sum () {
- MD5SUM=`md5sum $1`
- MD5SUM=`echo $MD5SUM | cut -d ' ' -f1`
- echo "md5sum:"$MD5SUM
-}
-ums_test_file () {
- printf
"$COLOUR_GREEN========================================================================================= $COLOUR_DEFAULT\n"
- printf "File:$COLOUR_GREEN %s $COLOUR_DEFAULT\n" $1
- mount /dev/$MEM_DEV $MNT_DIR
- if [ -f $MNT_DIR/dat_* ]; then
- rm $MNT_DIR/dat_*
- fi
- cp ./$1 $MNT_DIR
- while true; do
- umount $MNT_DIR > /dev/null 2>&1
- if [ $? -eq 0 ]; then
break
- fi
- printf "$COLOUR_ORANGE\tSleeping to wait for
umount...$COLOUR_DEFAULT\n"
- sleep 1
- done
- echo -n "TX: "
- calculate_md5sum $1
- MD5_TX=$MD5SUM
- sleep 1
- N_FILE=$DIR$RCV_DIR${1:2}"_rcv"
- mount /dev/$MEM_DEV $MNT_DIR
- cp $MNT_DIR/$1 $N_FILE || die $?
- rm $MNT_DIR/$1
- umount $MNT_DIR
- echo -n "RX: "
- calculate_md5sum $N_FILE
- MD5_RX=$MD5SUM
- if [ "$MD5_TX" == "$MD5_RX" ]; then
- printf " $COLOUR_GREEN -------> OK $COLOUR_DEFAULT \n"
- else
- printf " $COLOUR_RED -------> FAILED $COLOUR_DEFAULT \n"
- cleanup
- exit 1
- fi
-}
-printf "$COLOUR_GREEN========================================================================================= $COLOUR_DEFAULT\n" -echo "U-boot UMS test program" - -if [ $EUID -ne 0 ]; then
- echo "You must be root to do this." 1>&2
- exit 100
-fi
-if [ $# -lt 3 ]; then
- echo "Wrong number of arguments"
- echo "Example:"
- echo "sudo ./ums_gadget_test.sh VID PID PART_NUM [-f ext4]
[test_file]"
- die
-fi
-MNT_DIR="/mnt/tmp-ums-test"
-VID=$1; shift -PID=$1; shift -PART_NUM=$1; shift
-if [ "$1" == "-f" ]; then
- shift
- FS_TO_FORMAT=$1; shift
-fi
-TEST_FILE=$1
-for f in `find /sys -type f -name idProduct`; do
d=`dirname ${f}`
if [ `cat ${d}/idVendor` != "${VID}" ]; then
continue
fi
if [ `cat ${d}/idProduct` != "${PID}" ]; then
continue
fi
USB_DEV=${d}
break
-done
-if [ -z "${USB_DEV}" ]; then
echo "Connect target"
echo "e.g. ums 0 mmc 0"
exit 1
-fi
-MEM_DEV=`find $USB_DEV -type d -name "sd[a-z]" | awk -F/ '{print $(NF)}' -` - -mkdir -p $RCV_DIR -if [ ! -d $MNT_DIR ]; then
- mkdir -p $MNT_DIR
-fi
-if [ "$PART_NUM" == "-" ]; then
- PART_NUM=""
-fi -MEM_DEV=$MEM_DEV$PART_NUM
-if [ -n "$FS_TO_FORMAT" ]; then
- echo -n "Formatting partition /dev/$MEM_DEV to $FS_TO_FORMAT"
- mkfs -t $FS_TO_FORMAT /dev/$MEM_DEV > /dev/null 2>&1
- if [ $? -eq 0 ]; then
- printf " $COLOUR_GREEN DONE $COLOUR_DEFAULT \n"
- else
- die
- fi
-fi
-printf "Mount: /dev/$MEM_DEV \n"
-if [ -n "$TEST_FILE" ]; then
- if [ ! -e $TEST_FILE ]; then
- echo "No file: $TEST_FILE"
- die
- fi
- ums_test_file $TEST_FILE
-else
- for file in $DIR*.$SUFFIX
- do
- ums_test_file $file
- done
-fi
-cleanup
-exit 0
Acked-by: Lukasz Majewski l.majewski@samsung.com
Stephen, thanks for converting DFU and UMS to pytest code.

On 21 January 2016 at 04:26, Lukasz Majewski l.majewski@samsung.com wrote:
Hi Stephen,
From: Stephen Warren swarren@nvidia.com
Enhance the UMS test to optionally mount a partition and read/write a file to it, validating that the content written and read back are identical.
This enhancement is backwards-compatible; old boardenv contents that don't define the new configuration data will cause the test code to perform as before.
test/ums/ is deleted since the Python test now performs the same testing that it did.
The code is also re-written to make use of the recently added utility module, and split it up into nested functions so the overall logic of the test process can be followed more easily without the details cluttering the code.
Cc: Lukasz Majewski l.majewski@samsung.com Signed-off-by: Stephen Warren swarren@nvidia.com
test/py/tests/test_ums.py | 212 +++++++++++++++++++++++++++++++++++--------- test/ums/README | 30 ------- test/ums/ums_gadget_test.sh | 183 -------------------------------------- 3 files changed, 169 insertions(+), 256 deletions(-) delete mode 100644 test/ums/README delete mode 100755 test/ums/ums_gadget_test.sh
Acked-by: Simon Glass sjg@chromium.org

From: Stephen Warren swarren@nvidia.com
Add a test of DFU functionality to the Python test suite. The test starts DFU in U-Boot, waits for USB device enumeration on the host, executes dfu-util multiple times to test various transfer sizes, many of which trigger USB driver edge cases, and finally aborts the DFU command in U-Boot.
This test mirrors the functionality previously available via the shell scripts in test/dfu, and hence those are removed too.
Cc: Lukasz Majewski l.majewski@majess.pl Signed-off-by: Stephen Warren swarren@nvidia.com --- test/dfu/README | 44 ------- test/dfu/dfu_gadget_test.sh | 108 ---------------- test/dfu/dfu_gadget_test_init.sh | 45 ------- test/py/tests/test_dfu.py | 257 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 257 insertions(+), 197 deletions(-) delete mode 100644 test/dfu/README delete mode 100755 test/dfu/dfu_gadget_test.sh delete mode 100755 test/dfu/dfu_gadget_test_init.sh create mode 100644 test/py/tests/test_dfu.py
diff --git a/test/dfu/README b/test/dfu/README deleted file mode 100644 index 408d5594219a..000000000000 --- a/test/dfu/README +++ /dev/null @@ -1,44 +0,0 @@ -DFU TEST CASE DESCRIPTION: - -The prerequisites for running this script are assured by -dfu_gadget_test_init.sh, which is automatically invoked by dfu_gadget_test.sh. -In this file user is able to generate their own set of test files by altering -the default set of TEST_FILES_SIZES variable. -The dfu_gadget_test_init.sh would generate test images only if they are not -already generated. - -On the target device, environment variable "dfu_alt_info" must contain at -least: - - dfu_test.bin fat 0 6;dfudummy.bin fat 0 6 - -Depending on your device, you may need to replace "fat" with -"ext4", and "6" with the relevant partition number. For reference please -consult the config file for TRATS/TRATS2 devices -(../../include/configs/trats{2}.h) - -One can use fat, ext4 or any other supported file system supported by U-Boot. -These can be created by exporting storage devices via UMS (ums 0 mmc 0) and -using standard tools on host (like mkfs.ext4). - -Example usage: -1. On the target: - setenv dfu_alt_info dfu_test.bin fat 0 6;dfudummy.bin fat 0 6 - dfu 0 mmc 0 -2. On the host: - test/dfu/dfu_gadget_test.sh X Y [test file name] [usb device vendor:product] - e.g. test/dfu/dfu_gadget_test.sh 0 1 - or - e.g. test/dfu/dfu_gadget_test.sh 0 1 ./dat_960.img - or - e.g. test/dfu/dfu_gadget_test.sh 0 1 0451:d022 - or - e.g. test/dfu/dfu_gadget_test.sh 0 1 ./dat_960.img 0451:d022 - -... where X and Y are dfu_test.bin's and dfudummy.bin's alt setting numbers. -They can be obtained from dfu-util -l or $dfu_alt_info. -It is also possible to pass optional [test file name] to force the script to -test one particular file. -If many DFU devices are connected, it may be useful to filter on USB -vendor/product ID (0451:d022). -One can get them by running "lsusb" command on a host PC. diff --git a/test/dfu/dfu_gadget_test.sh b/test/dfu/dfu_gadget_test.sh deleted file mode 100755 index 9c7942257b44..000000000000 --- a/test/dfu/dfu_gadget_test.sh +++ /dev/null @@ -1,108 +0,0 @@ -#! /bin/bash - -# Copyright (C) 2014 Samsung Electronics -# Lukasz Majewski l.majewski@samsung.com -# -# Script fixes, enhancements and testing: -# Stephen Warren swarren@nvidia.com -# -# DFU operation test script -# -# SPDX-License-Identifier: GPL-2.0+ - -set -e # any command return if not equal to zero -clear - -COLOUR_RED="\33[31m" -COLOUR_GREEN="\33[32m" -COLOUR_DEFAULT="\33[0m" - -DIR=./ -SUFFIX=img -RCV_DIR=rcv/ -LOG_FILE=./log/log-`date +%d-%m-%Y_%H-%M-%S` - -cd `dirname $0` -./dfu_gadget_test_init.sh - -cleanup () { - rm -rf $DIR$RCV_DIR -} - -die () { - printf " $COLOUR_RED FAILED $COLOUR_DEFAULT \n" - cleanup - exit 1 -} - -calculate_md5sum () { - MD5SUM=`md5sum $1` - MD5SUM=`echo $MD5SUM | cut -d ' ' -f1` - echo "md5sum:"$MD5SUM -} - -dfu_test_file () { - printf "$COLOUR_GREEN ========================================================================================= $COLOUR_DEFAULT\n" - printf "File:$COLOUR_GREEN %s $COLOUR_DEFAULT\n" $1 - - dfu-util $USB_DEV -D $1 -a $TARGET_ALT_SETTING >> $LOG_FILE 2>&1 || die $? - - echo -n "TX: " - calculate_md5sum $1 - - MD5_TX=$MD5SUM - - dfu-util $USB_DEV -D ${DIR}/dfudummy.bin -a $TARGET_ALT_SETTING_B >> $LOG_FILE 2>&1 || die $? - - N_FILE=$DIR$RCV_DIR${1:2}"_rcv" - - dfu-util $USB_DEV -U $N_FILE -a $TARGET_ALT_SETTING >> $LOG_FILE 2>&1 || die $? - - echo -n "RX: " - calculate_md5sum $N_FILE - MD5_RX=$MD5SUM - - if [ "$MD5_TX" == "$MD5_RX" ]; then - printf " $COLOUR_GREEN -------> OK $COLOUR_DEFAULT \n" - else - printf " $COLOUR_RED -------> FAILED $COLOUR_DEFAULT \n" - cleanup - exit 1 - fi - -} - -printf "$COLOUR_GREEN========================================================================================= $COLOUR_DEFAULT\n" -echo "DFU EP0 transmission test program" -echo "Trouble shoot -> disable DBG (even the KERN_DEBUG) in the UDC driver" -echo "@ -> TRATS2 # dfu 0 mmc 0" -cleanup -mkdir -p $DIR$RCV_DIR -touch $LOG_FILE - -if [ $# -eq 0 ] -then - printf " $COLOUR_RED Please pass alt setting number!! $COLOUR_DEFAULT \n" - exit 0 -fi - -TARGET_ALT_SETTING=$1 -TARGET_ALT_SETTING_B=$2 - -file=$3 -[[ $3 == *':'* ]] && USB_DEV="-d $3" && file="" -[ $# -eq 4 ] && USB_DEV="-d $4" - -if [ -n "$file" ] -then - dfu_test_file $file -else - for f in $DIR*.$SUFFIX - do - dfu_test_file $f - done -fi - -cleanup - -exit 0 diff --git a/test/dfu/dfu_gadget_test_init.sh b/test/dfu/dfu_gadget_test_init.sh deleted file mode 100755 index 640628eecb7a..000000000000 --- a/test/dfu/dfu_gadget_test_init.sh +++ /dev/null @@ -1,45 +0,0 @@ -#! /bin/bash - -# Copyright (C) 2014 Samsung Electronics -# Lukasz Majewski l.majewski@samsung.com -# -# Script fixes, enhancements and testing: -# Stephen Warren swarren@nvidia.com -# -# Script for test files generation -# -# SPDX-License-Identifier: GPL-2.0+ - -set -e # any command return if not equal to zero -clear - -COLOUR_RED="\33[31m" -COLOUR_GREEN="\33[32m" -COLOUR_DEFAULT="\33[0m" - -LOG_DIR="./log" - -if [ $# -eq 0 ]; then - TEST_FILES_SIZES="63 64 65 127 128 129 4095 4096 4097 959 960 961 1048575 1048576 8M" -else - TEST_FILES_SIZES=$@ -fi - -printf "Init script for generating data necessary for DFU test script" - -if [ ! -d $LOG_DIR ]; then - `mkdir $LOG_DIR` -fi - -for size in $TEST_FILES_SIZES -do - FILE="./dat_$size.img" - if [ ! -f $FILE ]; then - dd if=/dev/urandom of="./dat_$size.img" bs=$size count=1 > /dev/null 2>&1 || exit $? - fi -done -dd if=/dev/urandom of="./dfudummy.bin" bs=1024 count=1 > /dev/null 2>&1 || exit $? - -printf "$COLOUR_GREEN OK $COLOUR_DEFAULT \n" - -exit 0 diff --git a/test/py/tests/test_dfu.py b/test/py/tests/test_dfu.py new file mode 100644 index 000000000000..e86ea9a1887c --- /dev/null +++ b/test/py/tests/test_dfu.py @@ -0,0 +1,257 @@ +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +# Test U-Boot's "ums" command. At present, this test only ensures that a UMS +# device can be enumerated by the host/test machine. In the future, this test +# should be enhanced to validate disk IO. + +import os +import os.path +import pytest +import u_boot_utils + +''' +Note: This test relies on: + +a) boardenv_* to contain configuration values to define which USB ports are +available for testing. Without this, this test will be automatically skipped. +For example: + +env__usb_dev_ports = ( + { + "tgt_usb_ctlr": "0", + "host_usb_dev_node": "/dev/usbdev-p2371-2180", + # This parameter is optional /if/ you only have a single board + # attached to your host at a time. + "host_usb_port_path": "3-13", + }, +) + +env__dfu_configs = ( + # eMMC, partition 1 + { + "alt_info": "/dfu_test.bin ext4 0 1;/dfu_dummy.bin ext4 0 1", + "cmd_params": "mmc 0", + }, +) +b) udev rules to set permissions on devices nodes, so that sudo is not +required. For example: + +ACTION=="add", SUBSYSTEM=="block", SUBSYSTEMS=="usb", KERNELS=="3-13", MODE:="666" + +(You may wish to change the group ID instead of setting the permissions wide +open. All that matters is that the user ID running the test can access the +device.) +''' + +# The set of file sizes to test. These values trigger various edge-cases such +# as one less than, equal to, and one greater than typical USB max packet +# sizes, and similar boundary conditions. +test_sizes = ( + 64 - 1, + 64, + 64 + 1, + 128 - 1, + 128, + 128 + 1, + 960 - 1, + 960, + 960 + 1, + 4096 - 1, + 4096, + 4096 + 1, + 1024 * 1024 - 1, + 1024 * 1024, + 8 * 1024 * 1024, +) + +first_usb_dev_port = None + +@pytest.mark.buildconfigspec('cmd_dfu') +def test_dfu(u_boot_console, env__usb_dev_port, env__dfu_config): + '''Test DFU functionality, using numerous file sizes. + + Args: + u_boot_console: A U-Boot console connection. + env__usb_dev_port: The single USB device-mode port specification on + which to run the test. + env__dfu_config: The single DFU (memory region) configuration on which + to run the test. + + Returns: + Nothing. + ''' + + def start_dfu(): + '''Start U-Boot's dfu shell command. + + This also waits for the host-side USB enumeration process to complete. + + Args: + None. + + Returns: + Nothing. + ''' + + u_boot_console.log.action( + 'Starting long-running U-Boot dfu shell command') + + cmd = 'setenv dfu_alt_info "%s"' % env__dfu_config['alt_info'] + u_boot_console.run_command(cmd) + + cmd = 'dfu 0 ' + env__dfu_config['cmd_params'] + u_boot_console.run_command(cmd, wait_for_prompt=False) + u_boot_console.log.action('Waiting for DFU USB device to appear') + fh = u_boot_utils.wait_until_open_succeeds( + env__usb_dev_port['host_usb_dev_node']) + fh.close() + + def stop_dfu(ignore_errors): + '''Stop U-Boot's dfu shell command from executing. + + This also waits for the host-side USB de-enumeration process to + complete. + + Args: + ignore_errors: Ignore any errors. This is useful if an error has + already been detected, and the code is performing best-effort + cleanup. In this case, we do not want to mask the original + error by "honoring" any new errors. + + Returns: + Nothing. + ''' + + try: + u_boot_console.log.action( + 'Stopping long-running U-Boot dfu shell command') + u_boot_console.ctrlc() + u_boot_console.log.action( + 'Waiting for DFU USB device to disappear') + u_boot_utils.wait_until_file_open_fails( + env__usb_dev_port['host_usb_dev_node'], ignore_errors) + except: + if not ignore_errors: + raise + + def run_dfu_util(alt_setting, fn, up_dn_load_arg): + '''Invoke dfu-util on the host. + + Args: + alt_setting: The DFU "alternate setting" identifier to interact + with. + fn: The host-side file name to transfer. + up_dn_load_arg: '-U' or '-D' depending on whether a DFU upload or + download operation should be performed. + + Returns: + Nothing. + ''' + + cmd = ['dfu-util', '-a', str(alt_setting), up_dn_load_arg, fn] + if 'host_usb_port_path' in env__usb_dev_port: + cmd += ['-p', env__usb_dev_port['host_usb_port_path']] + u_boot_utils.run_and_log(u_boot_console, cmd) + u_boot_console.wait_for('Ctrl+C to exit ...') + + def dfu_write(alt_setting, fn): + '''Write a file to the target board using DFU. + + Args: + alt_setting: The DFU "alternate setting" identifier to interact + with. + fn: The host-side file name to transfer. + + Returns: + Nothing. + ''' + + run_dfu_util(alt_setting, fn, '-D') + + def dfu_read(alt_setting, fn): + '''Read a file from the target board using DFU. + + Args: + alt_setting: The DFU "alternate setting" identifier to interact + with. + fn: The host-side file name to transfer. + + Returns: + Nothing. + ''' + + # dfu-util fails reads/uploads if the host file already exists + if os.path.exists(fn): + os.remove(fn) + run_dfu_util(alt_setting, fn, '-U') + + def dfu_write_read_check(size): + '''Test DFU transfers of a specific size of data + + This function first writes data to the board then reads it back and + compares the written and read back data. Measures are taken to avoid + certain types of false positives. + + Args: + size: The data size to test. + + Returns: + Nothing. + ''' + + test_f = u_boot_utils.PersistentRandomFile(u_boot_console, + 'dfu_%d.bin' % size, size) + readback_fn = u_boot_console.config.result_dir + '/dfu_readback.bin' + + u_boot_console.log.action('Writing test data to DFU primary ' + + 'altsetting') + dfu_write(0, test_f.abs_fn) + + u_boot_console.log.action('Writing dummy data to DFU secondary ' + + 'altsetting to clear DFU buffers') + dfu_write(1, dummy_f.abs_fn) + + u_boot_console.log.action('Reading DFU primary altsetting for ' + + 'comparison') + dfu_read(0, readback_fn) + + u_boot_console.log.action('Comparing written and read data') + written_hash = test_f.content_hash + read_back_hash = u_boot_utils.md5sum_file(readback_fn, size) + assert(written_hash == read_back_hash) + + # This test may be executed against multiple USB ports. The test takes a + # long time, so we don't want to do the whole thing each time. Instead, + # execute the full test on the first USB port, and perform a very limited + # test on other ports. In the limited case, we solely validate that the + # host PC can enumerate the U-Boot USB device. + global first_usb_dev_port + if not first_usb_dev_port: + first_usb_dev_port = env__usb_dev_port + if env__usb_dev_port == first_usb_dev_port: + sizes = test_sizes + else: + sizes = [] + + dummy_f = u_boot_utils.PersistentRandomFile(u_boot_console, + 'dfu_dummy.bin', 1024) + + ignore_cleanup_errors = True + try: + start_dfu() + + u_boot_console.log.action( + 'Overwriting DFU primary altsetting with dummy data') + dfu_write(0, dummy_f.abs_fn) + + for size in sizes: + with u_boot_console.log.section("Data size %d" % size): + dfu_write_read_check(size) + # Make the status of each sub-test obvious. If the test didn't + # pass, an exception was thrown so this code isn't executed. + u_boot_console.log.status_pass('OK') + ignore_cleanup_errors = False + finally: + stop_dfu(ignore_cleanup_errors)

Hi Stephen,
From: Stephen Warren swarren@nvidia.com
Add a test of DFU functionality to the Python test suite. The test starts DFU in U-Boot, waits for USB device enumeration on the host, executes dfu-util multiple times to test various transfer sizes, many of which trigger USB driver edge cases, and finally aborts the DFU command in U-Boot.
This test mirrors the functionality previously available via the shell scripts in test/dfu, and hence those are removed too.
Cc: Lukasz Majewski l.majewski@majess.pl Signed-off-by: Stephen Warren swarren@nvidia.com
test/dfu/README | 44 ------- test/dfu/dfu_gadget_test.sh | 108 ---------------- test/dfu/dfu_gadget_test_init.sh | 45 ------- test/py/tests/test_dfu.py | 257 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 257 insertions(+), 197 deletions(-) delete mode 100644 test/dfu/README delete mode 100755 test/dfu/dfu_gadget_test.sh delete mode 100755 test/dfu/dfu_gadget_test_init.sh create mode 100644 test/py/tests/test_dfu.py
diff --git a/test/dfu/README b/test/dfu/README deleted file mode 100644 index 408d5594219a..000000000000 --- a/test/dfu/README +++ /dev/null @@ -1,44 +0,0 @@ -DFU TEST CASE DESCRIPTION:
-The prerequisites for running this script are assured by -dfu_gadget_test_init.sh, which is automatically invoked by dfu_gadget_test.sh. -In this file user is able to generate their own set of test files by altering -the default set of TEST_FILES_SIZES variable. -The dfu_gadget_test_init.sh would generate test images only if they are not -already generated.
-On the target device, environment variable "dfu_alt_info" must contain at -least:
- dfu_test.bin fat 0 6;dfudummy.bin fat 0 6
-Depending on your device, you may need to replace "fat" with -"ext4", and "6" with the relevant partition number. For reference please -consult the config file for TRATS/TRATS2 devices -(../../include/configs/trats{2}.h)
-One can use fat, ext4 or any other supported file system supported by U-Boot. -These can be created by exporting storage devices via UMS (ums 0 mmc 0) and -using standard tools on host (like mkfs.ext4).
-Example usage: -1. On the target:
- setenv dfu_alt_info dfu_test.bin fat 0 6;dfudummy.bin fat 0 6
- dfu 0 mmc 0
-2. On the host:
- test/dfu/dfu_gadget_test.sh X Y [test file name] [usb device
vendor:product]
- e.g. test/dfu/dfu_gadget_test.sh 0 1
- or
- e.g. test/dfu/dfu_gadget_test.sh 0 1 ./dat_960.img
- or
- e.g. test/dfu/dfu_gadget_test.sh 0 1 0451:d022
- or
- e.g. test/dfu/dfu_gadget_test.sh 0 1 ./dat_960.img 0451:d022
-... where X and Y are dfu_test.bin's and dfudummy.bin's alt setting numbers. -They can be obtained from dfu-util -l or $dfu_alt_info. -It is also possible to pass optional [test file name] to force the script to -test one particular file. -If many DFU devices are connected, it may be useful to filter on USB -vendor/product ID (0451:d022). -One can get them by running "lsusb" command on a host PC. diff --git a/test/dfu/dfu_gadget_test.sh b/test/dfu/dfu_gadget_test.sh deleted file mode 100755 index 9c7942257b44..000000000000 --- a/test/dfu/dfu_gadget_test.sh +++ /dev/null @@ -1,108 +0,0 @@ -#! /bin/bash
-# Copyright (C) 2014 Samsung Electronics -# Lukasz Majewski l.majewski@samsung.com -# -# Script fixes, enhancements and testing: -# Stephen Warren swarren@nvidia.com -# -# DFU operation test script -# -# SPDX-License-Identifier: GPL-2.0+
-set -e # any command return if not equal to zero -clear
-COLOUR_RED="\33[31m" -COLOUR_GREEN="\33[32m" -COLOUR_DEFAULT="\33[0m"
-DIR=./ -SUFFIX=img -RCV_DIR=rcv/ -LOG_FILE=./log/log-`date +%d-%m-%Y_%H-%M-%S`
-cd `dirname $0` -./dfu_gadget_test_init.sh
-cleanup () {
- rm -rf $DIR$RCV_DIR
-}
-die () {
- printf " $COLOUR_RED FAILED $COLOUR_DEFAULT \n"
- cleanup
- exit 1
-}
-calculate_md5sum () {
- MD5SUM=`md5sum $1`
- MD5SUM=`echo $MD5SUM | cut -d ' ' -f1`
- echo "md5sum:"$MD5SUM
-}
-dfu_test_file () {
- printf "$COLOUR_GREEN
========================================================================================= $COLOUR_DEFAULT\n"
- printf "File:$COLOUR_GREEN %s $COLOUR_DEFAULT\n" $1
- dfu-util $USB_DEV -D $1 -a $TARGET_ALT_SETTING >> $LOG_FILE 2>&1
|| die $? -
- echo -n "TX: "
- calculate_md5sum $1
- MD5_TX=$MD5SUM
- dfu-util $USB_DEV -D ${DIR}/dfudummy.bin -a
$TARGET_ALT_SETTING_B >> $LOG_FILE 2>&1 || die $? -
- N_FILE=$DIR$RCV_DIR${1:2}"_rcv"
- dfu-util $USB_DEV -U $N_FILE -a $TARGET_ALT_SETTING >> $LOG_FILE
2>&1 || die $? -
- echo -n "RX: "
- calculate_md5sum $N_FILE
- MD5_RX=$MD5SUM
- if [ "$MD5_TX" == "$MD5_RX" ]; then
- printf " $COLOUR_GREEN -------> OK $COLOUR_DEFAULT \n"
- else
- printf " $COLOUR_RED -------> FAILED $COLOUR_DEFAULT \n"
- cleanup
- exit 1
- fi
-}
-printf "$COLOUR_GREEN========================================================================================= $COLOUR_DEFAULT\n" -echo "DFU EP0 transmission test program" -echo "Trouble shoot -> disable DBG (even the KERN_DEBUG) in the UDC driver" -echo "@ -> TRATS2 # dfu 0 mmc 0" -cleanup -mkdir -p $DIR$RCV_DIR -touch $LOG_FILE
-if [ $# -eq 0 ] -then
- printf " $COLOUR_RED Please pass alt setting number!!
$COLOUR_DEFAULT \n"
- exit 0
-fi
-TARGET_ALT_SETTING=$1 -TARGET_ALT_SETTING_B=$2
-file=$3 -[[ $3 == *':'* ]] && USB_DEV="-d $3" && file="" -[ $# -eq 4 ] && USB_DEV="-d $4"
-if [ -n "$file" ] -then
- dfu_test_file $file
-else
- for f in $DIR*.$SUFFIX
- do
dfu_test_file $f
- done
-fi
-cleanup
-exit 0 diff --git a/test/dfu/dfu_gadget_test_init.sh b/test/dfu/dfu_gadget_test_init.sh deleted file mode 100755 index 640628eecb7a..000000000000 --- a/test/dfu/dfu_gadget_test_init.sh +++ /dev/null @@ -1,45 +0,0 @@ -#! /bin/bash
-# Copyright (C) 2014 Samsung Electronics -# Lukasz Majewski l.majewski@samsung.com -# -# Script fixes, enhancements and testing: -# Stephen Warren swarren@nvidia.com -# -# Script for test files generation -# -# SPDX-License-Identifier: GPL-2.0+
-set -e # any command return if not equal to zero -clear
-COLOUR_RED="\33[31m" -COLOUR_GREEN="\33[32m" -COLOUR_DEFAULT="\33[0m"
-LOG_DIR="./log"
-if [ $# -eq 0 ]; then
- TEST_FILES_SIZES="63 64 65 127 128 129 4095 4096 4097 959 960
961 1048575 1048576 8M" -else
- TEST_FILES_SIZES=$@
-fi
-printf "Init script for generating data necessary for DFU test script" - -if [ ! -d $LOG_DIR ]; then
- `mkdir $LOG_DIR`
-fi
-for size in $TEST_FILES_SIZES -do
- FILE="./dat_$size.img"
- if [ ! -f $FILE ]; then
- dd if=/dev/urandom of="./dat_$size.img" bs=$size count=1
/dev/null 2>&1 || exit $?
- fi
-done -dd if=/dev/urandom of="./dfudummy.bin" bs=1024 count=1 > /dev/null 2>&1 || exit $? - -printf "$COLOUR_GREEN OK $COLOUR_DEFAULT \n"
-exit 0 diff --git a/test/py/tests/test_dfu.py b/test/py/tests/test_dfu.py new file mode 100644 index 000000000000..e86ea9a1887c --- /dev/null +++ b/test/py/tests/test_dfu.py @@ -0,0 +1,257 @@ +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0
+# Test U-Boot's "ums" command. At present, this test only ensures that a UMS +# device can be enumerated by the host/test machine. In the future, this test +# should be enhanced to validate disk IO.
Please update this comment to regard dfu, not ums.
+import os +import os.path +import pytest +import u_boot_utils
+''' +Note: This test relies on:
+a) boardenv_* to contain configuration values to define which USB ports are +available for testing. Without this, this test will be automatically skipped. +For example:
+env__usb_dev_ports = (
- {
"tgt_usb_ctlr": "0",
"host_usb_dev_node": "/dev/usbdev-p2371-2180",
# This parameter is optional /if/ you only have a single
board
# attached to your host at a time.
"host_usb_port_path": "3-13",
- },
+)
+env__dfu_configs = (
- # eMMC, partition 1
- {
"alt_info": "/dfu_test.bin ext4 0 1;/dfu_dummy.bin ext4 0 1",
"cmd_params": "mmc 0",
- },
+) +b) udev rules to set permissions on devices nodes, so that sudo is not +required. For example:
+ACTION=="add", SUBSYSTEM=="block", SUBSYSTEMS=="usb", KERNELS=="3-13", MODE:="666" + +(You may wish to change the group ID instead of setting the permissions wide +open. All that matters is that the user ID running the test can access the +device.) +'''
+# The set of file sizes to test. These values trigger various edge-cases such +# as one less than, equal to, and one greater than typical USB max packet +# sizes, and similar boundary conditions. +test_sizes = (
- 64 - 1,
- 64,
- 64 + 1,
- 128 - 1,
- 128,
- 128 + 1,
- 960 - 1,
- 960,
- 960 + 1,
- 4096 - 1,
- 4096,
- 4096 + 1,
- 1024 * 1024 - 1,
- 1024 * 1024,
- 8 * 1024 * 1024,
+)
+first_usb_dev_port = None
+@pytest.mark.buildconfigspec('cmd_dfu') +def test_dfu(u_boot_console, env__usb_dev_port, env__dfu_config):
- '''Test DFU functionality, using numerous file sizes.
- Args:
u_boot_console: A U-Boot console connection.
env__usb_dev_port: The single USB device-mode port
specification on
which to run the test.
env__dfu_config: The single DFU (memory region)
configuration on which
to run the test.
- Returns:
Nothing.
- '''
- def start_dfu():
'''Start U-Boot's dfu shell command.
This also waits for the host-side USB enumeration process to
complete. +
Args:
None.
Returns:
Nothing.
'''
u_boot_console.log.action(
'Starting long-running U-Boot dfu shell command')
cmd = 'setenv dfu_alt_info "%s"' %
env__dfu_config['alt_info']
u_boot_console.run_command(cmd)
cmd = 'dfu 0 ' + env__dfu_config['cmd_params']
u_boot_console.run_command(cmd, wait_for_prompt=False)
u_boot_console.log.action('Waiting for DFU USB device to
appear')
fh = u_boot_utils.wait_until_open_succeeds(
env__usb_dev_port['host_usb_dev_node'])
fh.close()
- def stop_dfu(ignore_errors):
'''Stop U-Boot's dfu shell command from executing.
This also waits for the host-side USB de-enumeration process
to
complete.
Args:
ignore_errors: Ignore any errors. This is useful if an
error has
already been detected, and the code is performing
best-effort
cleanup. In this case, we do not want to mask the
original
error by "honoring" any new errors.
Returns:
Nothing.
'''
try:
u_boot_console.log.action(
'Stopping long-running U-Boot dfu shell command')
u_boot_console.ctrlc()
u_boot_console.log.action(
'Waiting for DFU USB device to disappear')
u_boot_utils.wait_until_file_open_fails(
env__usb_dev_port['host_usb_dev_node'],
ignore_errors)
except:
if not ignore_errors:
raise
- def run_dfu_util(alt_setting, fn, up_dn_load_arg):
'''Invoke dfu-util on the host.
Args:
alt_setting: The DFU "alternate setting" identifier to
interact
with.
fn: The host-side file name to transfer.
up_dn_load_arg: '-U' or '-D' depending on whether a DFU
upload or
download operation should be performed.
Returns:
Nothing.
'''
cmd = ['dfu-util', '-a', str(alt_setting), up_dn_load_arg,
fn]
if 'host_usb_port_path' in env__usb_dev_port:
cmd += ['-p', env__usb_dev_port['host_usb_port_path']]
u_boot_utils.run_and_log(u_boot_console, cmd)
u_boot_console.wait_for('Ctrl+C to exit ...')
- def dfu_write(alt_setting, fn):
'''Write a file to the target board using DFU.
Args:
alt_setting: The DFU "alternate setting" identifier to
interact
with.
fn: The host-side file name to transfer.
Returns:
Nothing.
'''
run_dfu_util(alt_setting, fn, '-D')
- def dfu_read(alt_setting, fn):
'''Read a file from the target board using DFU.
Args:
alt_setting: The DFU "alternate setting" identifier to
interact
with.
fn: The host-side file name to transfer.
Returns:
Nothing.
'''
# dfu-util fails reads/uploads if the host file already
exists
if os.path.exists(fn):
os.remove(fn)
run_dfu_util(alt_setting, fn, '-U')
- def dfu_write_read_check(size):
'''Test DFU transfers of a specific size of data
This function first writes data to the board then reads it
back and
compares the written and read back data. Measures are taken
to avoid
certain types of false positives.
Args:
size: The data size to test.
Returns:
Nothing.
'''
test_f = u_boot_utils.PersistentRandomFile(u_boot_console,
'dfu_%d.bin' % size, size)
readback_fn = u_boot_console.config.result_dir +
'/dfu_readback.bin' +
u_boot_console.log.action('Writing test data to DFU primary
' +
'altsetting')
dfu_write(0, test_f.abs_fn)
u_boot_console.log.action('Writing dummy data to DFU
secondary ' +
'altsetting to clear DFU buffers')
dfu_write(1, dummy_f.abs_fn)
u_boot_console.log.action('Reading DFU primary altsetting
for ' +
'comparison')
dfu_read(0, readback_fn)
u_boot_console.log.action('Comparing written and read data')
written_hash = test_f.content_hash
read_back_hash = u_boot_utils.md5sum_file(readback_fn, size)
assert(written_hash == read_back_hash)
- # This test may be executed against multiple USB ports. The test
takes a
- # long time, so we don't want to do the whole thing each time.
Instead,
- # execute the full test on the first USB port, and perform a
very limited
- # test on other ports. In the limited case, we solely validate
that the
- # host PC can enumerate the U-Boot USB device.
- global first_usb_dev_port
- if not first_usb_dev_port:
first_usb_dev_port = env__usb_dev_port
- if env__usb_dev_port == first_usb_dev_port:
sizes = test_sizes
- else:
sizes = []
- dummy_f = u_boot_utils.PersistentRandomFile(u_boot_console,
'dfu_dummy.bin', 1024)
- ignore_cleanup_errors = True
- try:
start_dfu()
u_boot_console.log.action(
'Overwriting DFU primary altsetting with dummy data')
dfu_write(0, dummy_f.abs_fn)
for size in sizes:
with u_boot_console.log.section("Data size %d" % size):
dfu_write_read_check(size)
# Make the status of each sub-test obvious. If the
test didn't
# pass, an exception was thrown so this code isn't
executed.
u_boot_console.log.status_pass('OK')
ignore_cleanup_errors = False
- finally:
stop_dfu(ignore_cleanup_errors)
Acked-by: Lukasz Majewski l.majewski@samsung.com
Great work Stephen, Thanks !

On 01/21/2016 03:50 AM, Lukasz Majewski wrote:
Hi Stephen,
From: Stephen Warren swarren@nvidia.com
Add a test of DFU functionality to the Python test suite. The test starts DFU in U-Boot, waits for USB device enumeration on the host, executes dfu-util multiple times to test various transfer sizes, many of which trigger USB driver edge cases, and finally aborts the DFU command in U-Boot.
This test mirrors the functionality previously available via the shell scripts in test/dfu, and hence those are removed too.
...
Acked-by: Lukasz Majewski l.majewski@samsung.com
Great work Stephen, Thanks !
Thanks for the review. I'm glad you're OK with the patch, given it deletes your previous script.
I'll hold off on posting v2 (for the comment fix) for a few days in case there are other comments.

On 21 January 2016 at 11:17, Stephen Warren swarren@wwwdotorg.org wrote:
On 01/21/2016 03:50 AM, Lukasz Majewski wrote:
Hi Stephen,
From: Stephen Warren swarren@nvidia.com
Add a test of DFU functionality to the Python test suite. The test starts DFU in U-Boot, waits for USB device enumeration on the host, executes dfu-util multiple times to test various transfer sizes, many of which trigger USB driver edge cases, and finally aborts the DFU command in U-Boot.
This test mirrors the functionality previously available via the shell scripts in test/dfu, and hence those are removed too.
...
Acked-by: Lukasz Majewski l.majewski@samsung.com
Great work Stephen, Thanks !
Thanks for the review. I'm glad you're OK with the patch, given it deletes your previous script.
I'll hold off on posting v2 (for the comment fix) for a few days in case there are other comments.
Acked-by: Simon Glass sjg@chromium.org

Hi Stephen,
From: Stephen Warren swarren@nvidia.com
Currently, Spawn.expect() imposes its timeout solely upon receipt of new data, not on its overall operation. In theory, this could cause the timeout not to fire if U-Boot continually generated output that did not match the expected patterns.
Fix the code to additionally impose a timeout on overall operation, which is the intended mode of operation.
Signed-off-by: Stephen Warren swarren@nvidia.com
test/py/u_boot_spawn.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py index 1baee63df25c..df4c67597cab 100644 --- a/test/py/u_boot_spawn.py +++ b/test/py/u_boot_spawn.py @@ -122,6 +122,7 @@ class Spawn(object): if type(patterns[pi]) == type(''): patterns[pi] = re.compile(patterns[pi])
tstart_s = time.time() try: while True: earliest_m = None
@@ -142,7 +143,11 @@ class Spawn(object): self.after = self.buf[pos:posafter] self.buf = self.buf[posafter:] return earliest_pi
events = self.poll.poll(self.timeout)
tnow_s = time.time()
tdelta_ms = (tnow_s - tstart_s) * 1000
if tdelta_ms > self.timeout:
raise Timeout()
events = self.poll.poll(self.timeout - tdelta_ms) if not events: raise Timeout() c = os.read(self.fd, 1024)
Reviewed-by: Lukasz Majewski l.majewski@samsung.com

On 20 January 2016 at 15:15, Stephen Warren swarren@wwwdotorg.org wrote:
From: Stephen Warren swarren@nvidia.com
Currently, Spawn.expect() imposes its timeout solely upon receipt of new data, not on its overall operation. In theory, this could cause the timeout not to fire if U-Boot continually generated output that did not match the expected patterns.
Fix the code to additionally impose a timeout on overall operation, which is the intended mode of operation.
Signed-off-by: Stephen Warren swarren@nvidia.com
test/py/u_boot_spawn.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
Acked-by: Simon Glass sjg@chromium.org
participants (3)
-
Lukasz Majewski
-
Simon Glass
-
Stephen Warren