[PATCH v7 00/11] labgrid: Provide an integration with Labgrid
Labgrid provides access to a hardware lab in an automated way. It is possible to boot U-Boot on boards in the lab without physically touching them. It relies on relays, USB UARTs and SD muxes, among other things.
By way of background, about 4 years ago I wrong a thing called Labman[1] which allowed my lab of about 30 devices to be operated remotely, using tbot for the console and build integration. While it worked OK and I used it for many bisects, I didn't take it any further.
It turns out that there was already an existing program, called Labgrid, which I did not know about at time (thank you Tom for telling me). It is more rounded than Labman and has a number of advantages:
- does not need udev rules, mostly - has several existing users who rely on it - supports multiple machines exporting their devices
It lacks a 'lab check' feature and a few other things, but these can be remedied.
On and off over the past several weeks I have been experimenting with Labgrid. I have managed to create an initial U-Boot integration (this series) by adding various features to Labgrid[2] and the U-Boot test hooks.
I hope that this might inspire others to set up boards and run tests automatically, rather than relying on infrequent, manual test. Perhaps it may even be possible to have a number of labs available.
Included in the integration are a number of simple scripts which make it easy to connect to boards and run tests:
ub-int <target> Build and boot on a target, starting an interactive session
ub-cli <target> Build and boot on a target, ensure U-Boot starts and provide an interactive session from there
ub-smoke <target> Smoke test U-Boot to check that it boots to a prompt on a target
ub-bisect <target> Bisect a git tree to locate a failure on a particular target
ub-pyt <target> <testspec> Run U-Boot pytests on a target
Some of these help to provide the same tbot[4] workflow which I have relied on for several years, albeit much simpler versions.
The goal here is to create some sort of script which can collect patches from the mailing list, apply them and test them on a selection of boards. I suspect that script already exists, so please let me know what you suggest.
I hope you find this interesting and take a look!
[1] https://github.com/sjg20/u-boot/tree/lab6a [2] https://github.com/labgrid-project/labgrid/pull/1411 [3] https://github.com/sjg20/uboot-test-hooks/tree/labgrid [4] https://tbot.tools/index.html
Changes in v7: - Rebase on testb series - Expand commit message - Rename option to --use-running-system - Rebase on broken-out testb series
Changes in v6: - Rebase without an earlier patch - Avoid doing the special shutdown unless USE_LABGRID is enabled - Drop patch 'test: Pass stderr to stdout'
Changes in v5: - Add a few more comments - Comment out the debugging, which might be useful later - Add a patch to support testing with two board-builds
Changes in v3: - Split out most patches into two new series and update cover letter
Changes in v2: - Only disable echo if a terminal is detected - Avoid running a docker image for skipped lab tests
Simon Glass (11): test: Allow signaling that U-Boot is ready test: Release board after tests complete test: Allow connecting to a running board test: Introduce the concept of a role test: Introduce lab mode test: Improve handling of sending commands test: Avoid double echo when starting up test: Try to shut down the lab console gracefully test: Add a section for closing the connection test: Support testing with two board-builds CI: Allow running tests on sjg lab
.gitlab-ci.yml | 153 ++++++++++++++++++++++++++ test/py/conftest.py | 73 ++++++++++-- test/py/u_boot_console_base.py | 123 +++++++++++++++------ test/py/u_boot_console_exec_attach.py | 31 ++++-- test/py/u_boot_spawn.py | 36 +++++- 5 files changed, 363 insertions(+), 53 deletions(-)
When Labgrid is used, it can get U-Boot ready for running tests. It prints a message when it has done so.
Add logic to detect this message and accept it.
Note that this does not change pytest, which still (also) looks for the U-Boot banner. This change merely makes it possible for pytest to believe Labgrid when it says that the board is ready for use.
In several cases, the board starts up and Labgrid receives some initial output, then pytest starts and misses some of that output, because it came in while Labgrid had the console open. Then pytest fails because it doesn't see the expected banners.
With this change, Labgrid handles getting U-Boot to a prompt, in a fully reliable manner. Then pytest starts up and can simply start running its tests.
But, again, this does not prevent pytest from handling a banner if one is provided (e.g. if not using the Labgrid integration).
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v7: - Rebase on testb series - Expand commit message
test/py/u_boot_console_base.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index d8d0bdf9fd4..965832a4989 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -23,6 +23,7 @@ pattern_stop_autoboot_prompt = re.compile('Hit any key to stop autoboot: ') pattern_unknown_command = re.compile('Unknown command '.*' - try 'help'') pattern_error_notification = re.compile('## Error: ') pattern_error_please_reset = re.compile('### ERROR ### Please RESET the board ###') +pattern_ready_prompt = re.compile('U-Boot is ready')
PAT_ID = 0 PAT_RE = 1 @@ -200,15 +201,15 @@ class ConsoleBase(object): self.bad_pattern_ids[m - 1]) self.u_boot_version_string = self.p.after while True: - m = self.p.expect([self.prompt_compiled, + m = self.p.expect([self.prompt_compiled, pattern_ready_prompt, pattern_stop_autoboot_prompt] + self.bad_patterns) - if m == 0: + if m == 0 or m == 1: break - if m == 1: + if m == 2: self.p.send(' ') continue raise BootFail('Bad pattern found on console: ' + - self.bad_pattern_ids[m - 2]) + self.bad_pattern_ids[m - 3])
finally: self.log.timestamp()
On Wed, Oct 09, 2024 at 07:51:35PM -0600, Simon Glass wrote:
When Labgrid is used, it can get U-Boot ready for running tests. It prints a message when it has done so.
Add logic to detect this message and accept it.
Note that this does not change pytest, which still (also) looks for the U-Boot banner. This change merely makes it possible for pytest to believe Labgrid when it says that the board is ready for use.
In several cases, the board starts up and Labgrid receives some initial output, then pytest starts and misses some of that output, because it came in while Labgrid had the console open. Then pytest fails because it doesn't see the expected banners.
With this change, Labgrid handles getting U-Boot to a prompt, in a fully reliable manner. Then pytest starts up and can simply start running its tests.
But, again, this does not prevent pytest from handling a banner if one is provided (e.g. if not using the Labgrid integration).
Signed-off-by: Simon Glass sjg@chromium.org
I _may_ see this myself sometimes, but I'm not sure. But I'm not sure where the bug would be between us and labgrid. Looking at test/py/u_boot_console_exec_attach.py we grab the console and then reset the target. If the spawn hasn't completed, especially before the power cycle has completed (which in labgrid defaults to 2 seconds) we perhaps need our own delay. But as I'll point out further on, NOT counting the banners means we aren't able to verify other functionality. Even if they are tests that I think you don't like because they verify the functionality of resetting the device.
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:35PM -0600, Simon Glass wrote:
When Labgrid is used, it can get U-Boot ready for running tests. It prints a message when it has done so.
Add logic to detect this message and accept it.
Note that this does not change pytest, which still (also) looks for the U-Boot banner. This change merely makes it possible for pytest to believe Labgrid when it says that the board is ready for use.
In several cases, the board starts up and Labgrid receives some initial output, then pytest starts and misses some of that output, because it came in while Labgrid had the console open. Then pytest fails because it doesn't see the expected banners.
With this change, Labgrid handles getting U-Boot to a prompt, in a fully reliable manner. Then pytest starts up and can simply start running its tests.
But, again, this does not prevent pytest from handling a banner if one is provided (e.g. if not using the Labgrid integration).
Signed-off-by: Simon Glass sjg@chromium.org
I _may_ see this myself sometimes, but I'm not sure.
OK, but it isn't really necessary for you to see it yourself. I've described above the repeatable problems I have with this approach.
But I'm not sure where the bug would be between us and labgrid. Looking at test/py/u_boot_console_exec_attach.py we grab the console and then reset the target. If the spawn hasn't completed, especially before the power cycle has completed (which in labgrid defaults to 2 seconds) we perhaps need our own delay. But as I'll point out further on, NOT counting the banners means we aren't able to verify other functionality. Even if they are tests that I think you don't like because they verify the functionality of resetting the device.
Yes, that's a use case that needs special support. If a board is booted from USB, then resets, it will run whatever it can find, perhaps just hang. Labgrid doesn't know the board is resetting and cannot therefore ensure that it sends the firmware over USB a second time (or sets the recovery line on the board). Resetting a board works OK using SDwire and some other cases, but will fail on quite a few of my boards.
Regards, Simon
On Thu, Oct 31, 2024 at 07:00:10PM +0100, Simon Glass wrote:
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:35PM -0600, Simon Glass wrote:
When Labgrid is used, it can get U-Boot ready for running tests. It prints a message when it has done so.
Add logic to detect this message and accept it.
Note that this does not change pytest, which still (also) looks for the U-Boot banner. This change merely makes it possible for pytest to believe Labgrid when it says that the board is ready for use.
In several cases, the board starts up and Labgrid receives some initial output, then pytest starts and misses some of that output, because it came in while Labgrid had the console open. Then pytest fails because it doesn't see the expected banners.
With this change, Labgrid handles getting U-Boot to a prompt, in a fully reliable manner. Then pytest starts up and can simply start running its tests.
But, again, this does not prevent pytest from handling a banner if one is provided (e.g. if not using the Labgrid integration).
Signed-off-by: Simon Glass sjg@chromium.org
I _may_ see this myself sometimes, but I'm not sure.
OK, but it isn't really necessary for you to see it yourself. I've described above the repeatable problems I have with this approach.
Well, with what I just now said in reply to v6 of this series:
Reviewed-by: Tom Rini trini@konsulko.com
So long as v8 also includes a much longer commit message.
But I'm not sure where the bug would be between us and labgrid. Looking at test/py/u_boot_console_exec_attach.py we grab the console and then reset the target. If the spawn hasn't completed, especially before the power cycle has completed (which in labgrid defaults to 2 seconds) we perhaps need our own delay. But as I'll point out further on, NOT counting the banners means we aren't able to verify other functionality. Even if they are tests that I think you don't like because they verify the functionality of resetting the device.
Yes, that's a use case that needs special support. If a board is booted from USB, then resets, it will run whatever it can find, perhaps just hang. Labgrid doesn't know the board is resetting and cannot therefore ensure that it sends the firmware over USB a second time (or sets the recovery line on the board). Resetting a board works OK using SDwire and some other cases, but will fail on quite a few of my boards.
It is indeed tricky matching up and catching everything given the number of platforms where it's technically possible to do CI on the firmware but practically a large complicated pain.
When a board is finished with, the lab may want to power it off, or perform some other function. Add a new script which is called when tests are complete.
Signed-off-by: Simon Glass sjg@chromium.org ---
(no changes since v1)
test/py/u_boot_console_exec_attach.py | 10 ++++++++++ 1 file changed, 10 insertions(+)
diff --git a/test/py/u_boot_console_exec_attach.py b/test/py/u_boot_console_exec_attach.py index 8dd8cc1230c..5f4916b7da2 100644 --- a/test/py/u_boot_console_exec_attach.py +++ b/test/py/u_boot_console_exec_attach.py @@ -70,3 +70,13 @@ class ConsoleExecAttach(ConsoleBase): raise
return s + + def close(self): + super().close() + + self.log.action('Releasing board') + args = [self.config.board_type, self.config.board_identity] + cmd = ['u-boot-test-release'] + args + runner = self.log.get_runner(cmd[0], sys.stdout) + runner.run(cmd) + runner.close()
On Wed, Oct 09, 2024 at 07:51:36PM -0600, Simon Glass wrote:
When a board is finished with, the lab may want to power it off, or perform some other function. Add a new script which is called when tests are complete.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com
But please note I don't see an up to date version of the u-boot-test-hooks side of this which would include the release wrapper, and presumably files.
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:36PM -0600, Simon Glass wrote:
When a board is finished with, the lab may want to power it off, or perform some other function. Add a new script which is called when tests are complete.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com
But please note I don't see an up to date version of the u-boot-test-hooks side of this which would include the release wrapper, and presumably files.
Please see my patch from late August "Provide some basic scripts for Labgrid integration"
Regards, Simon
On Thu, Oct 31, 2024 at 07:00:14PM +0100, Simon Glass wrote:
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:36PM -0600, Simon Glass wrote:
When a board is finished with, the lab may want to power it off, or perform some other function. Add a new script which is called when tests are complete.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com
But please note I don't see an up to date version of the u-boot-test-hooks side of this which would include the release wrapper, and presumably files.
Please see my patch from late August "Provide some basic scripts for Labgrid integration"
Yes. There was quite a long discussion on patch 3/3 and I'm not 100% sure where the next steps there are on what the next iteration should have. If the release script was it's own patch then I believe the first 3 patches could be merged today and then that leaves the actual labgrid stuff that conflicts with my setup to figure out.
Hi Tom,
On Thu, 31 Oct 2024 at 19:34, Tom Rini trini@konsulko.com wrote:
On Thu, Oct 31, 2024 at 07:00:14PM +0100, Simon Glass wrote:
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:36PM -0600, Simon Glass wrote:
When a board is finished with, the lab may want to power it off, or perform some other function. Add a new script which is called when tests are complete.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com
But please note I don't see an up to date version of the u-boot-test-hooks side of this which would include the release wrapper, and presumably files.
Please see my patch from late August "Provide some basic scripts for Labgrid integration"
Yes. There was quite a long discussion on patch 3/3 and I'm not 100% sure where the next steps there are on what the next iteration should have. If the release script was it's own patch then I believe the first 3 patches could be merged today and then that leaves the actual labgrid stuff that conflicts with my setup to figure out.
I tested my patches with my non-Labgrid setup (pyttest, tbot) and it still worked. That's not to say there aren't problems, but I am hopeful that it will not affect you.
Regards, Simon
On Fri, Nov 01, 2024 at 04:33:47PM +0100, Simon Glass wrote:
Hi Tom,
On Thu, 31 Oct 2024 at 19:34, Tom Rini trini@konsulko.com wrote:
On Thu, Oct 31, 2024 at 07:00:14PM +0100, Simon Glass wrote:
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:36PM -0600, Simon Glass wrote:
When a board is finished with, the lab may want to power it off, or perform some other function. Add a new script which is called when tests are complete.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com
But please note I don't see an up to date version of the u-boot-test-hooks side of this which would include the release wrapper, and presumably files.
Please see my patch from late August "Provide some basic scripts for Labgrid integration"
Yes. There was quite a long discussion on patch 3/3 and I'm not 100% sure where the next steps there are on what the next iteration should have. If the release script was it's own patch then I believe the first 3 patches could be merged today and then that leaves the actual labgrid stuff that conflicts with my setup to figure out.
I tested my patches with my non-Labgrid setup (pyttest, tbot) and it still worked. That's not to say there aren't problems, but I am hopeful that it will not affect you.
I can re-name my scripts to labgrid-trini or whatever, sure. But it wasn't clear if what you were suggesting was flexible enough in general for how others setup labs too.
Hi Tom,
On Fri, 1 Nov 2024 at 18:08, Tom Rini trini@konsulko.com wrote:
On Fri, Nov 01, 2024 at 04:33:47PM +0100, Simon Glass wrote:
Hi Tom,
On Thu, 31 Oct 2024 at 19:34, Tom Rini trini@konsulko.com wrote:
On Thu, Oct 31, 2024 at 07:00:14PM +0100, Simon Glass wrote:
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:36PM -0600, Simon Glass wrote:
When a board is finished with, the lab may want to power it off, or perform some other function. Add a new script which is called when tests are complete.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com
But please note I don't see an up to date version of the u-boot-test-hooks side of this which would include the release wrapper, and presumably files.
Please see my patch from late August "Provide some basic scripts for Labgrid integration"
Yes. There was quite a long discussion on patch 3/3 and I'm not 100% sure where the next steps there are on what the next iteration should have. If the release script was it's own patch then I believe the first 3 patches could be merged today and then that leaves the actual labgrid stuff that conflicts with my setup to figure out.
I tested my patches with my non-Labgrid setup (pyttest, tbot) and it still worked. That's not to say there aren't problems, but I am hopeful that it will not affect you.
I can re-name my scripts to labgrid-trini or whatever, sure. But it wasn't clear if what you were suggesting was flexible enough in general for how others setup labs too.
Well I can rename mine to labgrid-sjg if that helps. It makes sense, since we have two ways of operating.
Regards, Simon
Sometimes we know that the board is already running the right software, so provide an option to allow running of tests directly, without first resetting the board.
This saves time when re-running a test where only the Python code is changing.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v7: - Rename option to --use-running-system
test/py/conftest.py | 3 +++ test/py/u_boot_console_base.py | 14 ++++++++++---- test/py/u_boot_console_exec_attach.py | 21 ++++++++++++--------- 3 files changed, 25 insertions(+), 13 deletions(-)
diff --git a/test/py/conftest.py b/test/py/conftest.py index 46a410cf268..7a52b966154 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -80,6 +80,8 @@ def pytest_addoption(parser): parser.addoption('--gdbserver', default=None, help='Run sandbox under gdbserver. The argument is the channel '+ 'over which gdbserver should communicate, e.g. localhost:1234') + parser.addoption('--use-running-system', default=False, action='store_true', + help="Assume that U-Boot is ready and don't wait for a prompt")
def run_build(config, source_dir, build_dir, board_type, log): """run_build: Build U-Boot @@ -254,6 +256,7 @@ def pytest_configure(config): ubconfig.board_type = board_type ubconfig.board_identity = board_identity ubconfig.gdbserver = gdbserver + ubconfig.use_running_system = config.getoption('use_running_system') ubconfig.dtb = build_dir + '/arch/sandbox/dts/test.dtb' ubconfig.connection_ok = True
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 965832a4989..311e649d55b 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -441,11 +441,17 @@ class ConsoleBase(object): if not self.config.gdbserver: self.p.timeout = TIMEOUT_MS self.p.logfile_read = self.logstream - if expect_reset: - loop_num = 2 + if self.config.use_running_system: + # Send an empty command to set up the 'expect' logic. This has + # the side effect of ensuring that there was no partial command + # line entered + self.run_command(' ') else: - loop_num = 1 - self.wait_for_boot_prompt(loop_num = loop_num) + if expect_reset: + loop_num = 2 + else: + loop_num = 1 + self.wait_for_boot_prompt(loop_num = loop_num) self.at_prompt = True self.at_prompt_logevt = self.logstream.logfile.cur_evt except Exception as ex: diff --git a/test/py/u_boot_console_exec_attach.py b/test/py/u_boot_console_exec_attach.py index 5f4916b7da2..8b253b4451d 100644 --- a/test/py/u_boot_console_exec_attach.py +++ b/test/py/u_boot_console_exec_attach.py @@ -59,15 +59,18 @@ class ConsoleExecAttach(ConsoleBase): args = [self.config.board_type, self.config.board_identity] s = Spawn(['u-boot-test-console'] + args)
- try: - self.log.action('Resetting board') - cmd = ['u-boot-test-reset'] + args - runner = self.log.get_runner(cmd[0], sys.stdout) - runner.run(cmd) - runner.close() - except: - s.close() - raise + if self.config.use_running_system: + self.log.action('Connecting to board without reset') + else: + try: + self.log.action('Resetting board') + cmd = ['u-boot-test-reset'] + args + runner = self.log.get_runner(cmd[0], sys.stdout) + runner.run(cmd) + runner.close() + except: + s.close() + raise
return s
On Wed, Oct 09, 2024 at 07:51:37PM -0600, Simon Glass wrote:
Sometimes we know that the board is already running the right software, so provide an option to allow running of tests directly, without first resetting the board.
This saves time when re-running a test where only the Python code is changing.
Signed-off-by: Simon Glass sjg@chromium.org
I'm not NAK'ing this, but this is not a open to errors kind of change that I like. It also feels like an unrelated "I find this change useful" kind of change.
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:37PM -0600, Simon Glass wrote:
Sometimes we know that the board is already running the right software, so provide an option to allow running of tests directly, without first resetting the board.
This saves time when re-running a test where only the Python code is changing.
Signed-off-by: Simon Glass sjg@chromium.org
I'm not NAK'ing this, but this is not a open to errors kind of change that I like. It also feels like an unrelated "I find this change useful" kind of change.
Are you saying this change is open to errors?
It just avoids needing to reset the board when I know it is already in a good state. So it is a time-saver.
Regards, Simon
On Thu, Oct 31, 2024 at 07:00:24PM +0100, Simon Glass wrote:
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:37PM -0600, Simon Glass wrote:
Sometimes we know that the board is already running the right software, so provide an option to allow running of tests directly, without first resetting the board.
This saves time when re-running a test where only the Python code is changing.
Signed-off-by: Simon Glass sjg@chromium.org
I'm not NAK'ing this, but this is not a open to errors kind of change that I like. It also feels like an unrelated "I find this change useful" kind of change.
Are you saying this change is open to errors?
It just avoids needing to reset the board when I know it is already in a good state. So it is a time-saver.
Yes, I'm saying it's open to errors since your assumption is "the board is running what I want to test". As a human, I've violated that assumption in the past running tests and short-circuiting the loop to just re-test what's running.
Hi Tom,
On Thu, 31 Oct 2024 at 19:42, Tom Rini trini@konsulko.com wrote:
On Thu, Oct 31, 2024 at 07:00:24PM +0100, Simon Glass wrote:
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:37PM -0600, Simon Glass wrote:
Sometimes we know that the board is already running the right software, so provide an option to allow running of tests directly, without first resetting the board.
This saves time when re-running a test where only the Python code is changing.
Signed-off-by: Simon Glass sjg@chromium.org
I'm not NAK'ing this, but this is not a open to errors kind of change that I like. It also feels like an unrelated "I find this change useful" kind of change.
Are you saying this change is open to errors?
It just avoids needing to reset the board when I know it is already in a good state. So it is a time-saver.
Yes, I'm saying it's open to errors since your assumption is "the board is running what I want to test". As a human, I've violated that assumption in the past running tests and short-circuiting the loop to just re-test what's running.
Yes, that's why it is an option. I sometimes find myself iterating on the pytest part of a test, where U-Boot is not changing.
Regards, Simon
In Labgrid there is the concept of a 'role', which is similar to the U-Boot board ID in U-Boot's pytest subsystem.
The role indicates both the target and information about the U-Boot build to use. It can also provide any amount of other configuration. The information is obtained using the 'labgrid-client query' operation.
Make use of this in tests, so that only the role is required in gitlab and other situations. The board type and other things can be queried as needed.
Use a new 'u-boot-test-getrole' script to obtain the requested information.
With this it is possible to run lab tests in gitlab with just a single 'ROLE' variable for each board.
Signed-off-by: Simon Glass sjg@chromium.org ---
(no changes since v5)
Changes in v5: - Add a few more comments - Comment out the debugging, which might be useful later
test/py/conftest.py | 38 ++++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-)
diff --git a/test/py/conftest.py b/test/py/conftest.py index 7a52b966154..4a45e080df9 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -23,6 +23,7 @@ from pathlib import Path import pytest import re from _pytest.runner import runtestprotocol +import subprocess import sys from u_boot_spawn import BootFail, Timeout, Unexpected, handle_exception
@@ -80,6 +81,7 @@ def pytest_addoption(parser): parser.addoption('--gdbserver', default=None, help='Run sandbox under gdbserver. The argument is the channel '+ 'over which gdbserver should communicate, e.g. localhost:1234') + parser.addoption('--role', help='U-Boot board role (for Labgrid)') parser.addoption('--use-running-system', default=False, action='store_true', help="Assume that U-Boot is ready and don't wait for a prompt")
@@ -131,12 +133,40 @@ def get_details(config): str: Build directory str: Source directory """ - board_type = config.getoption('board_type') - board_identity = config.getoption('board_identity') + role = config.getoption('role') + + # Get a few provided parameters build_dir = config.getoption('build_dir') + if role: + # When using a role, build_dir and build_dir_extra are normally not set, + # since they are picked up from Labgrid via the u-boot-test-getrole + # script + board_identity = role + cmd = ['u-boot-test-getrole', role, '--configure'] + env = os.environ.copy() + if build_dir: + env['U_BOOT_BUILD_DIR'] = build_dir + proc = subprocess.run(cmd, capture_output=True, encoding='utf-8', + env=env) + if proc.returncode: + raise ValueError(proc.stderr) + # For debugging + # print('conftest: lab:', proc.stdout) + vals = {} + for line in proc.stdout.splitlines(): + item, value = line.split(' ', maxsplit=1) + k = item.split(':')[-1] + vals[k] = value + # For debugging + # print('conftest: lab info:', vals) + board_type, default_build_dir, source_dir = (vals['board'], + vals['build_dir'], vals['source_dir']) + else: + board_type = config.getoption('board_type') + board_identity = config.getoption('board_identity')
- source_dir = os.path.dirname(os.path.dirname(TEST_PY_DIR)) - default_build_dir = source_dir + '/build-' + board_type + source_dir = os.path.dirname(os.path.dirname(TEST_PY_DIR)) + default_build_dir = source_dir + '/build-' + board_type if not build_dir: build_dir = default_build_dir
On Wed, Oct 09, 2024 at 07:51:38PM -0600, Simon Glass wrote:
In Labgrid there is the concept of a 'role', which is similar to the U-Boot board ID in U-Boot's pytest subsystem.
The role indicates both the target and information about the U-Boot build to use. It can also provide any amount of other configuration. The information is obtained using the 'labgrid-client query' operation.
Make use of this in tests, so that only the role is required in gitlab and other situations. The board type and other things can be queried as needed.
Use a new 'u-boot-test-getrole' script to obtain the requested information.
With this it is possible to run lab tests in gitlab with just a single 'ROLE' variable for each board.
Signed-off-by: Simon Glass sjg@chromium.org
Aside from the "look ma, only one conf file" trick, I still do not see how this differs from the typical board/id system we use today. One can easily make N symlinks to the generic conf file.
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:38PM -0600, Simon Glass wrote:
In Labgrid there is the concept of a 'role', which is similar to the U-Boot board ID in U-Boot's pytest subsystem.
The role indicates both the target and information about the U-Boot build to use. It can also provide any amount of other configuration. The information is obtained using the 'labgrid-client query' operation.
Make use of this in tests, so that only the role is required in gitlab and other situations. The board type and other things can be queried as needed.
Use a new 'u-boot-test-getrole' script to obtain the requested information.
With this it is possible to run lab tests in gitlab with just a single 'ROLE' variable for each board.
Signed-off-by: Simon Glass sjg@chromium.org
Aside from the "look ma, only one conf file" trick, I still do not see how this differs from the typical board/id system we use today. One can easily make N symlinks to the generic conf file.
You can still do that if you like. I have always found that approach confusing, so I prefer the configuration to be in Labgrid. It also allows me to connect to the boards using an interactive console, something that isn't possible without some kind of mapping.
Regards, Simon
On Thu, Oct 31, 2024 at 07:01:56PM +0100, Simon Glass wrote:
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:38PM -0600, Simon Glass wrote:
In Labgrid there is the concept of a 'role', which is similar to the U-Boot board ID in U-Boot's pytest subsystem.
The role indicates both the target and information about the U-Boot build to use. It can also provide any amount of other configuration. The information is obtained using the 'labgrid-client query' operation.
Make use of this in tests, so that only the role is required in gitlab and other situations. The board type and other things can be queried as needed.
Use a new 'u-boot-test-getrole' script to obtain the requested information.
With this it is possible to run lab tests in gitlab with just a single 'ROLE' variable for each board.
Signed-off-by: Simon Glass sjg@chromium.org
Aside from the "look ma, only one conf file" trick, I still do not see how this differs from the typical board/id system we use today. One can easily make N symlinks to the generic conf file.
You can still do that if you like. I have always found that approach confusing, so I prefer the configuration to be in Labgrid. It also allows me to connect to the boards using an interactive console, something that isn't possible without some kind of mapping.
This is where we get to the point where I honestly don't know. On the one hand, there's the position I usually take. Minimize changes that are due to preferences and locally wrap things as needed. I have a wrapper around buildman for example, for the tasks I need. I have wrappers around labgrid. There's a few things mine does that yours doesn't but there's a few things your does that mine doesn't. On the other hand, I'm sure I'm not optimally using labgrid for managing the lab. My hesitation boils down to that your implementation seems phrased / positioned as the only way to do things. And no two labs are alike.
There is quite a bit of code in pytest to try to start up U-Boot on a board, with timeouts, expects, etc.
This is tedious to maintain and is peripheral to the test system's purpose. It seems better to put this logic in the lab itself, where is can provide such support.
With Labgrid we can use the UbootStrategy class to get the board into a useful state, however it needs to do it. Then it can report to pytest by writing a suitable string along with the U-Boot version it detected.
Add support for detecting 'lab mode' and simply assume that all is well in that case. Collect the version string when Labgrid says it is ready.
Signed-off-by: Simon Glass sjg@chromium.org ---
(no changes since v1)
test/py/u_boot_console_base.py | 68 ++++++++++++++++++++++++++-------- 1 file changed, 53 insertions(+), 15 deletions(-)
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 311e649d55b..573d76dd832 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -23,13 +23,21 @@ pattern_stop_autoboot_prompt = re.compile('Hit any key to stop autoboot: ') pattern_unknown_command = re.compile('Unknown command '.*' - try 'help'') pattern_error_notification = re.compile('## Error: ') pattern_error_please_reset = re.compile('### ERROR ### Please RESET the board ###') -pattern_ready_prompt = re.compile('U-Boot is ready') +pattern_ready_prompt = re.compile('{lab ready in (.*)s: (.*)}') +pattern_lab_mode = re.compile('{lab mode.*}')
PAT_ID = 0 PAT_RE = 1
# Timeout before expecting the console to be ready (in milliseconds) -TIMEOUT_MS = 30000 +TIMEOUT_MS = 30000 # Standard timeout + +# Timeout for board preparation in lab mode. This needs to be enough to build +# U-Boot, write it to the board and then boot the board. Since this process is +# under the control of another program (e.g. Labgrid), it will failure sooner +# if something goes way. So use a very long timeout here to cover all possible +# situations. +TIMEOUT_PREPARE_MS = 3 * 60 * 1000
bad_pattern_defs = ( ('spl_signon', pattern_u_boot_spl_signon), @@ -143,6 +151,7 @@ class ConsoleBase(object):
self.at_prompt = False self.at_prompt_logevt = None + self.lab_mode = False
def get_spawn(self): # This is not called, ssubclass must define this. @@ -176,40 +185,69 @@ class ConsoleBase(object): self.p.close() self.logstream.close()
+ def set_lab_mode(self): + """Select lab mode + + This tells us that we will get a 'lab ready' message when the board is + ready for use. We don't need to look for signon messages. + """ + self.log.info(f'test.py: Lab mode is active') + self.p.timeout = TIMEOUT_PREPARE_MS + self.lab_mode = True + def wait_for_boot_prompt(self, loop_num = 1): """Wait for the boot up until command prompt. This is for internal use only. """ try: + self.log.info('Waiting for U-Boot to be ready') bcfg = self.config.buildconfig config_spl_serial = bcfg.get('config_spl_serial', 'n') == 'y' env_spl_skipped = self.config.env.get('env__spl_skipped', False) env_spl_banner_times = self.config.env.get('env__spl_banner_times', 1)
- while loop_num > 0: + while not self.lab_mode and loop_num > 0: loop_num -= 1 while config_spl_serial and not env_spl_skipped and env_spl_banner_times > 0: - m = self.p.expect([pattern_u_boot_spl_signon] + - self.bad_patterns) - if m != 0: + m = self.p.expect([pattern_u_boot_spl_signon, + pattern_lab_mode] + self.bad_patterns) + if m == 1: + self.set_lab_mode() + break + elif m != 0: raise BootFail('Bad pattern found on SPL console: ' + - self.bad_pattern_ids[m - 1]) + self.bad_pattern_ids[m - 1]) env_spl_banner_times -= 1
- m = self.p.expect([pattern_u_boot_main_signon] + self.bad_patterns) - if m != 0: - raise BootFail('Bad pattern found on console: ' + - self.bad_pattern_ids[m - 1]) - self.u_boot_version_string = self.p.after + if not self.lab_mode: + m = self.p.expect([pattern_u_boot_main_signon, + pattern_lab_mode] + self.bad_patterns) + if m == 1: + self.set_lab_mode() + elif m != 0: + raise BootFail('Bad pattern found on console: ' + + self.bad_pattern_ids[m - 1]) + if not self.lab_mode: + self.u_boot_version_string = self.p.after while True: m = self.p.expect([self.prompt_compiled, pattern_ready_prompt, pattern_stop_autoboot_prompt] + self.bad_patterns) - if m == 0 or m == 1: + if m == 0: + self.log.info(f'Found ready prompt {m}') + break + elif m == 1: + m = pattern_ready_prompt.search(self.p.after) + self.u_boot_version_string = m.group(2) + self.log.info(f'Lab: Board is ready') + self.p.timeout = TIMEOUT_MS break if m == 2: + self.log.info(f'Found autoboot prompt {m}') self.p.send(' ') continue - raise BootFail('Bad pattern found on console: ' + - self.bad_pattern_ids[m - 3]) + if not self.lab_mode: + raise BootFail('Missing prompt / ready message on console: ' + + self.bad_pattern_ids[m - 3]) + self.log.info(f'U-Boot is ready')
finally: self.log.timestamp()
On Wed, Oct 09, 2024 at 07:51:39PM -0600, Simon Glass wrote:
There is quite a bit of code in pytest to try to start up U-Boot on a board, with timeouts, expects, etc.
This is tedious to maintain and is peripheral to the test system's purpose. It seems better to put this logic in the lab itself, where is can provide such support.
With Labgrid we can use the UbootStrategy class to get the board into a useful state, however it needs to do it. Then it can report to pytest by writing a suitable string along with the U-Boot version it detected.
Add support for detecting 'lab mode' and simply assume that all is well in that case. Collect the version string when Labgrid says it is ready.
Signed-off-by: Simon Glass sjg@chromium.org
First, I worry that this kind of change will make verifying boards can reset harder, not easier. But second, taking the "tedious" part and putting in another project's config file instead makes it no less "tedious". If I follow the labgrid code right, it's got the same kind of "check for a string" that we have, except it's not caring about the correct number / order of banner prints. Which I find worrisome and a problem because of the times I've caught "oops, didn't actually test the right stuff" because that was wrong.
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:39PM -0600, Simon Glass wrote:
There is quite a bit of code in pytest to try to start up U-Boot on a board, with timeouts, expects, etc.
This is tedious to maintain and is peripheral to the test system's purpose. It seems better to put this logic in the lab itself, where is can provide such support.
With Labgrid we can use the UbootStrategy class to get the board into a useful state, however it needs to do it. Then it can report to pytest by writing a suitable string along with the U-Boot version it detected.
Add support for detecting 'lab mode' and simply assume that all is well in that case. Collect the version string when Labgrid says it is ready.
Signed-off-by: Simon Glass sjg@chromium.org
First, I worry that this kind of change will make verifying boards can reset harder, not easier. But second, taking the "tedious" part and putting in another project's config file instead makes it no less "tedious". If I follow the labgrid code right, it's got the same kind of "check for a string" that we have, except it's not caring about the correct number / order of banner prints. Which I find worrisome and a problem because of the times I've caught "oops, didn't actually test the right stuff" because that was wrong.
I am not asking you to use this Labgrid integration., though Please feel free to do things as you currently do. For me, this works very well and avoids the persistent failures I was otherwise seeing.
Regards, Simon
We expect commands to be echoed and this should happen quite quickly, since U-Boot is sitting at the prompt waiting for a command.
Reduce the timeout for this situation. Try to produce a more useful error message when something goes wrong. Also handle the case where the connection has gone away since the last command was issued.
Signed-off-by: Simon Glass sjg@chromium.org ---
(no changes since v1)
test/py/u_boot_console_base.py | 35 ++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-)
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 573d76dd832..42c81b2d13f 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -31,6 +31,7 @@ PAT_RE = 1
# Timeout before expecting the console to be ready (in milliseconds) TIMEOUT_MS = 30000 # Standard timeout +TIMEOUT_CMD_MS = 10000 # Command-echo timeout
# Timeout for board preparation in lab mode. This needs to be enough to build # U-Boot, write it to the board and then boot the board. Since this process is @@ -300,22 +301,28 @@ class ConsoleBase(object):
try: self.at_prompt = False + if not self.p: + raise BootFail( + f"Lab failure: Connection lost when sending command '{cmd}'") + if send_nl: cmd += '\n' - while cmd: - # Limit max outstanding data, so UART FIFOs don't overflow - chunk = cmd[:self.max_fifo_fill] - cmd = cmd[self.max_fifo_fill:] - self.p.send(chunk) - if not wait_for_echo: - continue - chunk = re.escape(chunk) - chunk = chunk.replace('\\n', '[\r\n]') - m = self.p.expect([chunk] + self.bad_patterns) - if m != 0: - self.at_prompt = False - raise BootFail('Bad pattern found on console: ' + - self.bad_pattern_ids[m - 1]) + rem = cmd # Remaining to be sent + with self.temporary_timeout(TIMEOUT_CMD_MS): + while rem: + # Limit max outstanding data, so UART FIFOs don't overflow + chunk = rem[:self.max_fifo_fill] + rem = rem[self.max_fifo_fill:] + self.p.send(chunk) + if not wait_for_echo: + continue + chunk = re.escape(chunk) + chunk = chunk.replace('\\n', '[\r\n]') + m = self.p.expect([chunk] + self.bad_patterns) + if m != 0: + self.at_prompt = False + raise BootFail(f"Failed to get echo on console (cmd '{cmd}':rem '{rem}'): " + + self.bad_pattern_ids[m - 1]) if not wait_for_prompt: return if wait_for_reboot:
On Wed, Oct 09, 2024 at 07:51:40PM -0600, Simon Glass wrote:
We expect commands to be echoed and this should happen quite quickly, since U-Boot is sitting at the prompt waiting for a command.
Reduce the timeout for this situation. Try to produce a more useful error message when something goes wrong. Also handle the case where the connection has gone away since the last command was issued.
Signed-off-by: Simon Glass sjg@chromium.org
(no changes since v1)
test/py/u_boot_console_base.py | 35 ++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-)
This looks good and I wish you had refactored the series to include this (as well as patch #7) in the generic test/py improvements series.
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:40PM -0600, Simon Glass wrote:
We expect commands to be echoed and this should happen quite quickly, since U-Boot is sitting at the prompt waiting for a command.
Reduce the timeout for this situation. Try to produce a more useful error message when something goes wrong. Also handle the case where the connection has gone away since the last command was issued.
Signed-off-by: Simon Glass sjg@chromium.org
(no changes since v1)
test/py/u_boot_console_base.py | 35 ++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-)
This looks good and I wish you had refactored the series to include this (as well as patch #7) in the generic test/py improvements series.
Good, well let's get this series in and we will both be happy :-)
Regards, Simon
There is a very annoying bug at present where the terminal echos part of the first command sent to the board. This happens because the terminal is still set to echo for a period until Labgrid starts up and can change this.
Fix this by disabling echo (and other terminal features) as soon as the spawn happens.
Signed-off-by: Simon Glass sjg@chromium.org ---
(no changes since v6)
Changes in v6: - Rebase without an earlier patch
Changes in v2: - Only disable echo if a terminal is detected
test/py/u_boot_spawn.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py index 24d369035e5..f2398098a00 100644 --- a/test/py/u_boot_spawn.py +++ b/test/py/u_boot_spawn.py @@ -11,6 +11,8 @@ import pty import pytest import signal import select +import sys +import termios import time import traceback
@@ -115,11 +117,23 @@ class Spawn: finally: os._exit(255)
+ old = None try: + if os.isatty(sys.stdout.fileno()): + new = termios.tcgetattr(self.fd) + old = new + new[3] = new[3] & ~(termios.ICANON | termios.ISIG) + new[3] = new[3] & ~termios.ECHO + new[6][termios.VMIN] = 0 + new[6][termios.VTIME] = 0 + termios.tcsetattr(self.fd, termios.TCSANOW, new) + self.poll = select.poll() self.poll.register(self.fd, select.POLLIN | select.POLLPRI | select.POLLERR | select.POLLHUP | select.POLLNVAL) except: + if old: + termios.tcsetattr(self.fd, termios.TCSANOW, old) self.close() raise
Send the Labgrid quit characters to ask it to exit gracefully. This typically allows it to power off the board being used. Only do this when labgrid is being used (detected with an env var).
If that doesn't work, try the less graceful approach.
Signed-off-by: Simon Glass sjg@chromium.org ---
(no changes since v6)
Changes in v6: - Avoid doing the special shutdown unless USE_LABGRID is enabled
test/py/u_boot_spawn.py | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)
diff --git a/test/py/u_boot_spawn.py b/test/py/u_boot_spawn.py index f2398098a00..72d3d5e77b1 100644 --- a/test/py/u_boot_spawn.py +++ b/test/py/u_boot_spawn.py @@ -16,6 +16,9 @@ import termios import time import traceback
+# Character to send (twice) to exit the terminal +EXIT_CHAR = 0x1d # FS (Ctrl + ]) + class Timeout(Exception): """An exception sub-class that indicates that a timeout occurred."""
@@ -303,15 +306,28 @@ class Spawn: None.
Returns: - Nothing. + str: Type of closure completed """ - + # For Labgrid, ask it is exit gracefully, so it can transition the board + # to the final state (like 'off') before exiting. + if os.environ.get('USE_LABGRID'): + self.send(chr(EXIT_CHAR) * 2) + + # Wait about 10 seconds for Labgrid to close and power off the board + for _ in range(100): + if not self.isalive(): + return 'normal' + time.sleep(0.1) + + # That didn't work, so try closing the PTY os.close(self.fd) for _ in range(100): if not self.isalive(): - break + return 'break' time.sleep(0.1)
+ return 'timeout' + def get_expect_output(self): """Return the output read by expect()
On Wed, Oct 09, 2024 at 07:51:42PM -0600, Simon Glass wrote:
Send the Labgrid quit characters to ask it to exit gracefully. This typically allows it to power off the board being used. Only do this when labgrid is being used (detected with an env var).
If that doesn't work, try the less graceful approach.
Signed-off-by: Simon Glass sjg@chromium.org
I forget why we need this, and not just have the release hook power off the board, as part of releasing it.
Hi Tom,
On Wed, 30 Oct 2024 at 00:53, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:42PM -0600, Simon Glass wrote:
Send the Labgrid quit characters to ask it to exit gracefully. This typically allows it to power off the board being used. Only do this when labgrid is being used (detected with an env var).
If that doesn't work, try the less graceful approach.
Signed-off-by: Simon Glass sjg@chromium.org
I forget why we need this, and not just have the release hook power off the board, as part of releasing it.
It allows labgrid to exit gracefully, rather than being killed. This allows it to do the 'off' part of strategy (which may power it off) and release the device. Without this, every board disconnect leaves the board in a bad state, requiring separate steps to recover the board, then power it off.
For your lab, this won't be used, since USE_LABGRID is not set.
Regards, SImon
This can take a while and involve multiple steps (e.g. turning the board back off). Add a section for it and show the output.
Signed-off-by: Simon Glass sjg@chromium.org ---
(no changes since v1)
test/py/u_boot_console_base.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/test/py/u_boot_console_base.py b/test/py/u_boot_console_base.py index 42c81b2d13f..fa9cd57b04b 100644 --- a/test/py/u_boot_console_base.py +++ b/test/py/u_boot_console_base.py @@ -183,7 +183,10 @@ class ConsoleBase(object): """
if self.p: - self.p.close() + self.log.start_section('Stopping U-Boot') + close_type = self.p.close() + self.log.info(f'Close type: {close_type}') + self.log.end_section('Stopping U-Boot') self.logstream.close()
def set_lab_mode(self):
On Wed, Oct 09, 2024 at 07:51:43PM -0600, Simon Glass wrote:
This can take a while and involve multiple steps (e.g. turning the board back off). Add a section for it and show the output.
Signed-off-by: Simon Glass sjg@chromium.org
Logging can be useful, so this is fine.
Reviewed-by: Tom Rini trini@konsulko.com
The Beagleplay board uses two entirely separate builds to produce an image, rather than using an SPL build for this purpose.
Handle this in test.py by adding more parameters.
Signed-off-by: Simon Glass sjg@chromium.org ---
(no changes since v5)
Changes in v5: - Add a patch to support testing with two board-builds
test/py/conftest.py | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-)
diff --git a/test/py/conftest.py b/test/py/conftest.py index 4a45e080df9..a19f94a635a 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -66,12 +66,16 @@ def pytest_addoption(parser):
parser.addoption('--build-dir', default=None, help='U-Boot build directory (O=)') + parser.addoption('--build-dir-extra', default=None, + help='U-Boot build directory for extra build (O=)') parser.addoption('--result-dir', default=None, help='U-Boot test result/tmp directory') parser.addoption('--persistent-data-dir', default=None, help='U-Boot test persistent generated data directory') parser.addoption('--board-type', '--bd', '-B', default='sandbox', help='U-Boot board type') + parser.addoption('--board-type-extra', '--bde', default='sandbox', + help='U-Boot extra board type') parser.addoption('--board-identity', '--id', default='na', help='U-Boot board identity/instance') parser.addoption('--build', default=False, action='store_true', @@ -129,14 +133,17 @@ def get_details(config): Returns: tuple: str: Board type (U-Boot build name) + str: Extra board type (where two U-Boot builds are needed) str: Identity for the lab board str: Build directory + str: Extra build directory (where two U-Boot builds are needed) str: Source directory """ role = config.getoption('role')
# Get a few provided parameters build_dir = config.getoption('build_dir') + build_dir_extra = config.getoption('build_dir_extra') if role: # When using a role, build_dir and build_dir_extra are normally not set, # since they are picked up from Labgrid via the u-boot-test-getrole @@ -146,6 +153,8 @@ def get_details(config): env = os.environ.copy() if build_dir: env['U_BOOT_BUILD_DIR'] = build_dir + if build_dir_extra: + env['U_BOOT_BUILD_DIR_EXTRA'] = build_dir_extra proc = subprocess.run(cmd, capture_output=True, encoding='utf-8', env=env) if proc.returncode: @@ -159,24 +168,36 @@ def get_details(config): vals[k] = value # For debugging # print('conftest: lab info:', vals) - board_type, default_build_dir, source_dir = (vals['board'], - vals['build_dir'], vals['source_dir']) + + # Read the build directories here, in case none were provided in the + # command-line arguments + (board_type, board_type_extra, default_build_dir, + default_build_dir_extra, source_dir) = (vals['board'], + vals['board_extra'], vals['build_dir'], vals['build_dir_extra'], + vals['source_dir']) else: board_type = config.getoption('board_type') + board_type_extra = config.getoption('board_type_extra') board_identity = config.getoption('board_identity')
source_dir = os.path.dirname(os.path.dirname(TEST_PY_DIR)) default_build_dir = source_dir + '/build-' + board_type + default_build_dir_extra = source_dir + '/build-' + board_type_extra + + # Use the provided command-line arguments if present, else fall back to if not build_dir: build_dir = default_build_dir + if not build_dir_extra: + build_dir_extra = default_build_dir_extra
- return board_type, board_identity, build_dir, source_dir + return (board_type, board_type_extra, board_identity, build_dir, + build_dir_extra, source_dir)
def pytest_xdist_setupnodes(config, specs): """Clear out any 'done' file from a previous build""" global build_done_file
- build_dir = get_details(config)[2] + build_dir = get_details(config)[3]
build_done_file = Path(build_dir) / 'build.done' if build_done_file.exists(): @@ -216,7 +237,8 @@ def pytest_configure(config): global console global ubconfig
- board_type, board_identity, build_dir, source_dir = get_details(config) + (board_type, board_type_extra, board_identity, build_dir, build_dir_extra, + source_dir) = get_details(config)
board_type_filename = board_type.replace('-', '_') board_identity_filename = board_identity.replace('-', '_') @@ -281,9 +303,11 @@ def pytest_configure(config): ubconfig.test_py_dir = TEST_PY_DIR ubconfig.source_dir = source_dir ubconfig.build_dir = build_dir + ubconfig.build_dir_extra = build_dir_extra ubconfig.result_dir = result_dir ubconfig.persistent_data_dir = persistent_data_dir ubconfig.board_type = board_type + ubconfig.board_type_extra = board_type_extra ubconfig.board_identity = board_identity ubconfig.gdbserver = gdbserver ubconfig.use_running_system = config.getoption('use_running_system') @@ -292,10 +316,12 @@ def pytest_configure(config):
env_vars = ( 'board_type', + 'board_type_extra', 'board_identity', 'source_dir', 'test_py_dir', 'build_dir', + 'build_dir_extra', 'result_dir', 'persistent_data_dir', )
On Wed, Oct 09, 2024 at 07:51:44PM -0600, Simon Glass wrote:
The Beagleplay board uses two entirely separate builds to produce an image, rather than using an SPL build for this purpose.
JFTR, this is an incorrect explanation of what's going on. The TI K3 families have both a Cortex-R core and a Cortex-A core and the R core needs to come up before the A core. But in both cases we have U-Boot SPL and then U-Boot proper being used.
This is aside from having come back to re-review the series and still feel like this is taking the wrong approach to the problem. We wouldn't add a --optee-build-directory for going off and building OP-TEE and so forth. It's just the case that we need assorted binaries for the system and some of them come from building U-Boot for another platform.
Hi Tom,
On Wed, 30 Oct 2024 at 00:52, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:44PM -0600, Simon Glass wrote:
The Beagleplay board uses two entirely separate builds to produce an image, rather than using an SPL build for this purpose.
JFTR, this is an incorrect explanation of what's going on. The TI K3 families have both a Cortex-R core and a Cortex-A core and the R core needs to come up before the A core. But in both cases we have U-Boot SPL and then U-Boot proper being used.
Yes, I understand that. I will reword the commit message.
This is aside from having come back to re-review the series and still feel like this is taking the wrong approach to the problem. We wouldn't add a --optee-build-directory for going off and building OP-TEE and so forth. It's just the case that we need assorted binaries for the system and some of them come from building U-Boot for another platform.
Yes, and that is supported by this integration. For example my rk3399 board brings in optee as a binary.
Regards, Simon
On Thu, Oct 31, 2024 at 06:59:37PM +0100, Simon Glass wrote:
Hi Tom,
On Wed, 30 Oct 2024 at 00:52, Tom Rini trini@konsulko.com wrote:
On Wed, Oct 09, 2024 at 07:51:44PM -0600, Simon Glass wrote:
The Beagleplay board uses two entirely separate builds to produce an image, rather than using an SPL build for this purpose.
JFTR, this is an incorrect explanation of what's going on. The TI K3 families have both a Cortex-R core and a Cortex-A core and the R core needs to come up before the A core. But in both cases we have U-Boot SPL and then U-Boot proper being used.
Yes, I understand that. I will reword the commit message.
Thanks.
Add a way to run tests on a real hardware lab. This is in the very early experimental stages. There are only 23 boards and 3 of those are broken! (bob, ff3399, samus). A fourth fails due to problems with the TPM tests.
To try this, assuming you have gitlab access, set SJG_LAB=1, e.g.:
git push -o ci.variable="SJG_LAB=1" dm HEAD:try
This relies on the two previous series targeted at -next as well as the bugfix series for -master
Signed-off-by: Simon Glass sjg@chromium.org Reviewed-by: Andrejs Cainikovs andrejs.cainikovs@toradex.com ---
Changes in v7: - Rebase on broken-out testb series
Changes in v6: - Drop patch 'test: Pass stderr to stdout'
Changes in v3: - Split out most patches into two new series and update cover letter
Changes in v2: - Avoid running a docker image for skipped lab tests
.gitlab-ci.yml | 153 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 7d621031b85..4d66e53b5b2 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -17,6 +17,7 @@ stages: - testsuites - test.py - world build + - sjg-lab
.buildman_and_testpy_template: &buildman_and_testpy_dfn stage: test.py @@ -491,3 +492,155 @@ coreboot test.py: TEST_PY_TEST_SPEC: "not sleep" TEST_PY_ID: "--id qemu" <<: *buildman_and_testpy_dfn + +.lab_template: &lab_dfn + stage: sjg-lab + rules: + - if: $SJG_LAB == "1" + when: always + - when: manual + tags: [ 'lab' ] + script: + - if [[ -z "${SJG_LAB}" ]]; then + exit 0; + fi + # Environment: + # SRC - source tree + # OUT - output directory for builds + - export SRC="$(pwd)" + - export OUT="${SRC}/build/${BOARD}" + - export PATH=$PATH:~/bin + - export PATH=$PATH:/vid/software/devel/ubtest/u-boot-test-hooks/bin + + # Load it on the device + - ret=0 + - echo "role ${ROLE}" + - export strategy="-s uboot -e off" + # export verbose="-v" + - ${SRC}/test/py/test.py --role ${ROLE} --build-dir "${OUT}" + --capture=tee-sys -k "not bootstd"|| ret=$? + - U_BOOT_BOARD_IDENTITY="${ROLE}" u-boot-test-release || true + - if [[ $ret -ne 0 ]]; then + exit $ret; + fi + artifacts: + when: always + paths: + - "build/${BOARD}/test-log.html" + - "build/${BOARD}/multiplexed_log.css" + expire_in: 1 week + +rpi3: + variables: + ROLE: rpi3 + <<: *lab_dfn + +opi_pc: + variables: + ROLE: opi_pc + <<: *lab_dfn + +pcduino3_nano: + variables: + ROLE: pcduino3_nano + <<: *lab_dfn + +samus: + variables: + ROLE: samus + <<: *lab_dfn + +link: + variables: + ROLE: link + <<: *lab_dfn + +jerry: + variables: + ROLE: jerry + <<: *lab_dfn + +minnowmax: + variables: + ROLE: minnowmax + <<: *lab_dfn + +opi_pc2: + variables: + ROLE: opi_pc2 + <<: *lab_dfn + +bpi: + variables: + ROLE: bpi + <<: *lab_dfn + +rpi2: + variables: + ROLE: rpi2 + <<: *lab_dfn + +bob: + variables: + ROLE: bob + <<: *lab_dfn + +ff3399: + variables: + ROLE: ff3399 + <<: *lab_dfn + +coral: + variables: + ROLE: coral + <<: *lab_dfn + +rpi3z: + variables: + ROLE: rpi3z + <<: *lab_dfn + +bbb: + variables: + ROLE: bbb + <<: *lab_dfn + +kevin: + variables: + ROLE: kevin + <<: *lab_dfn + +pine64: + variables: + ROLE: pine64 + <<: *lab_dfn + +c4: + variables: + ROLE: c4 + <<: *lab_dfn + +rpi4: + variables: + ROLE: rpi4 + <<: *lab_dfn + +rpi0: + variables: + ROLE: rpi0 + <<: *lab_dfn + +snow: + variables: + ROLE: snow + <<: *lab_dfn + +pcduino3: + variables: + ROLE: pcduino3 + <<: *lab_dfn + +nyan-big: + variables: + ROLE: nyan-big + <<: *lab_dfn
Hi Tom,
On Thu, 10 Oct 2024 at 03:51, Simon Glass sjg@chromium.org wrote:
Labgrid provides access to a hardware lab in an automated way. It is possible to boot U-Boot on boards in the lab without physically touching them. It relies on relays, USB UARTs and SD muxes, among other things.
By way of background, about 4 years ago I wrong a thing called Labman[1] which allowed my lab of about 30 devices to be operated remotely, using tbot for the console and build integration. While it worked OK and I used it for many bisects, I didn't take it any further.
It turns out that there was already an existing program, called Labgrid, which I did not know about at time (thank you Tom for telling me). It is more rounded than Labman and has a number of advantages:
- does not need udev rules, mostly
- has several existing users who rely on it
- supports multiple machines exporting their devices
It lacks a 'lab check' feature and a few other things, but these can be remedied.
On and off over the past several weeks I have been experimenting with Labgrid. I have managed to create an initial U-Boot integration (this series) by adding various features to Labgrid[2] and the U-Boot test hooks.
I hope that this might inspire others to set up boards and run tests automatically, rather than relying on infrequent, manual test. Perhaps it may even be possible to have a number of labs available.
Included in the integration are a number of simple scripts which make it easy to connect to boards and run tests:
ub-int <target> Build and boot on a target, starting an interactive session
ub-cli <target> Build and boot on a target, ensure U-Boot starts and provide an interactive session from there
ub-smoke <target> Smoke test U-Boot to check that it boots to a prompt on a target
ub-bisect <target> Bisect a git tree to locate a failure on a particular target
ub-pyt <target> <testspec> Run U-Boot pytests on a target
Some of these help to provide the same tbot[4] workflow which I have relied on for several years, albeit much simpler versions.
The goal here is to create some sort of script which can collect patches from the mailing list, apply them and test them on a selection of boards. I suspect that script already exists, so please let me know what you suggest.
I hope you find this interesting and take a look!
[1] https://github.com/sjg20/u-boot/tree/lab6a [2] https://github.com/labgrid-project/labgrid/pull/1411 [3] https://github.com/sjg20/uboot-test-hooks/tree/labgrid [4] https://tbot.tools/index.html
Changes in v7:
- Rebase on testb series
- Expand commit message
- Rename option to --use-running-system
- Rebase on broken-out testb series
Changes in v6:
- Rebase without an earlier patch
- Avoid doing the special shutdown unless USE_LABGRID is enabled
- Drop patch 'test: Pass stderr to stdout'
Changes in v5:
- Add a few more comments
- Comment out the debugging, which might be useful later
- Add a patch to support testing with two board-builds
Changes in v3:
- Split out most patches into two new series and update cover letter
Changes in v2:
- Only disable echo if a terminal is detected
- Avoid running a docker image for skipped lab tests
Simon Glass (11): test: Allow signaling that U-Boot is ready test: Release board after tests complete test: Allow connecting to a running board test: Introduce the concept of a role test: Introduce lab mode test: Improve handling of sending commands test: Avoid double echo when starting up test: Try to shut down the lab console gracefully test: Add a section for closing the connection test: Support testing with two board-builds CI: Allow running tests on sjg lab
.gitlab-ci.yml | 153 ++++++++++++++++++++++++++ test/py/conftest.py | 73 ++++++++++-- test/py/u_boot_console_base.py | 123 +++++++++++++++------ test/py/u_boot_console_exec_attach.py | 31 ++++-- test/py/u_boot_spawn.py | 36 +++++- 5 files changed, 363 insertions(+), 53 deletions(-)
-- 2.34.1
Is there any news on these patches? I would dearly love to enable my lab and this is what is needed...
Regards, Simon
On Mon, Oct 21, 2024 at 01:12:49PM +0200, Simon Glass wrote:
Hi Tom,
On Thu, 10 Oct 2024 at 03:51, Simon Glass sjg@chromium.org wrote:
Labgrid provides access to a hardware lab in an automated way. It is possible to boot U-Boot on boards in the lab without physically touching them. It relies on relays, USB UARTs and SD muxes, among other things.
By way of background, about 4 years ago I wrong a thing called Labman[1] which allowed my lab of about 30 devices to be operated remotely, using tbot for the console and build integration. While it worked OK and I used it for many bisects, I didn't take it any further.
It turns out that there was already an existing program, called Labgrid, which I did not know about at time (thank you Tom for telling me). It is more rounded than Labman and has a number of advantages:
- does not need udev rules, mostly
- has several existing users who rely on it
- supports multiple machines exporting their devices
It lacks a 'lab check' feature and a few other things, but these can be remedied.
On and off over the past several weeks I have been experimenting with Labgrid. I have managed to create an initial U-Boot integration (this series) by adding various features to Labgrid[2] and the U-Boot test hooks.
I hope that this might inspire others to set up boards and run tests automatically, rather than relying on infrequent, manual test. Perhaps it may even be possible to have a number of labs available.
Included in the integration are a number of simple scripts which make it easy to connect to boards and run tests:
ub-int <target> Build and boot on a target, starting an interactive session
ub-cli <target> Build and boot on a target, ensure U-Boot starts and provide an interactive session from there
ub-smoke <target> Smoke test U-Boot to check that it boots to a prompt on a target
ub-bisect <target> Bisect a git tree to locate a failure on a particular target
ub-pyt <target> <testspec> Run U-Boot pytests on a target
Some of these help to provide the same tbot[4] workflow which I have relied on for several years, albeit much simpler versions.
The goal here is to create some sort of script which can collect patches from the mailing list, apply them and test them on a selection of boards. I suspect that script already exists, so please let me know what you suggest.
I hope you find this interesting and take a look!
[1] https://github.com/sjg20/u-boot/tree/lab6a [2] https://github.com/labgrid-project/labgrid/pull/1411 [3] https://github.com/sjg20/uboot-test-hooks/tree/labgrid [4] https://tbot.tools/index.html
Changes in v7:
- Rebase on testb series
- Expand commit message
- Rename option to --use-running-system
- Rebase on broken-out testb series
Changes in v6:
- Rebase without an earlier patch
- Avoid doing the special shutdown unless USE_LABGRID is enabled
- Drop patch 'test: Pass stderr to stdout'
Changes in v5:
- Add a few more comments
- Comment out the debugging, which might be useful later
- Add a patch to support testing with two board-builds
Changes in v3:
- Split out most patches into two new series and update cover letter
Changes in v2:
- Only disable echo if a terminal is detected
- Avoid running a docker image for skipped lab tests
Simon Glass (11): test: Allow signaling that U-Boot is ready test: Release board after tests complete test: Allow connecting to a running board test: Introduce the concept of a role test: Introduce lab mode test: Improve handling of sending commands test: Avoid double echo when starting up test: Try to shut down the lab console gracefully test: Add a section for closing the connection test: Support testing with two board-builds CI: Allow running tests on sjg lab
.gitlab-ci.yml | 153 ++++++++++++++++++++++++++ test/py/conftest.py | 73 ++++++++++-- test/py/u_boot_console_base.py | 123 +++++++++++++++------ test/py/u_boot_console_exec_attach.py | 31 ++++-- test/py/u_boot_spawn.py | 36 +++++- 5 files changed, 363 insertions(+), 53 deletions(-)
-- 2.34.1
Is there any news on these patches? I would dearly love to enable my lab and this is what is needed...
As our approaches are radically different I continue to be looking for feedback from a third person to set things up and say what does / doesn't work for them and their lab. I assume you're just pulling this in locally for use at least.
Hi Tom,
On Mon, 21 Oct 2024 at 20:07, Tom Rini trini@konsulko.com wrote:
On Mon, Oct 21, 2024 at 01:12:49PM +0200, Simon Glass wrote:
Hi Tom,
On Thu, 10 Oct 2024 at 03:51, Simon Glass sjg@chromium.org wrote:
Labgrid provides access to a hardware lab in an automated way. It is possible to boot U-Boot on boards in the lab without physically touching them. It relies on relays, USB UARTs and SD muxes, among other things.
By way of background, about 4 years ago I wrong a thing called Labman[1] which allowed my lab of about 30 devices to be operated remotely, using tbot for the console and build integration. While it worked OK and I used it for many bisects, I didn't take it any further.
It turns out that there was already an existing program, called Labgrid, which I did not know about at time (thank you Tom for telling me). It is more rounded than Labman and has a number of advantages:
- does not need udev rules, mostly
- has several existing users who rely on it
- supports multiple machines exporting their devices
It lacks a 'lab check' feature and a few other things, but these can be remedied.
On and off over the past several weeks I have been experimenting with Labgrid. I have managed to create an initial U-Boot integration (this series) by adding various features to Labgrid[2] and the U-Boot test hooks.
I hope that this might inspire others to set up boards and run tests automatically, rather than relying on infrequent, manual test. Perhaps it may even be possible to have a number of labs available.
Included in the integration are a number of simple scripts which make it easy to connect to boards and run tests:
ub-int <target> Build and boot on a target, starting an interactive session
ub-cli <target> Build and boot on a target, ensure U-Boot starts and provide an interactive session from there
ub-smoke <target> Smoke test U-Boot to check that it boots to a prompt on a target
ub-bisect <target> Bisect a git tree to locate a failure on a particular target
ub-pyt <target> <testspec> Run U-Boot pytests on a target
Some of these help to provide the same tbot[4] workflow which I have relied on for several years, albeit much simpler versions.
The goal here is to create some sort of script which can collect patches from the mailing list, apply them and test them on a selection of boards. I suspect that script already exists, so please let me know what you suggest.
I hope you find this interesting and take a look!
[1] https://github.com/sjg20/u-boot/tree/lab6a [2] https://github.com/labgrid-project/labgrid/pull/1411 [3] https://github.com/sjg20/uboot-test-hooks/tree/labgrid [4] https://tbot.tools/index.html
Changes in v7:
- Rebase on testb series
- Expand commit message
- Rename option to --use-running-system
- Rebase on broken-out testb series
Changes in v6:
- Rebase without an earlier patch
- Avoid doing the special shutdown unless USE_LABGRID is enabled
- Drop patch 'test: Pass stderr to stdout'
Changes in v5:
- Add a few more comments
- Comment out the debugging, which might be useful later
- Add a patch to support testing with two board-builds
Changes in v3:
- Split out most patches into two new series and update cover letter
Changes in v2:
- Only disable echo if a terminal is detected
- Avoid running a docker image for skipped lab tests
Simon Glass (11): test: Allow signaling that U-Boot is ready test: Release board after tests complete test: Allow connecting to a running board test: Introduce the concept of a role test: Introduce lab mode test: Improve handling of sending commands test: Avoid double echo when starting up test: Try to shut down the lab console gracefully test: Add a section for closing the connection test: Support testing with two board-builds CI: Allow running tests on sjg lab
.gitlab-ci.yml | 153 ++++++++++++++++++++++++++ test/py/conftest.py | 73 ++++++++++-- test/py/u_boot_console_base.py | 123 +++++++++++++++------ test/py/u_boot_console_exec_attach.py | 31 ++++-- test/py/u_boot_spawn.py | 36 +++++- 5 files changed, 363 insertions(+), 53 deletions(-)
-- 2.34.1
Is there any news on these patches? I would dearly love to enable my lab and this is what is needed...
As our approaches are radically different I continue to be looking for feedback from a third person to set things up and say what does / doesn't work for them and their lab. I assume you're just pulling this in locally for use at least.
No, I cannot pull this in locally. I would like to set up gitlab so I (and others) can push things to it for running on the lab. Without my patches the lab only works on a few boards.
Could you perhaps try them out on your lab, to check they don't break anything?
Regards, Simon
On Tue, Oct 22, 2024 at 02:15:40PM +0200, Simon Glass wrote:
Hi Tom,
On Mon, 21 Oct 2024 at 20:07, Tom Rini trini@konsulko.com wrote:
On Mon, Oct 21, 2024 at 01:12:49PM +0200, Simon Glass wrote:
Hi Tom,
On Thu, 10 Oct 2024 at 03:51, Simon Glass sjg@chromium.org wrote:
Labgrid provides access to a hardware lab in an automated way. It is possible to boot U-Boot on boards in the lab without physically touching them. It relies on relays, USB UARTs and SD muxes, among other things.
By way of background, about 4 years ago I wrong a thing called Labman[1] which allowed my lab of about 30 devices to be operated remotely, using tbot for the console and build integration. While it worked OK and I used it for many bisects, I didn't take it any further.
It turns out that there was already an existing program, called Labgrid, which I did not know about at time (thank you Tom for telling me). It is more rounded than Labman and has a number of advantages:
- does not need udev rules, mostly
- has several existing users who rely on it
- supports multiple machines exporting their devices
It lacks a 'lab check' feature and a few other things, but these can be remedied.
On and off over the past several weeks I have been experimenting with Labgrid. I have managed to create an initial U-Boot integration (this series) by adding various features to Labgrid[2] and the U-Boot test hooks.
I hope that this might inspire others to set up boards and run tests automatically, rather than relying on infrequent, manual test. Perhaps it may even be possible to have a number of labs available.
Included in the integration are a number of simple scripts which make it easy to connect to boards and run tests:
ub-int <target> Build and boot on a target, starting an interactive session
ub-cli <target> Build and boot on a target, ensure U-Boot starts and provide an interactive session from there
ub-smoke <target> Smoke test U-Boot to check that it boots to a prompt on a target
ub-bisect <target> Bisect a git tree to locate a failure on a particular target
ub-pyt <target> <testspec> Run U-Boot pytests on a target
Some of these help to provide the same tbot[4] workflow which I have relied on for several years, albeit much simpler versions.
The goal here is to create some sort of script which can collect patches from the mailing list, apply them and test them on a selection of boards. I suspect that script already exists, so please let me know what you suggest.
I hope you find this interesting and take a look!
[1] https://github.com/sjg20/u-boot/tree/lab6a [2] https://github.com/labgrid-project/labgrid/pull/1411 [3] https://github.com/sjg20/uboot-test-hooks/tree/labgrid [4] https://tbot.tools/index.html
Changes in v7:
- Rebase on testb series
- Expand commit message
- Rename option to --use-running-system
- Rebase on broken-out testb series
Changes in v6:
- Rebase without an earlier patch
- Avoid doing the special shutdown unless USE_LABGRID is enabled
- Drop patch 'test: Pass stderr to stdout'
Changes in v5:
- Add a few more comments
- Comment out the debugging, which might be useful later
- Add a patch to support testing with two board-builds
Changes in v3:
- Split out most patches into two new series and update cover letter
Changes in v2:
- Only disable echo if a terminal is detected
- Avoid running a docker image for skipped lab tests
Simon Glass (11): test: Allow signaling that U-Boot is ready test: Release board after tests complete test: Allow connecting to a running board test: Introduce the concept of a role test: Introduce lab mode test: Improve handling of sending commands test: Avoid double echo when starting up test: Try to shut down the lab console gracefully test: Add a section for closing the connection test: Support testing with two board-builds CI: Allow running tests on sjg lab
.gitlab-ci.yml | 153 ++++++++++++++++++++++++++ test/py/conftest.py | 73 ++++++++++-- test/py/u_boot_console_base.py | 123 +++++++++++++++------ test/py/u_boot_console_exec_attach.py | 31 ++++-- test/py/u_boot_spawn.py | 36 +++++- 5 files changed, 363 insertions(+), 53 deletions(-)
-- 2.34.1
Is there any news on these patches? I would dearly love to enable my lab and this is what is needed...
As our approaches are radically different I continue to be looking for feedback from a third person to set things up and say what does / doesn't work for them and their lab. I assume you're just pulling this in locally for use at least.
No, I cannot pull this in locally. I would like to set up gitlab so I (and others) can push things to it for running on the lab. Without my patches the lab only works on a few boards.
Could you perhaps try them out on your lab, to check they don't break anything?
Your setup requires changes to labgrid itself, yes?
Hi Tom,
On Tue, 22 Oct 2024 at 16:43, Tom Rini trini@konsulko.com wrote:
On Tue, Oct 22, 2024 at 02:15:40PM +0200, Simon Glass wrote:
Hi Tom,
On Mon, 21 Oct 2024 at 20:07, Tom Rini trini@konsulko.com wrote:
On Mon, Oct 21, 2024 at 01:12:49PM +0200, Simon Glass wrote:
Hi Tom,
On Thu, 10 Oct 2024 at 03:51, Simon Glass sjg@chromium.org wrote:
Labgrid provides access to a hardware lab in an automated way. It is possible to boot U-Boot on boards in the lab without physically touching them. It relies on relays, USB UARTs and SD muxes, among other things.
By way of background, about 4 years ago I wrong a thing called Labman[1] which allowed my lab of about 30 devices to be operated remotely, using tbot for the console and build integration. While it worked OK and I used it for many bisects, I didn't take it any further.
It turns out that there was already an existing program, called Labgrid, which I did not know about at time (thank you Tom for telling me). It is more rounded than Labman and has a number of advantages:
- does not need udev rules, mostly
- has several existing users who rely on it
- supports multiple machines exporting their devices
It lacks a 'lab check' feature and a few other things, but these can be remedied.
On and off over the past several weeks I have been experimenting with Labgrid. I have managed to create an initial U-Boot integration (this series) by adding various features to Labgrid[2] and the U-Boot test hooks.
I hope that this might inspire others to set up boards and run tests automatically, rather than relying on infrequent, manual test. Perhaps it may even be possible to have a number of labs available.
Included in the integration are a number of simple scripts which make it easy to connect to boards and run tests:
ub-int <target> Build and boot on a target, starting an interactive session
ub-cli <target> Build and boot on a target, ensure U-Boot starts and provide an interactive session from there
ub-smoke <target> Smoke test U-Boot to check that it boots to a prompt on a target
ub-bisect <target> Bisect a git tree to locate a failure on a particular target
ub-pyt <target> <testspec> Run U-Boot pytests on a target
Some of these help to provide the same tbot[4] workflow which I have relied on for several years, albeit much simpler versions.
The goal here is to create some sort of script which can collect patches from the mailing list, apply them and test them on a selection of boards. I suspect that script already exists, so please let me know what you suggest.
I hope you find this interesting and take a look!
[1] https://github.com/sjg20/u-boot/tree/lab6a [2] https://github.com/labgrid-project/labgrid/pull/1411 [3] https://github.com/sjg20/uboot-test-hooks/tree/labgrid [4] https://tbot.tools/index.html
Changes in v7:
- Rebase on testb series
- Expand commit message
- Rename option to --use-running-system
- Rebase on broken-out testb series
Changes in v6:
- Rebase without an earlier patch
- Avoid doing the special shutdown unless USE_LABGRID is enabled
- Drop patch 'test: Pass stderr to stdout'
Changes in v5:
- Add a few more comments
- Comment out the debugging, which might be useful later
- Add a patch to support testing with two board-builds
Changes in v3:
- Split out most patches into two new series and update cover letter
Changes in v2:
- Only disable echo if a terminal is detected
- Avoid running a docker image for skipped lab tests
Simon Glass (11): test: Allow signaling that U-Boot is ready test: Release board after tests complete test: Allow connecting to a running board test: Introduce the concept of a role test: Introduce lab mode test: Improve handling of sending commands test: Avoid double echo when starting up test: Try to shut down the lab console gracefully test: Add a section for closing the connection test: Support testing with two board-builds CI: Allow running tests on sjg lab
.gitlab-ci.yml | 153 ++++++++++++++++++++++++++ test/py/conftest.py | 73 ++++++++++-- test/py/u_boot_console_base.py | 123 +++++++++++++++------ test/py/u_boot_console_exec_attach.py | 31 ++++-- test/py/u_boot_spawn.py | 36 +++++- 5 files changed, 363 insertions(+), 53 deletions(-)
-- 2.34.1
Is there any news on these patches? I would dearly love to enable my lab and this is what is needed...
As our approaches are radically different I continue to be looking for feedback from a third person to set things up and say what does / doesn't work for them and their lab. I assume you're just pulling this in locally for use at least.
No, I cannot pull this in locally. I would like to set up gitlab so I (and others) can push things to it for running on the lab. Without my patches the lab only works on a few boards.
Could you perhaps try them out on your lab, to check they don't break anything?
Your setup requires changes to labgrid itself, yes?
Yes, but I have that installed on the Gitlab runner.
As to getting them upstream, it is very slow. I would gain new motivation to push harder if I could actually use it!
Regards, Simon
participants (2)
-
Simon Glass -
Tom Rini