[u-boot-test-hooks PATCH v4 0/3] Add support for Labgrid
This adds hooks for use with Labgrid. The only existing hook that really does anything now is the 'console' one. There is also a new 'getrole' hook which looks up a role to find various things about it, such as the U-Boot board name.
Changes in v4: - Use an environment variable to select the Labgrid hooks - Add new patch to add .none scripts for power off/on - Support pytest fully with dual-build boards like Beagleplay
Changes in v3: - Fall back to conf.${board_type}_${board_ident} if conf.all missing - Update scripts for latest version of Labgrid integration - Add poweroff.none and poweron.none - Provide -n flag when querying board info - Target the grpc version of Labgrid which is now in -master - Update README to cover the changes
Changes in v2: - Make use of the common script (only) to set bin_dir - Make use of the common script (only) to set bin_dir
Simon Glass (3): Create a common file for test scripts Add .none scripts for power off/on Provide some basic scripts for Labgrid integration
README.md | 50 ++++++++++++++++++++++++++++++++++++ bin/console.labgrid | 42 ++++++++++++++++++++++++++++++ bin/ellesmere/common-labgrid | 46 +++++++++++++++++++++++++++++++++ bin/ellesmere/conf.all | 24 +++++++++++++++++ bin/getrole.labgrid | 25 ++++++++++++++++++ bin/poweroff.none | 22 ++++++++++++++++ bin/poweron.none | 22 ++++++++++++++++ bin/release.labgrid | 22 ++++++++++++++++ bin/release.none | 22 ++++++++++++++++ bin/u-boot-test-common | 36 ++++++++++++++++++++++++++ bin/u-boot-test-console | 8 +----- bin/u-boot-test-flash | 8 +----- bin/u-boot-test-getrole | 38 +++++++++++++++++++++++++++ bin/u-boot-test-power-off | 8 +----- bin/u-boot-test-power-on | 8 +----- bin/u-boot-test-release | 26 +++++++++++++++++++ bin/u-boot-test-reset | 8 +----- 17 files changed, 380 insertions(+), 35 deletions(-) create mode 100644 bin/console.labgrid create mode 100755 bin/ellesmere/common-labgrid create mode 100644 bin/ellesmere/conf.all create mode 100644 bin/getrole.labgrid create mode 100755 bin/poweroff.none create mode 100755 bin/poweron.none create mode 100644 bin/release.labgrid create mode 100644 bin/release.none create mode 100755 bin/u-boot-test-common create mode 100755 bin/u-boot-test-getrole create mode 100755 bin/u-boot-test-release
The top part of each of the u-boot-test-* files is common. Put it in a common script file to avoid duplication and to allow it to be replaced for the Labgrid integration.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v4: - Use an environment variable to select the Labgrid hooks
Changes in v3: - Fall back to conf.${board_type}_${board_ident} if conf.all missing
Changes in v2: - Make use of the common script (only) to set bin_dir
bin/u-boot-test-common | 36 ++++++++++++++++++++++++++++++++++++ bin/u-boot-test-console | 8 +------- bin/u-boot-test-flash | 8 +------- bin/u-boot-test-power-off | 8 +------- bin/u-boot-test-power-on | 8 +------- bin/u-boot-test-reset | 8 +------- 6 files changed, 41 insertions(+), 35 deletions(-) create mode 100755 bin/u-boot-test-common
diff --git a/bin/u-boot-test-common b/bin/u-boot-test-common new file mode 100755 index 0000000..5adffd4 --- /dev/null +++ b/bin/u-boot-test-common @@ -0,0 +1,36 @@ +#!/bin/bash + +# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +set -e + +bin_dir="`dirname $0`" +board_type="$1" +board_ident="$2" +hostname="`hostname`" + +# Use either the Labgrid integration or the traditional hooks +if [ -n "${USE_LABGRID}" ]; then + . "${bin_dir}/${hostname}/conf.all" +else + . "${bin_dir}/${hostname}/conf.${board_type}_${board_ident}" +fi diff --git a/bin/u-boot-test-console b/bin/u-boot-test-console index 0b6b4ac..8b7377a 100755 --- a/bin/u-boot-test-console +++ b/bin/u-boot-test-console @@ -20,12 +20,6 @@ # FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER # DEALINGS IN THE SOFTWARE.
-set -e - -bin_dir="`dirname $0`" -board_type="$1" -board_ident="$2" -hostname="`hostname`" -. "${bin_dir}/${hostname}/conf.${board_type}_${board_ident}" +. "$(dirname $0)/u-boot-test-common"
. "${bin_dir}/console.${console_impl:-picocom}" diff --git a/bin/u-boot-test-flash b/bin/u-boot-test-flash index 8dcf198..bd89a50 100755 --- a/bin/u-boot-test-flash +++ b/bin/u-boot-test-flash @@ -20,12 +20,6 @@ # FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER # DEALINGS IN THE SOFTWARE.
-set -e - -bin_dir="`dirname $0`" -board_type="$1" -board_ident="$2" -hostname="`hostname`" -. "${bin_dir}/${hostname}/conf.${board_type}_${board_ident}" +. "$(dirname $0)/u-boot-test-common"
. "${bin_dir}/flash.${flash_impl}" diff --git a/bin/u-boot-test-power-off b/bin/u-boot-test-power-off index b59436a..83b69c0 100755 --- a/bin/u-boot-test-power-off +++ b/bin/u-boot-test-power-off @@ -20,12 +20,6 @@ # FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER # DEALINGS IN THE SOFTWARE.
-set -e - -bin_dir="`dirname $0`" -board_type="$1" -board_ident="$2" -hostname="`hostname`" -. "${bin_dir}/${hostname}/conf.${board_type}_${board_ident}" +. "$(dirname $0)/u-boot-test-common"
. "${bin_dir}/poweroff.${power_impl}" diff --git a/bin/u-boot-test-power-on b/bin/u-boot-test-power-on index ca87477..5a41c3e 100755 --- a/bin/u-boot-test-power-on +++ b/bin/u-boot-test-power-on @@ -20,12 +20,6 @@ # FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER # DEALINGS IN THE SOFTWARE.
-set -e - -bin_dir="`dirname $0`" -board_type="$1" -board_ident="$2" -hostname="`hostname`" -. "${bin_dir}/${hostname}/conf.${board_type}_${board_ident}" +. "$(dirname $0)/u-boot-test-common"
. "${bin_dir}/poweron.${power_impl}" diff --git a/bin/u-boot-test-reset b/bin/u-boot-test-reset index a160e0c..77e3f40 100755 --- a/bin/u-boot-test-reset +++ b/bin/u-boot-test-reset @@ -20,12 +20,6 @@ # FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER # DEALINGS IN THE SOFTWARE.
-set -e - -bin_dir="`dirname $0`" -board_type="$1" -board_ident="$2" -hostname="`hostname`" -. "${bin_dir}/${hostname}/conf.${board_type}_${board_ident}" +. "$(dirname $0)/u-boot-test-common"
. "${bin_dir}/reset.${reset_impl}"
On Wed, Aug 28, 2024 at 10:45:21AM -0600, Simon Glass wrote:
The top part of each of the u-boot-test-* files is common. Put it in a common script file to avoid duplication and to allow it to be replaced for the Labgrid integration.
Signed-off-by: Simon Glass sjg@chromium.org
Reviewed-by: Tom Rini trini@konsulko.com
These methods don't have a 'none' method right now, but it will be needed for Labgrid. Add these scripts.
Note that these are not used by the existing integration, which requires that boards can be powered off and on.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v4: - Add new patch to add .none scripts for power off/on
bin/poweroff.none | 22 ++++++++++++++++++++++ bin/poweron.none | 22 ++++++++++++++++++++++ 2 files changed, 44 insertions(+) create mode 100755 bin/poweroff.none create mode 100755 bin/poweron.none
diff --git a/bin/poweroff.none b/bin/poweroff.none new file mode 100755 index 0000000..cd9733e --- /dev/null +++ b/bin/poweroff.none @@ -0,0 +1,22 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +# No power-control needed diff --git a/bin/poweron.none b/bin/poweron.none new file mode 100755 index 0000000..cd9733e --- /dev/null +++ b/bin/poweron.none @@ -0,0 +1,22 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +# No power-control needed
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
Signed-off-by: Simon Glass sjg@chromium.org ---
Changes in v4: - Support pytest fully with dual-build boards like Beagleplay
Changes in v3: - Update scripts for latest version of Labgrid integration - Add poweroff.none and poweron.none - Provide -n flag when querying board info - Target the grpc version of Labgrid which is now in -master - Update README to cover the changes
Changes in v2: - Make use of the common script (only) to set bin_dir
README.md | 50 ++++++++++++++++++++++++++++++++++++ bin/console.labgrid | 42 ++++++++++++++++++++++++++++++ bin/ellesmere/common-labgrid | 46 +++++++++++++++++++++++++++++++++ bin/ellesmere/conf.all | 24 +++++++++++++++++ bin/getrole.labgrid | 25 ++++++++++++++++++ bin/release.labgrid | 22 ++++++++++++++++ bin/release.none | 22 ++++++++++++++++ bin/u-boot-test-getrole | 38 +++++++++++++++++++++++++++ bin/u-boot-test-release | 26 +++++++++++++++++++ 9 files changed, 295 insertions(+) create mode 100644 bin/console.labgrid create mode 100755 bin/ellesmere/common-labgrid create mode 100644 bin/ellesmere/conf.all create mode 100644 bin/getrole.labgrid create mode 100644 bin/release.labgrid create mode 100644 bin/release.none create mode 100755 bin/u-boot-test-getrole create mode 100755 bin/u-boot-test-release
diff --git a/README.md b/README.md index 290e4d0..18a9a5b 100644 --- a/README.md +++ b/README.md @@ -199,6 +199,56 @@ scripts must be replicated once per board instance, or their actions somehow serialized, since they copy files into their own directories when executing, and hence parallel execution would cause incorrect operation.
+## Labgrid Integration + +Labgrid is a python library for embedded-board-control. It includes a client +program which is used to integrate with the U-Boot pytests. + +Since Labgrid has all the information necessary to build and boot on a lab, +there is no per-board configuration required. The various flash.xxx and +recovery.xxx scripts are not used. To set it up: + +- In your bin/$hostname directory, create an executable file `common-labgrid` + and set your crossbar and environment information, for example: + + # Hostname and port for the gRPC coordinator + export LG_COORDINATOR=kea:20408 + + # Environment file for the lab + export LG_ENV="/path/to/env.cfg" + + # Location of the U-Boot test hooks + export UB_TEST_HOOKS=/path/to/u-boot-test-hooks + + # Make sure only one buildman can run at a time, since it uses all CPUs + export BUILDMAN_PROCESS_LIMIT=1 + + # Use the internal console since microcom can miss serial input at boot + export LG_CONSOLE="internal" + + # Tell u-boot-test-hooks to use the Labgrid integration + export USE_LABGRID=1 + + flash_impl=none + reset_impl=none + console_impl=labgrid + release_impl=labgrid + getrole_impl=labgrid + power_impl=none + +The last 6 lines tell the hooks to use Labgrid for console and board release +as well as a new 'getrole' hook which is only used by Labgrid. The flash, reset +and power features of boards are all handled by entirely by Labgrid. + +Then create another executable file (in the same directory) called 'conf.all', +containing:: + + .. code-block:: bash + + . "${bin_dir}/${hostname}/common-labgrid" + +That should be all that is needed. + ## Dependencies
The example scripts depend on various external tools, the installation location diff --git a/bin/console.labgrid b/bin/console.labgrid new file mode 100644 index 0000000..85e7da5 --- /dev/null +++ b/bin/console.labgrid @@ -0,0 +1,42 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +# The variables here can come from one of two places: +# +# 1. When using the ub-xxx scripts (e.g. ub-int) they come from those scripts, +# set by the get_args.sh script +# +# 2. When running from gitlab, the variables are all empty and so take the +# default values below, except for ${strategy} which is set in the gitlab +# script + +# On input: +# strategy: Strategy arguments to use, e.g. "-s start". Normally this is +# "-s uboot -e off" but it can be "-s start -e off" or even empty +# verbose: Verbose argument to use, e.g. "-v" +# +# These variables are set by .gitlab-ci.yml or by + +exec labgrid-client -V do-bootstrap ${bootstrap:-1} -V do-build ${build:-1} \ + -V do-send ${send:-0} -V do-clean ${clean:-0} \ + -V process-limit ${BUILDMAN_PROCESS_LIMIT:-0} \ + -r "${U_BOOT_BOARD_IDENTITY}" ${strategy} ${verbose} -a console \ + ${console_log} diff --git a/bin/ellesmere/common-labgrid b/bin/ellesmere/common-labgrid new file mode 100755 index 0000000..8bfb036 --- /dev/null +++ b/bin/ellesmere/common-labgrid @@ -0,0 +1,46 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +# Hostname and port for the gRPC coordinator +export LG_COORDINATOR=kea:20408 + +# Environment file for the lab +export LG_ENV=/vid/software/devel/ubtest/lab/cfg/env.cfg + +# Make sure only one buildman can run at a time, since it uses all CPUs +export BUILDMAN_PROCESS_LIMIT=1 + +# Location of the U-Boot test hooks +export UB_TEST_HOOKS=/vid/software/devel/ubtest/u-boot-test-hooks + +# Use the internal console since microcom can miss serial input at boot +export LG_CONSOLE=internal + +# Tell u-boot-test-hooks to use the Labgrid integration +export USE_LABGRID=1 + + +flash_impl=none +reset_impl=none +console_impl=labgrid +release_impl=labgrid +getrole_impl=labgrid +power_impl=none diff --git a/bin/ellesmere/conf.all b/bin/ellesmere/conf.all new file mode 100644 index 0000000..6cbe561 --- /dev/null +++ b/bin/ellesmere/conf.all @@ -0,0 +1,24 @@ +#!/bin/bash + +# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +. "${bin_dir}/${hostname}/common-labgrid" diff --git a/bin/getrole.labgrid b/bin/getrole.labgrid new file mode 100644 index 0000000..52fc3ce --- /dev/null +++ b/bin/getrole.labgrid @@ -0,0 +1,25 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +[ -n "${do_configure}" ] && config_arg=",config_file" + +exec labgrid-client -r "${target}" -a query -n \ + UBootProviderDriver:board,board_extra,source_dir,build_dir,build_dir_extra${config_arg} diff --git a/bin/release.labgrid b/bin/release.labgrid new file mode 100644 index 0000000..54e7895 --- /dev/null +++ b/bin/release.labgrid @@ -0,0 +1,22 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +exec labgrid-client -r "${U_BOOT_BOARD_IDENTITY}" release -a diff --git a/bin/release.none b/bin/release.none new file mode 100644 index 0000000..284d1b1 --- /dev/null +++ b/bin/release.none @@ -0,0 +1,22 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +# No releasing needed diff --git a/bin/u-boot-test-getrole b/bin/u-boot-test-getrole new file mode 100755 index 0000000..ac8e02a --- /dev/null +++ b/bin/u-boot-test-getrole @@ -0,0 +1,38 @@ +#!/bin/bash + +# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +# Prints the following pieces of information, each on a separate line +# +# - U-Boot board name +# - source path +# - build dir +# - filename of .config file +# +# For example + +. "$(dirname $0)/u-boot-test-common" + +target="$1" +[ -n "$2" ] && do_configure=1 + +. "${bin_dir}/getrole.${getrole_impl:-none}" diff --git a/bin/u-boot-test-release b/bin/u-boot-test-release new file mode 100755 index 0000000..5a3a849 --- /dev/null +++ b/bin/u-boot-test-release @@ -0,0 +1,26 @@ +#!/bin/bash + +# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +. "$(dirname $0)/u-boot-test-common" + +. "${bin_dir}/release.${release_impl:-none}"
Hi Simon,
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
What's ellesmere?
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v4:
- Support pytest fully with dual-build boards like Beagleplay
Changes in v3:
- Update scripts for latest version of Labgrid integration
- Add poweroff.none and poweron.none
- Provide -n flag when querying board info
- Target the grpc version of Labgrid which is now in -master
- Update README to cover the changes
Changes in v2:
- Make use of the common script (only) to set bin_dir
README.md | 50 ++++++++++++++++++++++++++++++++++++
Maybe that should be in a separate labsgrid readme?
bin/console.labgrid | 42 ++++++++++++++++++++++++++++++ bin/ellesmere/common-labgrid | 46 +++++++++++++++++++++++++++++++++ bin/ellesmere/conf.all | 24 +++++++++++++++++ bin/getrole.labgrid | 25 ++++++++++++++++++ bin/release.labgrid | 22 ++++++++++++++++ bin/release.none | 22 ++++++++++++++++ bin/u-boot-test-getrole | 38 +++++++++++++++++++++++++++ bin/u-boot-test-release | 26 +++++++++++++++++++ 9 files changed, 295 insertions(+) create mode 100644 bin/console.labgrid create mode 100755 bin/ellesmere/common-labgrid create mode 100644 bin/ellesmere/conf.all create mode 100644 bin/getrole.labgrid create mode 100644 bin/release.labgrid create mode 100644 bin/release.none create mode 100755 bin/u-boot-test-getrole create mode 100755 bin/u-boot-test-release
diff --git a/README.md b/README.md index 290e4d0..18a9a5b 100644 --- a/README.md +++ b/README.md @@ -199,6 +199,56 @@ scripts must be replicated once per board instance, or their actions somehow serialized, since they copy files into their own directories when executing, and hence parallel execution would cause incorrect operation.
+## Labgrid Integration
+Labgrid is a python library for embedded-board-control. It includes a client +program which is used to integrate with the U-Boot pytests.
+Since Labgrid has all the information necessary to build and boot on a lab, +there is no per-board configuration required. The various flash.xxx and +recovery.xxx scripts are not used. To set it up:
+- In your bin/$hostname directory, create an executable file `common-labgrid`
- and set your crossbar and environment information, for example:
# Hostname and port for the gRPC coordinatorexport LG_COORDINATOR=kea:20408# Environment file for the labexport LG_ENV="/path/to/env.cfg"# Location of the U-Boot test hooksexport UB_TEST_HOOKS=/path/to/u-boot-test-hooks# Make sure only one buildman can run at a time, since it uses all CPUsexport BUILDMAN_PROCESS_LIMIT=1# Use the internal console since microcom can miss serial input at bootexport LG_CONSOLE="internal"# Tell u-boot-test-hooks to use the Labgrid integrationexport USE_LABGRID=1flash_impl=nonereset_impl=noneconsole_impl=labgridrelease_impl=labgridgetrole_impl=labgridpower_impl=none+The last 6 lines tell the hooks to use Labgrid for console and board release +as well as a new 'getrole' hook which is only used by Labgrid. The flash, reset +and power features of boards are all handled by entirely by Labgrid.
+Then create another executable file (in the same directory) called 'conf.all', +containing::
- .. code-block:: bash
- . "${bin_dir}/${hostname}/common-labgrid"
+That should be all that is needed.
## Dependencies
The example scripts depend on various external tools, the installation location diff --git a/bin/console.labgrid b/bin/console.labgrid new file mode 100644 index 0000000..85e7da5 --- /dev/null +++ b/bin/console.labgrid @@ -0,0 +1,42 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE.
+# The variables here can come from one of two places: +# +# 1. When using the ub-xxx scripts (e.g. ub-int) they come from those scripts, +# set by the get_args.sh script +# +# 2. When running from gitlab, the variables are all empty and so take the +# default values below, except for ${strategy} which is set in the gitlab +# script
+# On input: +# strategy: Strategy arguments to use, e.g. "-s start". Normally this is +# "-s uboot -e off" but it can be "-s start -e off" or even empty +# verbose: Verbose argument to use, e.g. "-v" +# +# These variables are set by .gitlab-ci.yml or by
+exec labgrid-client -V do-bootstrap ${bootstrap:-1} -V do-build ${build:-1} \
- -V do-send ${send:-0} -V do-clean ${clean:-0} \
- -V process-limit ${BUILDMAN_PROCESS_LIMIT:-0} \
- -r "${U_BOOT_BOARD_IDENTITY}" ${strategy} ${verbose} -a console \
- ${console_log}
diff --git a/bin/ellesmere/common-labgrid b/bin/ellesmere/common-labgrid new file mode 100755 index 0000000..8bfb036 --- /dev/null +++ b/bin/ellesmere/common-labgrid @@ -0,0 +1,46 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE.
+# Hostname and port for the gRPC coordinator +export LG_COORDINATOR=kea:20408
+# Environment file for the lab +export LG_ENV=/vid/software/devel/ubtest/lab/cfg/env.cfg
+# Make sure only one buildman can run at a time, since it uses all CPUs +export BUILDMAN_PROCESS_LIMIT=1
+# Location of the U-Boot test hooks +export UB_TEST_HOOKS=/vid/software/devel/ubtest/u-boot-test-hooks
+# Use the internal console since microcom can miss serial input at boot +export LG_CONSOLE=internal
+# Tell u-boot-test-hooks to use the Labgrid integration +export USE_LABGRID=1
+flash_impl=none +reset_impl=none +console_impl=labgrid +release_impl=labgrid +getrole_impl=labgrid +power_impl=none diff --git a/bin/ellesmere/conf.all b/bin/ellesmere/conf.all new file mode 100644 index 0000000..6cbe561 --- /dev/null +++ b/bin/ellesmere/conf.all @@ -0,0 +1,24 @@ +#!/bin/bash
+# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE.
+. "${bin_dir}/${hostname}/common-labgrid" diff --git a/bin/getrole.labgrid b/bin/getrole.labgrid new file mode 100644 index 0000000..52fc3ce --- /dev/null +++ b/bin/getrole.labgrid @@ -0,0 +1,25 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE.
+[ -n "${do_configure}" ] && config_arg=",config_file"
+exec labgrid-client -r "${target}" -a query -n \
UBootProviderDriver:board,board_extra,source_dir,build_dir,build_dir_extra${config_arg}diff --git a/bin/release.labgrid b/bin/release.labgrid new file mode 100644 index 0000000..54e7895 --- /dev/null +++ b/bin/release.labgrid @@ -0,0 +1,22 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE.
+exec labgrid-client -r "${U_BOOT_BOARD_IDENTITY}" release -a diff --git a/bin/release.none b/bin/release.none new file mode 100644 index 0000000..284d1b1 --- /dev/null +++ b/bin/release.none @@ -0,0 +1,22 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE.
+# No releasing needed diff --git a/bin/u-boot-test-getrole b/bin/u-boot-test-getrole new file mode 100755 index 0000000..ac8e02a --- /dev/null +++ b/bin/u-boot-test-getrole @@ -0,0 +1,38 @@ +#!/bin/bash
+# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE.
+# Prints the following pieces of information, each on a separate line +# +# - U-Boot board name +# - source path +# - build dir +# - filename of .config file +# +# For example
+. "$(dirname $0)/u-boot-test-common"
+target="$1" +[ -n "$2" ] && do_configure=1
+. "${bin_dir}/getrole.${getrole_impl:-none}" diff --git a/bin/u-boot-test-release b/bin/u-boot-test-release new file mode 100755 index 0000000..5a3a849 --- /dev/null +++ b/bin/u-boot-test-release @@ -0,0 +1,26 @@ +#!/bin/bash
+# Copyright 2024 Google LLC +# Written by Simon Glass sjg@chromium.org +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE.
+. "$(dirname $0)/u-boot-test-common"
+. "${bin_dir}/release.${release_impl:-none}"
2.34.1
On Wed, Aug 28, 2024 at 07:14:01PM +0100, Peter Robinson wrote:
Hi Simon,
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
What's ellesmere?
That should probably read something like "Provide ellesmere as an example lab configuration".
[snip]
Changes in v2:
- Make use of the common script (only) to set bin_dir
README.md | 50 ++++++++++++++++++++++++++++++++++++
Maybe that should be in a separate labsgrid readme?
A followup / separate series to better document all of this would be good, but TBH I feel like what Simon's added here matches the rest of the document.
Hi Peter,
On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote:
Hi Simon,
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
What's ellesmere?
It is a lake but also the name of a computer.
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v4:
- Support pytest fully with dual-build boards like Beagleplay
Changes in v3:
- Update scripts for latest version of Labgrid integration
- Add poweroff.none and poweron.none
- Provide -n flag when querying board info
- Target the grpc version of Labgrid which is now in -master
- Update README to cover the changes
Changes in v2:
- Make use of the common script (only) to set bin_dir
README.md | 50 ++++++++++++++++++++++++++++++++++++
Maybe that should be in a separate labsgrid readme?
My hope is that Labgrid becomes the normal way of running these tests, so having it in the main place (and perhaps eventually removing the old way) is my goal.
bin/console.labgrid | 42 ++++++++++++++++++++++++++++++ bin/ellesmere/common-labgrid | 46 +++++++++++++++++++++++++++++++++ bin/ellesmere/conf.all | 24 +++++++++++++++++ bin/getrole.labgrid | 25 ++++++++++++++++++ bin/release.labgrid | 22 ++++++++++++++++ bin/release.none | 22 ++++++++++++++++ bin/u-boot-test-getrole | 38 +++++++++++++++++++++++++++ bin/u-boot-test-release | 26 +++++++++++++++++++ 9 files changed, 295 insertions(+) create mode 100644 bin/console.labgrid create mode 100755 bin/ellesmere/common-labgrid create mode 100644 bin/ellesmere/conf.all create mode 100644 bin/getrole.labgrid create mode 100644 bin/release.labgrid create mode 100644 bin/release.none create mode 100755 bin/u-boot-test-getrole create mode 100755 bin/u-boot-test-release
[..]
Regards, Simon
On Wed, 28 Aug 2024 at 22:25, Simon Glass sjg@chromium.org wrote:
Hi Peter,
On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote:
Hi Simon,
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
What's ellesmere?
It is a lake but also the name of a computer.
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v4:
- Support pytest fully with dual-build boards like Beagleplay
Changes in v3:
- Update scripts for latest version of Labgrid integration
- Add poweroff.none and poweron.none
- Provide -n flag when querying board info
- Target the grpc version of Labgrid which is now in -master
- Update README to cover the changes
Changes in v2:
- Make use of the common script (only) to set bin_dir
README.md | 50 ++++++++++++++++++++++++++++++++++++
Maybe that should be in a separate labsgrid readme?
My hope is that Labgrid becomes the normal way of running these tests,
Generally I agree with automated testing platforms, I think <insert you're preferred platform here> makes sense, there's a bunch like Linaro and a bunch in the arm ecosystem use LAVA [1] and there's a bunch more that I'm aware of.
Does it make sense, of course at some point in the future post this being merged, make sense to look at a general way of making it easier to plugin these sort of HW test platforms using this as a basis? I ask mostly because putting a bunch of my devices into some sort of platform can auto test things and of course everyone has an opinion to which is the best one :-P
[1] https://validation.linaro.org/
so having it in the main place (and perhaps eventually removing the old way) is my goal.
bin/console.labgrid | 42 ++++++++++++++++++++++++++++++ bin/ellesmere/common-labgrid | 46 +++++++++++++++++++++++++++++++++ bin/ellesmere/conf.all | 24 +++++++++++++++++ bin/getrole.labgrid | 25 ++++++++++++++++++ bin/release.labgrid | 22 ++++++++++++++++ bin/release.none | 22 ++++++++++++++++ bin/u-boot-test-getrole | 38 +++++++++++++++++++++++++++ bin/u-boot-test-release | 26 +++++++++++++++++++ 9 files changed, 295 insertions(+) create mode 100644 bin/console.labgrid create mode 100755 bin/ellesmere/common-labgrid create mode 100644 bin/ellesmere/conf.all create mode 100644 bin/getrole.labgrid create mode 100644 bin/release.labgrid create mode 100644 bin/release.none create mode 100755 bin/u-boot-test-getrole create mode 100755 bin/u-boot-test-release
[..]
Regards, Simon
Hi Peter,
On Thu, 29 Aug 2024 at 04:43, Peter Robinson pbrobinson@gmail.com wrote:
On Wed, 28 Aug 2024 at 22:25, Simon Glass sjg@chromium.org wrote:
Hi Peter,
On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote:
Hi Simon,
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
What's ellesmere?
It is a lake but also the name of a computer.
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v4:
- Support pytest fully with dual-build boards like Beagleplay
Changes in v3:
- Update scripts for latest version of Labgrid integration
- Add poweroff.none and poweron.none
- Provide -n flag when querying board info
- Target the grpc version of Labgrid which is now in -master
- Update README to cover the changes
Changes in v2:
- Make use of the common script (only) to set bin_dir
README.md | 50 ++++++++++++++++++++++++++++++++++++
Maybe that should be in a separate labsgrid readme?
My hope is that Labgrid becomes the normal way of running these tests,
Generally I agree with automated testing platforms, I think <insert you're preferred platform here> makes sense, there's a bunch like Linaro and a bunch in the arm ecosystem use LAVA [1] and there's a bunch more that I'm aware of.
I am somewhat familiar with LAVA and I believe it can be used to test U-Boot, although I need to learn how. Looking at a test run [2] for beaglebone black I see that it is using a recent kernel but the U-Boot seems to be older.
Does it make sense, of course at some point in the future post this being merged, make sense to look at a general way of making it easier to plugin these sort of HW test platforms using this as a basis? I ask mostly because putting a bunch of my devices into some sort of platform can auto test things and of course everyone has an opinion to which is the best one :-P
Yes. I had heard from Tom that Labgrid is the new hotness for now. Having dug into it I believe it is a good solution, although it can certainly be improved to handle scale better.
Anyway, IMO the current test hooks are not a great solution, just because the configuration is spread all over the place and it relies on lots of little shell scripts. So I believe that the Labgrid integration is a closer to where we want to be with others that come along.
I would love for you to add a board into Labgrid and see how you go.
[1] https://validation.linaro.org/
so having it in the main place (and perhaps eventually removing the old way) is my goal.
bin/console.labgrid | 42 ++++++++++++++++++++++++++++++ bin/ellesmere/common-labgrid | 46 +++++++++++++++++++++++++++++++++ bin/ellesmere/conf.all | 24 +++++++++++++++++ bin/getrole.labgrid | 25 ++++++++++++++++++ bin/release.labgrid | 22 ++++++++++++++++ bin/release.none | 22 ++++++++++++++++ bin/u-boot-test-getrole | 38 +++++++++++++++++++++++++++ bin/u-boot-test-release | 26 +++++++++++++++++++ 9 files changed, 295 insertions(+) create mode 100644 bin/console.labgrid create mode 100755 bin/ellesmere/common-labgrid create mode 100644 bin/ellesmere/conf.all create mode 100644 bin/getrole.labgrid create mode 100644 bin/release.labgrid create mode 100644 bin/release.none create mode 100755 bin/u-boot-test-getrole create mode 100755 bin/u-boot-test-release
[..]
Regards, Simon
Regards, Simon
[2] https://validation.linaro.org/scheduler/job/4089871#results_482796468
On 29/08/2024 14:17, Simon Glass wrote:
Hi Peter,
On Thu, 29 Aug 2024 at 04:43, Peter Robinson pbrobinson@gmail.com wrote:
On Wed, 28 Aug 2024 at 22:25, Simon Glass sjg@chromium.org wrote:
Hi Peter,
On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote:
Hi Simon,
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
What's ellesmere?
It is a lake but also the name of a computer.
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v4:
- Support pytest fully with dual-build boards like Beagleplay
Changes in v3:
- Update scripts for latest version of Labgrid integration
- Add poweroff.none and poweron.none
- Provide -n flag when querying board info
- Target the grpc version of Labgrid which is now in -master
- Update README to cover the changes
Changes in v2:
Make use of the common script (only) to set bin_dir
README.md | 50 ++++++++++++++++++++++++++++++++++++
Maybe that should be in a separate labsgrid readme?
My hope is that Labgrid becomes the normal way of running these tests,
Generally I agree with automated testing platforms, I think <insert you're preferred platform here> makes sense, there's a bunch like Linaro and a bunch in the arm ecosystem use LAVA [1] and there's a bunch more that I'm aware of.
I am somewhat familiar with LAVA and I believe it can be used to test U-Boot, although I need to learn how. Looking at a test run [2] for beaglebone black I see that it is using a recent kernel but the U-Boot seems to be older.
Does it make sense, of course at some point in the future post this being merged, make sense to look at a general way of making it easier to plugin these sort of HW test platforms using this as a basis? I ask mostly because putting a bunch of my devices into some sort of platform can auto test things and of course everyone has an opinion to which is the best one :-P
Yes. I had heard from Tom that Labgrid is the new hotness for now. Having dug into it I believe it is a good solution, although it can certainly be improved to handle scale better.
Anyway, IMO the current test hooks are not a great solution, just because the configuration is spread all over the place and it relies on lots of little shell scripts. So I believe that the Labgrid integration is a closer to where we want to be with others that come along.
I'd say all those scripts are actually here to ease integration with any system, booting U-Boot and Linux are two different beasts. Most of the time you need special jumpers enabled to setup vendor-specific bootrom mode to reflash or ram-boot U-Boot. So forcing us people to switch to Labgrid because it matches your boards behavior is a little adventurous, and while we can add some enhancement to the actual pytest and hooks, I'm against saying Labgrid should be the golden CI system.
Neil
I would love for you to add a board into Labgrid and see how you go.
[1] https://validation.linaro.org/
so having it in the main place (and perhaps eventually removing the old way) is my goal.
bin/console.labgrid | 42 ++++++++++++++++++++++++++++++ bin/ellesmere/common-labgrid | 46 +++++++++++++++++++++++++++++++++ bin/ellesmere/conf.all | 24 +++++++++++++++++ bin/getrole.labgrid | 25 ++++++++++++++++++ bin/release.labgrid | 22 ++++++++++++++++ bin/release.none | 22 ++++++++++++++++ bin/u-boot-test-getrole | 38 +++++++++++++++++++++++++++ bin/u-boot-test-release | 26 +++++++++++++++++++ 9 files changed, 295 insertions(+) create mode 100644 bin/console.labgrid create mode 100755 bin/ellesmere/common-labgrid create mode 100644 bin/ellesmere/conf.all create mode 100644 bin/getrole.labgrid create mode 100644 bin/release.labgrid create mode 100644 bin/release.none create mode 100755 bin/u-boot-test-getrole create mode 100755 bin/u-boot-test-release
[..]
Regards, Simon
Regards, Simon
[2] https://validation.linaro.org/scheduler/job/4089871#results_482796468
Hi Neil,
On Thu, 29 Aug 2024 at 08:44, neil.armstrong@linaro.org wrote:
On 29/08/2024 14:17, Simon Glass wrote:
Hi Peter,
On Thu, 29 Aug 2024 at 04:43, Peter Robinson pbrobinson@gmail.com wrote:
On Wed, 28 Aug 2024 at 22:25, Simon Glass sjg@chromium.org wrote:
Hi Peter,
On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote:
Hi Simon,
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
What's ellesmere?
It is a lake but also the name of a computer.
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v4:
- Support pytest fully with dual-build boards like Beagleplay
Changes in v3:
- Update scripts for latest version of Labgrid integration
- Add poweroff.none and poweron.none
- Provide -n flag when querying board info
- Target the grpc version of Labgrid which is now in -master
- Update README to cover the changes
Changes in v2:
Make use of the common script (only) to set bin_dir
README.md | 50 ++++++++++++++++++++++++++++++++++++
Maybe that should be in a separate labsgrid readme?
My hope is that Labgrid becomes the normal way of running these tests,
Generally I agree with automated testing platforms, I think <insert you're preferred platform here> makes sense, there's a bunch like Linaro and a bunch in the arm ecosystem use LAVA [1] and there's a bunch more that I'm aware of.
I am somewhat familiar with LAVA and I believe it can be used to test U-Boot, although I need to learn how. Looking at a test run [2] for beaglebone black I see that it is using a recent kernel but the U-Boot seems to be older.
Does it make sense, of course at some point in the future post this being merged, make sense to look at a general way of making it easier to plugin these sort of HW test platforms using this as a basis? I ask mostly because putting a bunch of my devices into some sort of platform can auto test things and of course everyone has an opinion to which is the best one :-P
Yes. I had heard from Tom that Labgrid is the new hotness for now. Having dug into it I believe it is a good solution, although it can certainly be improved to handle scale better.
Anyway, IMO the current test hooks are not a great solution, just because the configuration is spread all over the place and it relies on lots of little shell scripts. So I believe that the Labgrid integration is a closer to where we want to be with others that come along.
I'd say all those scripts are actually here to ease integration with any system, booting U-Boot and Linux are two different beasts.
That's fine, go ahead and use the scripts. My point is that Labgrid doesn't need them and in fact it makes everything pretty painful if we try to use all of them.
Most of the time you need special jumpers enabled to setup vendor-specific bootrom mode to reflash or ram-boot U-Boot.
I deal with this by soldering onto the recovery switch. For example Beagleplay needs it, otherwise it boots from internal eMMC or something.
Perhaps Linaro could create a standard here, where boards have a 'recovery' line which labs can twiddle?
So forcing us people to switch to Labgrid because it matches your boards behavior is a little adventurous, and while we can add some enhancement to the actual pytest and hooks, I'm against saying Labgrid should be the golden CI system.
Who is forcing anyone to use Labgrid?
I invented my own system (Labman [3]) several years ago. Let me know if you are interested and I can share some slides. I was thinking of trying to tidy it up, but Tom suggested that Labgrid might be worth a look. I had never even heard of it before that.
I don't mind about choosing golden CI systems. The more the merrier. I want to have my boards connected to gitlab so things like bloblist changes don't break my boards. At this point I have convinced myself that Labgrid (as amended) is a good solution for this.
Regards, Simon
Neil
I would love for you to add a board into Labgrid and see how you go.
[1] https://validation.linaro.org/
so having it in the main place (and perhaps eventually removing the old way) is my goal.
bin/console.labgrid | 42 ++++++++++++++++++++++++++++++ bin/ellesmere/common-labgrid | 46 +++++++++++++++++++++++++++++++++ bin/ellesmere/conf.all | 24 +++++++++++++++++ bin/getrole.labgrid | 25 ++++++++++++++++++ bin/release.labgrid | 22 ++++++++++++++++ bin/release.none | 22 ++++++++++++++++ bin/u-boot-test-getrole | 38 +++++++++++++++++++++++++++ bin/u-boot-test-release | 26 +++++++++++++++++++ 9 files changed, 295 insertions(+) create mode 100644 bin/console.labgrid create mode 100755 bin/ellesmere/common-labgrid create mode 100644 bin/ellesmere/conf.all create mode 100644 bin/getrole.labgrid create mode 100644 bin/release.labgrid create mode 100644 bin/release.none create mode 100755 bin/u-boot-test-getrole create mode 100755 bin/u-boot-test-release
[..]
Regards, Simon
Regards, Simon
[2] https://validation.linaro.org/scheduler/job/4089871#results_482796468
On Thu, Aug 29, 2024 at 09:02:38AM -0600, Simon Glass wrote:
Hi Neil,
On Thu, 29 Aug 2024 at 08:44, neil.armstrong@linaro.org wrote:
On 29/08/2024 14:17, Simon Glass wrote:
Hi Peter,
On Thu, 29 Aug 2024 at 04:43, Peter Robinson pbrobinson@gmail.com wrote:
On Wed, 28 Aug 2024 at 22:25, Simon Glass sjg@chromium.org wrote:
Hi Peter,
On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote:
Hi Simon,
> With Labgrid we don't need to specify the various methods, except for > the console, which simply calls labgrid-client. > > This allows supporting any boards in your lab, without adding per-board > configuration to these hooks. > > Provide ellesmere files as an example.
What's ellesmere?
It is a lake but also the name of a computer.
> Signed-off-by: Simon Glass sjg@chromium.org > --- > > Changes in v4: > - Support pytest fully with dual-build boards like Beagleplay > > Changes in v3: > - Update scripts for latest version of Labgrid integration > - Add poweroff.none and poweron.none > - Provide -n flag when querying board info > - Target the grpc version of Labgrid which is now in -master > - Update README to cover the changes > > Changes in v2: > - Make use of the common script (only) to set bin_dir > > README.md | 50 ++++++++++++++++++++++++++++++++++++
Maybe that should be in a separate labsgrid readme?
My hope is that Labgrid becomes the normal way of running these tests,
Generally I agree with automated testing platforms, I think <insert you're preferred platform here> makes sense, there's a bunch like Linaro and a bunch in the arm ecosystem use LAVA [1] and there's a bunch more that I'm aware of.
I am somewhat familiar with LAVA and I believe it can be used to test U-Boot, although I need to learn how. Looking at a test run [2] for beaglebone black I see that it is using a recent kernel but the U-Boot seems to be older.
Does it make sense, of course at some point in the future post this being merged, make sense to look at a general way of making it easier to plugin these sort of HW test platforms using this as a basis? I ask mostly because putting a bunch of my devices into some sort of platform can auto test things and of course everyone has an opinion to which is the best one :-P
Yes. I had heard from Tom that Labgrid is the new hotness for now. Having dug into it I believe it is a good solution, although it can certainly be improved to handle scale better.
Anyway, IMO the current test hooks are not a great solution, just because the configuration is spread all over the place and it relies on lots of little shell scripts. So I believe that the Labgrid integration is a closer to where we want to be with others that come along.
I'd say all those scripts are actually here to ease integration with any system, booting U-Boot and Linux are two different beasts.
That's fine, go ahead and use the scripts. My point is that Labgrid doesn't need them and in fact it makes everything pretty painful if we try to use all of them.
I guess I really need to clean-up and post my former co-workers scripts as I strongly disagree with that statement. The only "painful" part is the shared pain, regardless of framework, for "put blob $X at location $Y", which I believe you abstract out in to python?
Hi Tom,
On Thu, 29 Aug 2024 at 09:20, Tom Rini trini@konsulko.com wrote:
On Thu, Aug 29, 2024 at 09:02:38AM -0600, Simon Glass wrote:
Hi Neil,
On Thu, 29 Aug 2024 at 08:44, neil.armstrong@linaro.org wrote:
On 29/08/2024 14:17, Simon Glass wrote:
Hi Peter,
On Thu, 29 Aug 2024 at 04:43, Peter Robinson pbrobinson@gmail.com wrote:
On Wed, 28 Aug 2024 at 22:25, Simon Glass sjg@chromium.org wrote:
Hi Peter,
On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote: > > Hi Simon, > >> With Labgrid we don't need to specify the various methods, except for >> the console, which simply calls labgrid-client. >> >> This allows supporting any boards in your lab, without adding per-board >> configuration to these hooks. >> >> Provide ellesmere files as an example. > > What's ellesmere?
It is a lake but also the name of a computer.
> >> Signed-off-by: Simon Glass sjg@chromium.org >> --- >> >> Changes in v4: >> - Support pytest fully with dual-build boards like Beagleplay >> >> Changes in v3: >> - Update scripts for latest version of Labgrid integration >> - Add poweroff.none and poweron.none >> - Provide -n flag when querying board info >> - Target the grpc version of Labgrid which is now in -master >> - Update README to cover the changes >> >> Changes in v2: >> - Make use of the common script (only) to set bin_dir >> >> README.md | 50 ++++++++++++++++++++++++++++++++++++ > > Maybe that should be in a separate labsgrid readme?
My hope is that Labgrid becomes the normal way of running these tests,
Generally I agree with automated testing platforms, I think <insert you're preferred platform here> makes sense, there's a bunch like Linaro and a bunch in the arm ecosystem use LAVA [1] and there's a bunch more that I'm aware of.
I am somewhat familiar with LAVA and I believe it can be used to test U-Boot, although I need to learn how. Looking at a test run [2] for beaglebone black I see that it is using a recent kernel but the U-Boot seems to be older.
Does it make sense, of course at some point in the future post this being merged, make sense to look at a general way of making it easier to plugin these sort of HW test platforms using this as a basis? I ask mostly because putting a bunch of my devices into some sort of platform can auto test things and of course everyone has an opinion to which is the best one :-P
Yes. I had heard from Tom that Labgrid is the new hotness for now. Having dug into it I believe it is a good solution, although it can certainly be improved to handle scale better.
Anyway, IMO the current test hooks are not a great solution, just because the configuration is spread all over the place and it relies on lots of little shell scripts. So I believe that the Labgrid integration is a closer to where we want to be with others that come along.
I'd say all those scripts are actually here to ease integration with any system, booting U-Boot and Linux are two different beasts.
That's fine, go ahead and use the scripts. My point is that Labgrid doesn't need them and in fact it makes everything pretty painful if we try to use all of them.
I guess I really need to clean-up and post my former co-workers scripts as I strongly disagree with that statement.
See some examples below. Bear in mind also that my goal has been to get my lab fully running, that includes interactive access to boards, as well as running tests. For that I have been using tbot (with integrated build and software-loading features). WIth Labgrid I have been able to replace most of the pytest scripts, tbot and Labman with Labgrid + 60 patches and some configuration files.
Let's look through the scripts:
u-boot-test-common - new script
u-boot-test-console - needed
u-boot-test-flash - here's an example:
. poweroff.${power_impl} sleep 0.1 . flash.sdwire_common_mount . poweron.${power_impl}
Here's another:
# Handles the common SDwire mounting (caller does power control)
mount_dir=/media/${mount_point}
# Switch over to get USB card access sd-mux-ctrl --device-serial ${sdwire_serial} --ts
complete=false for i in {0..9}; do if out="$(mount UUID=${mount_uuid} 2>&1)"; then complete=true break fi echo $out
# If it is already mounted, try to unmount it first. It may have been # mounted by another user so we won't have the access we need. If this gives # an error then we know we cannot continue if [[ $out == *"already mounted"* ]]; then umount UUID=${mount_uuid} fi sleep 1 done if [[ $complete = false ]]; then echo "Failed to mount UUID ${mount_uuid} after 10 tries" exit 1 fi
# Sanity check if ! mountpoint -q ${mount_dir}; then echo "Mount ${mount_dir} not available after 'mount'" exit 1 fi
# Perform the write, pass along as much environment as possible . writer.${flash_writer}
complete=false for i in {0..9}; do if out="$(umount ${mount_dir} 2>&1)"; then complete=true break fi echo $out sleep 1 done
if [[ $complete = false ]]; then echo "Failed to umount UUID ${mount_uuid} after 10 tries" exit 1 fi
# Sanity check if mountpoint -q ${mount_dir}; then echo "Mount ${mount_dir} still available after 'umount'" exit 1 fi
# Back to card access for the DUT sd-mux-ctrl --device-serial ${sdwire_serial} --dut
Of course that assumes an SDwire which may not always be the case. Abstracting all of this into shell scripts is just too painful. Also I have found it to be error-prone.
There is also bin/flash.sdwire_relay_mount:
. poweroff.${power_impl}
sleep 0.1
. flash.sdwire_common_mount
. poweron.${power_impl}
u-boot-test-getrole - New script
u-boot-test-power-off - This one fits with Labgrid quite well, but of course there may not actually be power control
u-boot-test-power-on - This one also fits with Labgrid, but since serial output can come out as soon as it is called, jumping back into Labgrid later to bring up a console results in lots of missing output. Also there may not actually be power control
u-boot-test-release - New script. Actually this isn't really needed, since I added automatic release to Labgrid. I was just worried that Labgrid might crash
u-boot-test-reset - Unfortunately labgrid doesn't provide a reset option, although I'm sure it could be added. You still need to know whether to use power or reset or both, though, so again you have shell scripts with this stuff
reset.recovery_download - another thing I had to add, BTW
. "${bin_dir}/recovery.${recovery_impl}"
# USB enumeration delay for ((i = 0; i <= 100; i++)); do if [ -e "${board_usb_dev}" ]; then break fi sleep 0.1 done if [ ! -e "${board_usb_dev}" ]; then echo "Cannot find device ${board_usb_dev}" exit 1 fi sleep 1
. "${bin_dir}/download.${download_impl}"
or another case, this is Zynq which tries to wait a bit before calling picocom. It mostly works, but it loses output
# Some boards have a UART built into them and it doesn't appear until the # board is powered on. An example is zynq_zybo. Try to handle this by waiting # for the console and then continuing as soon as possible (so we don't miss # the U-Boot banner). if [ -n "${console_wait}" ]; then for i in {0..99}; do if [ -e "${console_dev}" ]; then break fi sleep .1 done if [ ! -e "${console_dev}" ]; then echo "Console ${console_dev} not found" exit 1 fi fi
exec picocom -b "${console_baud}" "${console_dev}"
So that's about a third of the reason why I don't think the scripts scale.
The only "painful" part is the shared pain, regardless of framework, for "put blob $X at location $Y", which I believe you abstract out in to python?
Sort-of. The Python knows how to handle all the different writing methods and is at [2], but it is entirely controlled by yaml[3] (the example_env.yaml). See the ti,am625 method for example and the 'play' board right at the bottom of the end file.
I very much like the configuration files (although I hope one day Labgrid will allow the two files to be combined). It makes it easier to add a new board, since there is much less to think about. It really did take a long time to fiddle with all those hook scripts and I got tired of it.
Regards, Simon
[1] https://github.com/sjg20/u-boot/tree/lab6a/tools/labman [2] https://github.com/labgrid-project/labgrid/pull/1411/commits/c7ea03987362af7... [3] https://github.com/labgrid-project/labgrid/pull/1411/commits/6d3b4a0cfc7e35e...
On 30/08/2024 03:09, Simon Glass wrote:
Hi Tom,
On Thu, 29 Aug 2024 at 09:20, Tom Rini trini@konsulko.com wrote:
On Thu, Aug 29, 2024 at 09:02:38AM -0600, Simon Glass wrote:
Hi Neil,
On Thu, 29 Aug 2024 at 08:44, neil.armstrong@linaro.org wrote:
On 29/08/2024 14:17, Simon Glass wrote:
Hi Peter,
On Thu, 29 Aug 2024 at 04:43, Peter Robinson pbrobinson@gmail.com wrote:
On Wed, 28 Aug 2024 at 22:25, Simon Glass sjg@chromium.org wrote: > > Hi Peter, > > On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote: >> >> Hi Simon, >> >>> With Labgrid we don't need to specify the various methods, except for >>> the console, which simply calls labgrid-client. >>> >>> This allows supporting any boards in your lab, without adding per-board >>> configuration to these hooks. >>> >>> Provide ellesmere files as an example. >> >> What's ellesmere? > > It is a lake but also the name of a computer. > >> >>> Signed-off-by: Simon Glass sjg@chromium.org >>> --- >>> >>> Changes in v4: >>> - Support pytest fully with dual-build boards like Beagleplay >>> >>> Changes in v3: >>> - Update scripts for latest version of Labgrid integration >>> - Add poweroff.none and poweron.none >>> - Provide -n flag when querying board info >>> - Target the grpc version of Labgrid which is now in -master >>> - Update README to cover the changes >>> >>> Changes in v2: >>> - Make use of the common script (only) to set bin_dir >>> >>> README.md | 50 ++++++++++++++++++++++++++++++++++++ >> >> Maybe that should be in a separate labsgrid readme? > > My hope is that Labgrid becomes the normal way of running these tests,
Generally I agree with automated testing platforms, I think <insert you're preferred platform here> makes sense, there's a bunch like Linaro and a bunch in the arm ecosystem use LAVA [1] and there's a bunch more that I'm aware of.
I am somewhat familiar with LAVA and I believe it can be used to test U-Boot, although I need to learn how. Looking at a test run [2] for beaglebone black I see that it is using a recent kernel but the U-Boot seems to be older.
Does it make sense, of course at some point in the future post this being merged, make sense to look at a general way of making it easier to plugin these sort of HW test platforms using this as a basis? I ask mostly because putting a bunch of my devices into some sort of platform can auto test things and of course everyone has an opinion to which is the best one :-P
Yes. I had heard from Tom that Labgrid is the new hotness for now. Having dug into it I believe it is a good solution, although it can certainly be improved to handle scale better.
Anyway, IMO the current test hooks are not a great solution, just because the configuration is spread all over the place and it relies on lots of little shell scripts. So I believe that the Labgrid integration is a closer to where we want to be with others that come along.
I'd say all those scripts are actually here to ease integration with any system, booting U-Boot and Linux are two different beasts.
That's fine, go ahead and use the scripts. My point is that Labgrid doesn't need them and in fact it makes everything pretty painful if we try to use all of them.
I guess I really need to clean-up and post my former co-workers scripts as I strongly disagree with that statement.
See some examples below. Bear in mind also that my goal has been to get my lab fully running, that includes interactive access to boards, as well as running tests. For that I have been using tbot (with integrated build and software-loading features). WIth Labgrid I have been able to replace most of the pytest scripts, tbot and Labman with Labgrid + 60 patches and some configuration files.
Let's look through the scripts:
u-boot-test-common - new script
u-boot-test-console - needed
u-boot-test-flash - here's an example:
. poweroff.${power_impl} sleep 0.1 . flash.sdwire_common_mount . poweron.${power_impl}
Here's another:
# Handles the common SDwire mounting (caller does power control)
mount_dir=/media/${mount_point}
# Switch over to get USB card access sd-mux-ctrl --device-serial ${sdwire_serial} --ts
complete=false for i in {0..9}; do if out="$(mount UUID=${mount_uuid} 2>&1)"; then complete=true break fi echo $out
# If it is already mounted, try to unmount it first. It may have been # mounted by another user so we won't have the access we need. If this gives # an error then we know we cannot continue if [[ $out == *"already mounted"* ]]; then umount UUID=${mount_uuid} fi sleep 1done if [[ $complete = false ]]; then echo "Failed to mount UUID ${mount_uuid} after 10 tries" exit 1 fi
# Sanity check if ! mountpoint -q ${mount_dir}; then echo "Mount ${mount_dir} not available after 'mount'" exit 1 fi
# Perform the write, pass along as much environment as possible . writer.${flash_writer}
complete=false for i in {0..9}; do if out="$(umount ${mount_dir} 2>&1)"; then complete=true break fi echo $out sleep 1 done
if [[ $complete = false ]]; then echo "Failed to umount UUID ${mount_uuid} after 10 tries" exit 1 fi
# Sanity check if mountpoint -q ${mount_dir}; then echo "Mount ${mount_dir} still available after 'umount'" exit 1 fi
# Back to card access for the DUT sd-mux-ctrl --device-serial ${sdwire_serial} --dut
Of course that assumes an SDwire which may not always be the case. Abstracting all of this into shell scripts is just too painful. Also I have found it to be error-prone.
There is also bin/flash.sdwire_relay_mount:
. poweroff.${power_impl}
sleep 0.1
. flash.sdwire_common_mount
. poweron.${power_impl}
u-boot-test-getrole
- New script
u-boot-test-power-off
- This one fits with Labgrid quite well, but of course there may not
actually be power control
u-boot-test-power-on
- This one also fits with Labgrid, but since serial output can come
out as soon as it is called, jumping back into Labgrid later to bring up a console results in lots of missing output. Also there may not actually be power control
u-boot-test-release
- New script. Actually this isn't really needed, since I added
automatic release to Labgrid. I was just worried that Labgrid might crash
u-boot-test-reset
- Unfortunately labgrid doesn't provide a reset option, although I'm
sure it could be added. You still need to know whether to use power or reset or both, though, so again you have shell scripts with this stuff
reset.recovery_download - another thing I had to add, BTW
. "${bin_dir}/recovery.${recovery_impl}"
# USB enumeration delay for ((i = 0; i <= 100; i++)); do if [ -e "${board_usb_dev}" ]; then break fi sleep 0.1 done if [ ! -e "${board_usb_dev}" ]; then echo "Cannot find device ${board_usb_dev}" exit 1 fi sleep 1
. "${bin_dir}/download.${download_impl}"
or another case, this is Zynq which tries to wait a bit before calling picocom. It mostly works, but it loses output
# Some boards have a UART built into them and it doesn't appear until the # board is powered on. An example is zynq_zybo. Try to handle this by waiting # for the console and then continuing as soon as possible (so we don't miss # the U-Boot banner). if [ -n "${console_wait}" ]; then for i in {0..99}; do if [ -e "${console_dev}" ]; then break fi sleep .1 done if [ ! -e "${console_dev}" ]; then echo "Console ${console_dev} not found" exit 1 fi fi
exec picocom -b "${console_baud}" "${console_dev}"
So that's about a third of the reason why I don't think the scripts scale.
My integration requires only this file:
bin/console.abcd ====><============ FIP_DIR=`mktemp -d`  pushd ${boot_fip_dir} ./build-fip.sh ${boot_fip_board} ${U_BOOT_BUILD_DIR}/u-boot.bin ${FIP_DIR} cp ${FIP_DIR}/u-boot.bin ${U_BOOT_BUILD_DIR}/u-boot.bin.abcd popd rm -rf ${FIP_DIR}  exec "${abcd_binary}" -h ${abcd_host} -b ${abcd_board} ${abcd_extra_args} ${U_BOOT_BUILD_DIR}/u-boot.bin.abcd ====><============
the abcd_binary does all the dirty work, connecting the the server and connecting the board TTY to stdout, and the server does all the dirty power-up and loading u-boot to memory.
I still wonder why you need so much changes if labgrid is so simple.
For LAVA, it would likely very similar, exact we should generate a job to load u-boot, submit it and somehow interact with the board.
Neil
The only "painful" part is the shared pain, regardless of framework, for "put blob $X at location $Y", which I believe you abstract out in to python?
Sort-of. The Python knows how to handle all the different writing methods and is at [2], but it is entirely controlled by yaml[3] (the example_env.yaml). See the ti,am625 method for example and the 'play' board right at the bottom of the end file.
I very much like the configuration files (although I hope one day Labgrid will allow the two files to be combined). It makes it easier to add a new board, since there is much less to think about. It really did take a long time to fiddle with all those hook scripts and I got tired of it.
Regards, Simon
[1] https://github.com/sjg20/u-boot/tree/lab6a/tools/labman [2] https://github.com/labgrid-project/labgrid/pull/1411/commits/c7ea03987362af7... [3] https://github.com/labgrid-project/labgrid/pull/1411/commits/6d3b4a0cfc7e35e...
On Thu, Aug 29, 2024 at 07:09:22PM -0600, Simon Glass wrote:
Hi Tom,
On Thu, 29 Aug 2024 at 09:20, Tom Rini trini@konsulko.com wrote:
On Thu, Aug 29, 2024 at 09:02:38AM -0600, Simon Glass wrote:
Hi Neil,
On Thu, 29 Aug 2024 at 08:44, neil.armstrong@linaro.org wrote:
On 29/08/2024 14:17, Simon Glass wrote:
Hi Peter,
On Thu, 29 Aug 2024 at 04:43, Peter Robinson pbrobinson@gmail.com wrote:
On Wed, 28 Aug 2024 at 22:25, Simon Glass sjg@chromium.org wrote: > > Hi Peter, > > On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote: >> >> Hi Simon, >> >>> With Labgrid we don't need to specify the various methods, except for >>> the console, which simply calls labgrid-client. >>> >>> This allows supporting any boards in your lab, without adding per-board >>> configuration to these hooks. >>> >>> Provide ellesmere files as an example. >> >> What's ellesmere? > > It is a lake but also the name of a computer. > >> >>> Signed-off-by: Simon Glass sjg@chromium.org >>> --- >>> >>> Changes in v4: >>> - Support pytest fully with dual-build boards like Beagleplay >>> >>> Changes in v3: >>> - Update scripts for latest version of Labgrid integration >>> - Add poweroff.none and poweron.none >>> - Provide -n flag when querying board info >>> - Target the grpc version of Labgrid which is now in -master >>> - Update README to cover the changes >>> >>> Changes in v2: >>> - Make use of the common script (only) to set bin_dir >>> >>> README.md | 50 ++++++++++++++++++++++++++++++++++++ >> >> Maybe that should be in a separate labsgrid readme? > > My hope is that Labgrid becomes the normal way of running these tests,
Generally I agree with automated testing platforms, I think <insert you're preferred platform here> makes sense, there's a bunch like Linaro and a bunch in the arm ecosystem use LAVA [1] and there's a bunch more that I'm aware of.
I am somewhat familiar with LAVA and I believe it can be used to test U-Boot, although I need to learn how. Looking at a test run [2] for beaglebone black I see that it is using a recent kernel but the U-Boot seems to be older.
Does it make sense, of course at some point in the future post this being merged, make sense to look at a general way of making it easier to plugin these sort of HW test platforms using this as a basis? I ask mostly because putting a bunch of my devices into some sort of platform can auto test things and of course everyone has an opinion to which is the best one :-P
Yes. I had heard from Tom that Labgrid is the new hotness for now. Having dug into it I believe it is a good solution, although it can certainly be improved to handle scale better.
Anyway, IMO the current test hooks are not a great solution, just because the configuration is spread all over the place and it relies on lots of little shell scripts. So I believe that the Labgrid integration is a closer to where we want to be with others that come along.
I'd say all those scripts are actually here to ease integration with any system, booting U-Boot and Linux are two different beasts.
That's fine, go ahead and use the scripts. My point is that Labgrid doesn't need them and in fact it makes everything pretty painful if we try to use all of them.
I guess I really need to clean-up and post my former co-workers scripts as I strongly disagree with that statement.
See some examples below. Bear in mind also that my goal has been to get my lab fully running, that includes interactive access to boards, as well as running tests. For that I have been using tbot (with integrated build and software-loading features). WIth Labgrid I have been able to replace most of the pytest scripts, tbot and Labman with Labgrid + 60 patches and some configuration files.
That's certainly good and useful progress. And I feel like one of the lessons of this thread is that every lab ends up a bit different and so there's just not going to be as much standardization as we'd all like.
Let's look through the scripts:
u-boot-test-common - new script
u-boot-test-console - needed
u-boot-test-flash - here's an example:
. poweroff.${power_impl} sleep 0.1 . flash.sdwire_common_mount . poweron.${power_impl}
Here's another:
# Handles the common SDwire mounting (caller does power control)
mount_dir=/media/${mount_point}
# Switch over to get USB card access sd-mux-ctrl --device-serial ${sdwire_serial} --ts
complete=false for i in {0..9}; do if out="$(mount UUID=${mount_uuid} 2>&1)"; then complete=true break fi echo $out
# If it is already mounted, try to unmount it first. It may have been # mounted by another user so we won't have the access we need. If this gives # an error then we know we cannot continue if [[ $out == *"already mounted"* ]]; then umount UUID=${mount_uuid} fi sleep 1done if [[ $complete = false ]]; then echo "Failed to mount UUID ${mount_uuid} after 10 tries" exit 1 fi
# Sanity check if ! mountpoint -q ${mount_dir}; then echo "Mount ${mount_dir} not available after 'mount'" exit 1 fi
# Perform the write, pass along as much environment as possible . writer.${flash_writer}
complete=false for i in {0..9}; do if out="$(umount ${mount_dir} 2>&1)"; then complete=true break fi echo $out sleep 1 done
if [[ $complete = false ]]; then echo "Failed to umount UUID ${mount_uuid} after 10 tries" exit 1 fi
# Sanity check if mountpoint -q ${mount_dir}; then echo "Mount ${mount_dir} still available after 'umount'" exit 1 fi
# Back to card access for the DUT sd-mux-ctrl --device-serial ${sdwire_serial} --dut
Of course that assumes an SDwire which may not always be the case.
Or some other piece of hardware that sd-mux-ctrl supports, but yes.
Abstracting all of this into shell scripts is just too painful.
Abstracting it to something else sounds even more painful however.
Also I have found it to be error-prone.
There is also bin/flash.sdwire_relay_mount:
. poweroff.${power_impl}
sleep 0.1
. flash.sdwire_common_mount
. poweron.${power_impl}
Yup. It's possible we could do some slightly nicer check/sleep loop but it probably wouldn't really get us much.
u-boot-test-getrole
- New script
Which is for how you abstracted hardware from software configurations it can run (for example, the Pi 4 case of rpi_4, rpi_4_32b and rpi_arm64 all being valid defconfigs).
u-boot-test-power-off
- This one fits with Labgrid quite well, but of course there may not
actually be power control
u-boot-test-power-on
- This one also fits with Labgrid, but since serial output can come
out as soon as it is called, jumping back into Labgrid later to bring up a console results in lots of missing output. Also there may not actually be power control
Right, but in both cases, poweron/poweroff.labgrid could just do: labgrid-client power $state || true if power isn't supported (which sounds odd, but, maybe makes sense in some setup).
u-boot-test-release
- New script. Actually this isn't really needed, since I added
automatic release to Labgrid. I was just worried that Labgrid might crash
What I saw in working on my scripts was that if you don't release you can't acquire later and there's not an option for "you already have acquired it". On my scripts, I should update the cleanup function to release the current LG_PLACE.
u-boot-test-reset
- Unfortunately labgrid doesn't provide a reset option, although I'm
sure it could be added. You still need to know whether to use power or reset or both, though, so again you have shell scripts with this stuff
Perhaps this is something to talk with upstream about, for the cases where "reset" is not just another name for "power cycle" ? I know we have scripts today for a number of cases but the only ones I wrote are just power cycle (digital-loggers) or power cycle via USB (ykush, the USB hubs that support per-port power control intentionally).
reset.recovery_download - another thing I had to add, BTW
. "${bin_dir}/recovery.${recovery_impl}"
# USB enumeration delay for ((i = 0; i <= 100; i++)); do if [ -e "${board_usb_dev}" ]; then break fi sleep 0.1 done if [ ! -e "${board_usb_dev}" ]; then echo "Cannot find device ${board_usb_dev}" exit 1 fi sleep 1
. "${bin_dir}/download.${download_impl}"
or another case, this is Zynq which tries to wait a bit before calling picocom. It mostly works, but it loses output
# Some boards have a UART built into them and it doesn't appear until the # board is powered on. An example is zynq_zybo. Try to handle this by waiting # for the console and then continuing as soon as possible (so we don't miss # the U-Boot banner). if [ -n "${console_wait}" ]; then for i in {0..99}; do if [ -e "${console_dev}" ]; then break fi sleep .1 done if [ ! -e "${console_dev}" ]; then echo "Console ${console_dev} not found" exit 1 fi fi
exec picocom -b "${console_baud}" "${console_dev}"
So that's about a third of the reason why I don't think the scripts scale.
This is where I think looking at https://github.com/bootlin/snagboot which Kevin Hillman pointed me at a while ago might come in handy, instead.
The only "painful" part is the shared pain, regardless of framework, for "put blob $X at location $Y", which I believe you abstract out in to python?
Sort-of. The Python knows how to handle all the different writing methods and is at [2], but it is entirely controlled by yaml[3] (the example_env.yaml). See the ti,am625 method for example and the 'play' board right at the bottom of the end file.
I very much like the configuration files (although I hope one day Labgrid will allow the two files to be combined). It makes it easier to add a new board, since there is much less to think about. It really did take a long time to fiddle with all those hook scripts and I got tired of it.
Regards, Simon
[1] https://github.com/sjg20/u-boot/tree/lab6a/tools/labman [2] https://github.com/labgrid-project/labgrid/pull/1411/commits/c7ea03987362af7... [3] https://github.com/labgrid-project/labgrid/pull/1411/commits/6d3b4a0cfc7e35e...
Yeah, personally I don't care for that. I know I'm not going to convince people and it's just one of those "goodness, I'm the old guy now" things, but, no. It would be easier if you had comments for each SoC as to what / why each of the magic values, but then shouldn't it be abstracted more? Perhaps to a file per SoC, and at that point how is this easier to follow than a commented shell script? But in sum, I'm just cranky about yaml and I know your opinion is the opposite.
As an aside, please review the ti,am625 stuff with some TI folks, that's both far too specific (you're going to need to add different am62 and am64 and am69 and j7... and on and on cases) and I forget if there's some work to have a slightly easier way to know what file to use going on.
Hi Tom,
On Fri, 30 Aug 2024 at 09:13, Tom Rini trini@konsulko.com wrote:
On Thu, Aug 29, 2024 at 07:09:22PM -0600, Simon Glass wrote:
Hi Tom,
On Thu, 29 Aug 2024 at 09:20, Tom Rini trini@konsulko.com wrote:
On Thu, Aug 29, 2024 at 09:02:38AM -0600, Simon Glass wrote:
Hi Neil,
On Thu, 29 Aug 2024 at 08:44, neil.armstrong@linaro.org wrote:
On 29/08/2024 14:17, Simon Glass wrote:
Hi Peter,
On Thu, 29 Aug 2024 at 04:43, Peter Robinson pbrobinson@gmail.com wrote: > > On Wed, 28 Aug 2024 at 22:25, Simon Glass sjg@chromium.org wrote: >> >> Hi Peter, >> >> On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote: >>> >>> Hi Simon, >>> >>>> With Labgrid we don't need to specify the various methods, except for >>>> the console, which simply calls labgrid-client. >>>> >>>> This allows supporting any boards in your lab, without adding per-board >>>> configuration to these hooks. >>>> >>>> Provide ellesmere files as an example. >>> >>> What's ellesmere? >> >> It is a lake but also the name of a computer. >> >>> >>>> Signed-off-by: Simon Glass sjg@chromium.org >>>> --- >>>> >>>> Changes in v4: >>>> - Support pytest fully with dual-build boards like Beagleplay >>>> >>>> Changes in v3: >>>> - Update scripts for latest version of Labgrid integration >>>> - Add poweroff.none and poweron.none >>>> - Provide -n flag when querying board info >>>> - Target the grpc version of Labgrid which is now in -master >>>> - Update README to cover the changes >>>> >>>> Changes in v2: >>>> - Make use of the common script (only) to set bin_dir >>>> >>>> README.md | 50 ++++++++++++++++++++++++++++++++++++ >>> >>> Maybe that should be in a separate labsgrid readme? >> >> My hope is that Labgrid becomes the normal way of running these tests, > > Generally I agree with automated testing platforms, I think <insert > you're preferred platform here> makes sense, there's a bunch like > Linaro and a bunch in the arm ecosystem use LAVA [1] and there's a > bunch more that I'm aware of.
I am somewhat familiar with LAVA and I believe it can be used to test U-Boot, although I need to learn how. Looking at a test run [2] for beaglebone black I see that it is using a recent kernel but the U-Boot seems to be older.
> > Does it make sense, of course at some point in the future post this > being merged, make sense to look at a general way of making it easier > to plugin these sort of HW test platforms using this as a basis? I ask > mostly because putting a bunch of my devices into some sort of > platform can auto test things and of course everyone has an opinion to > which is the best one :-P
Yes. I had heard from Tom that Labgrid is the new hotness for now. Having dug into it I believe it is a good solution, although it can certainly be improved to handle scale better.
Anyway, IMO the current test hooks are not a great solution, just because the configuration is spread all over the place and it relies on lots of little shell scripts. So I believe that the Labgrid integration is a closer to where we want to be with others that come along.
I'd say all those scripts are actually here to ease integration with any system, booting U-Boot and Linux are two different beasts.
That's fine, go ahead and use the scripts. My point is that Labgrid doesn't need them and in fact it makes everything pretty painful if we try to use all of them.
I guess I really need to clean-up and post my former co-workers scripts as I strongly disagree with that statement.
See some examples below. Bear in mind also that my goal has been to get my lab fully running, that includes interactive access to boards, as well as running tests. For that I have been using tbot (with integrated build and software-loading features). WIth Labgrid I have been able to replace most of the pytest scripts, tbot and Labman with Labgrid + 60 patches and some configuration files.
That's certainly good and useful progress. And I feel like one of the lessons of this thread is that every lab ends up a bit different and so there's just not going to be as much standardization as we'd all like.
Let's look through the scripts:
u-boot-test-common - new script
u-boot-test-console - needed
u-boot-test-flash - here's an example:
. poweroff.${power_impl} sleep 0.1 . flash.sdwire_common_mount . poweron.${power_impl}
Here's another:
# Handles the common SDwire mounting (caller does power control)
mount_dir=/media/${mount_point}
# Switch over to get USB card access sd-mux-ctrl --device-serial ${sdwire_serial} --ts
complete=false for i in {0..9}; do if out="$(mount UUID=${mount_uuid} 2>&1)"; then complete=true break fi echo $out
# If it is already mounted, try to unmount it first. It may have been # mounted by another user so we won't have the access we need. If this gives # an error then we know we cannot continue if [[ $out == *"already mounted"* ]]; then umount UUID=${mount_uuid} fi sleep 1done if [[ $complete = false ]]; then echo "Failed to mount UUID ${mount_uuid} after 10 tries" exit 1 fi
# Sanity check if ! mountpoint -q ${mount_dir}; then echo "Mount ${mount_dir} not available after 'mount'" exit 1 fi
# Perform the write, pass along as much environment as possible . writer.${flash_writer}
complete=false for i in {0..9}; do if out="$(umount ${mount_dir} 2>&1)"; then complete=true break fi echo $out sleep 1 done
if [[ $complete = false ]]; then echo "Failed to umount UUID ${mount_uuid} after 10 tries" exit 1 fi
# Sanity check if mountpoint -q ${mount_dir}; then echo "Mount ${mount_dir} still available after 'umount'" exit 1 fi
# Back to card access for the DUT sd-mux-ctrl --device-serial ${sdwire_serial} --dut
Of course that assumes an SDwire which may not always be the case.
Or some other piece of hardware that sd-mux-ctrl supports, but yes.
Abstracting all of this into shell scripts is just too painful.
Abstracting it to something else sounds even more painful however.
I see this a little bit like the situation with Binman. We now have a description for the image and can build for any SoC which has added its peculiarities in a Python script.
In a similar way, I believe it is valuable to have each SoC encode (in some sort of description) how it writes images to an SD card, over USB, etc.
Now I am not a huge fan of yaml but it does the job and it is what Labgrid uses. So my vision of all this is that if someone wants to set up a lab with 5 boards, they can get the gear together, wire it all up, then copy and modify some yaml and it should all work.
Also I have found it to be error-prone.
There is also bin/flash.sdwire_relay_mount:
. poweroff.${power_impl}
sleep 0.1
. flash.sdwire_common_mount
. poweron.${power_impl}
Yup. It's possible we could do some slightly nicer check/sleep loop but it probably wouldn't really get us much.
u-boot-test-getrole
- New script
Which is for how you abstracted hardware from software configurations it can run (for example, the Pi 4 case of rpi_4, rpi_4_32b and rpi_arm64 all being valid defconfigs).
Yes it maps a 'role' (like an 'identity' in test.py) to a particular DUT and the corresponding U-Boot board. It can be many-to-one.
u-boot-test-power-off
- This one fits with Labgrid quite well, but of course there may not
actually be power control
u-boot-test-power-on
- This one also fits with Labgrid, but since serial output can come
out as soon as it is called, jumping back into Labgrid later to bring up a console results in lots of missing output. Also there may not actually be power control
Right, but in both cases, poweron/poweroff.labgrid could just do: labgrid-client power $state || true if power isn't supported (which sounds odd, but, maybe makes sense in some setup).
Yes it can. I have quite a few boards with no power control (they are just on all the time). I find a reset line more useful than power in some cases.
u-boot-test-release
- New script. Actually this isn't really needed, since I added
automatic release to Labgrid. I was just worried that Labgrid might crash
What I saw in working on my scripts was that if you don't release you can't acquire later and there's not an option for "you already have acquired it". On my scripts, I should update the cleanup function to release the current LG_PLACE.
Yes I hit the same problem. I added an '--auto' option to acquire and release, so they are more forgiving.
u-boot-test-reset
- Unfortunately labgrid doesn't provide a reset option, although I'm
sure it could be added. You still need to know whether to use power or reset or both, though, so again you have shell scripts with this stuff
Perhaps this is something to talk with upstream about, for the cases where "reset" is not just another name for "power cycle" ? I know we have scripts today for a number of cases but the only ones I wrote are just power cycle (digital-loggers) or power cycle via USB (ykush, the USB hubs that support per-port power control intentionally).
If you did down a bit it ends up being intertwined with the U-Boot strategy...e.g. you for USB you often need to hold reset down while holding recovery, then release reset, then write U-Boot and wait for it to come up.
reset.recovery_download - another thing I had to add, BTW
. "${bin_dir}/recovery.${recovery_impl}"
# USB enumeration delay for ((i = 0; i <= 100; i++)); do if [ -e "${board_usb_dev}" ]; then break fi sleep 0.1 done if [ ! -e "${board_usb_dev}" ]; then echo "Cannot find device ${board_usb_dev}" exit 1 fi sleep 1
. "${bin_dir}/download.${download_impl}"
or another case, this is Zynq which tries to wait a bit before calling picocom. It mostly works, but it loses output
# Some boards have a UART built into them and it doesn't appear until the # board is powered on. An example is zynq_zybo. Try to handle this by waiting # for the console and then continuing as soon as possible (so we don't miss # the U-Boot banner). if [ -n "${console_wait}" ]; then for i in {0..99}; do if [ -e "${console_dev}" ]; then break fi sleep .1 done if [ ! -e "${console_dev}" ]; then echo "Console ${console_dev} not found" exit 1 fi fi
exec picocom -b "${console_baud}" "${console_dev}"
So that's about a third of the reason why I don't think the scripts scale.
This is where I think looking at https://github.com/bootlin/snagboot which Kevin Hillman pointed me at a while ago might come in handy, instead.
Oh yes I ran into that at a show a year or two back, very nice! Better than having a lot of little tools.
The only "painful" part is the shared pain, regardless of framework, for "put blob $X at location $Y", which I believe you abstract out in to python?
Sort-of. The Python knows how to handle all the different writing methods and is at [2], but it is entirely controlled by yaml[3] (the example_env.yaml). See the ti,am625 method for example and the 'play' board right at the bottom of the end file.
I very much like the configuration files (although I hope one day Labgrid will allow the two files to be combined). It makes it easier to add a new board, since there is much less to think about. It really did take a long time to fiddle with all those hook scripts and I got tired of it.
Regards, Simon
[1] https://github.com/sjg20/u-boot/tree/lab6a/tools/labman [2] https://github.com/labgrid-project/labgrid/pull/1411/commits/c7ea03987362af7... [3] https://github.com/labgrid-project/labgrid/pull/1411/commits/6d3b4a0cfc7e35e...
Yeah, personally I don't care for that. I know I'm not going to convince people and it's just one of those "goodness, I'm the old guy now" things, but, no. It would be easier if you had comments for each SoC as to what / why each of the magic values, but then shouldn't it be abstracted more? Perhaps to a file per SoC, and at that point how is this easier to follow than a commented shell script? But in sum, I'm just cranky about yaml and I know your opinion is the opposite.
As above I don't like it either, but it is what Labgrid has chosen. In fact I used it with Labman too...it really needs proper phandles like devicetree. At least is isn't JSON though...:-)
As an aside, please review the ti,am625 stuff with some TI folks, that's both far too specific (you're going to need to add different am62 and am64 and am69 and j7... and on and on cases) and I forget if there's some work to have a slightly easier way to know what file to use going on.
Well, perhaps they can take it over and get everything working? I have done the hard yards, e.g. supporting two builds together.
Regards, Simon
On Sun, Sep 01, 2024 at 02:09:47PM -0600, Simon Glass wrote:
[snip]
As an aside, please review the ti,am625 stuff with some TI folks, that's both far too specific (you're going to need to add different am62 and am64 and am69 and j7... and on and on cases) and I forget if there's some work to have a slightly easier way to know what file to use going on.
Well, perhaps they can take it over and get everything working? I have done the hard yards, e.g. supporting two builds together.
While I'm still hopeful about getting TI to run our pytests in their lab, currently there's not much hope in getting them augment their kernelci lab (which I think does use labgrid not LAVA) to also do U-Boot, so no, I don't think they're likely to pick up and work on this part. Given that things are described for binman I thought you might be wanting to look harder at things there.
Hi Tom,
On Tue, 3 Sept 2024 at 20:54, Tom Rini trini@konsulko.com wrote:
On Sun, Sep 01, 2024 at 02:09:47PM -0600, Simon Glass wrote:
[snip]
As an aside, please review the ti,am625 stuff with some TI folks, that's both far too specific (you're going to need to add different am62 and am64 and am69 and j7... and on and on cases) and I forget if there's some work to have a slightly easier way to know what file to use going on.
Well, perhaps they can take it over and get everything working? I have done the hard yards, e.g. supporting two builds together.
While I'm still hopeful about getting TI to run our pytests in their lab, currently there's not much hope in getting them augment their kernelci lab (which I think does use labgrid not LAVA) to also do U-Boot, so no, I don't think they're likely to pick up and work on this part. Given that things are described for binman I thought you might be wanting to look harder at things there.
I only have that one board (Beagleplay) in my lab so I haven't looked at anything else. If I end up getting a different board then I will probably add that as well. But for now I have quite a lot of outstanding patches and PRs so some things need to land, first.
Regards, Simon
On Thu, Aug 29, 2024 at 11:43:12AM +0100, Peter Robinson wrote:
On Wed, 28 Aug 2024 at 22:25, Simon Glass sjg@chromium.org wrote:
Hi Peter,
On Wed, 28 Aug 2024 at 12:14, Peter Robinson pbrobinson@gmail.com wrote:
Hi Simon,
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
What's ellesmere?
It is a lake but also the name of a computer.
Signed-off-by: Simon Glass sjg@chromium.org
Changes in v4:
- Support pytest fully with dual-build boards like Beagleplay
Changes in v3:
- Update scripts for latest version of Labgrid integration
- Add poweroff.none and poweron.none
- Provide -n flag when querying board info
- Target the grpc version of Labgrid which is now in -master
- Update README to cover the changes
Changes in v2:
- Make use of the common script (only) to set bin_dir
README.md | 50 ++++++++++++++++++++++++++++++++++++
Maybe that should be in a separate labsgrid readme?
My hope is that Labgrid becomes the normal way of running these tests,
Generally I agree with automated testing platforms, I think <insert you're preferred platform here> makes sense, there's a bunch like Linaro and a bunch in the arm ecosystem use LAVA [1] and there's a bunch more that I'm aware of.
Does it make sense, of course at some point in the future post this being merged, make sense to look at a general way of making it easier to plugin these sort of HW test platforms using this as a basis? I ask mostly because putting a bunch of my devices into some sort of platform can auto test things and of course everyone has an opinion to which is the best one :-P
Well, as the repository stands today, yes, assuming there's a handful of CLI tools for LAVA for "give me console", "change power state" and "write blob to filesystem/location", LAVA could be integrated fairly easily.
The alternative for labgrid support is what I'm using where U-Boot is built outside of test.py, and we have labgrid-client console/power cycle and then more per-SoC "write" scripts (as Pi is different from TI K3 which is different from rockchip ...).
On Wed, Aug 28, 2024 at 10:45:23AM -0600, Simon Glass wrote:
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
Signed-off-by: Simon Glass sjg@chromium.org
I think maybe we call this version "labgrid-buildman" and make sure the documentation emphasizes using buildman to provide U-Boot. This will make it easier for me to provide a follow-up of say "labgrid-external" that requires the binary to be provided, similar to our other hooks.
My thinking is that while it would be good to be able to expose labs to CI, and I would like to get mine showing up as well it's also important that I be able to keep doing the type I do where I extend the normal defconfig with additional features to test.
Hi Tom,
On Wed, 28 Aug 2024 at 14:19, Tom Rini trini@konsulko.com wrote:
On Wed, Aug 28, 2024 at 10:45:23AM -0600, Simon Glass wrote:
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
Signed-off-by: Simon Glass sjg@chromium.org
I think maybe we call this version "labgrid-buildman" and make sure the documentation emphasizes using buildman to provide U-Boot. This will make it easier for me to provide a follow-up of say "labgrid-external" that requires the binary to be provided, similar to our other hooks.
It can be used in this way. The 'ub-pyt -B' does a bootstrap of the board without doing a build, if that helps to show the path through the code.
If you don't pass --build to test.py then it won't do a build.
If you like, you can pass --build-dir (and with the testb series --build-dir-extra) as you do now. The binary will be written to the board as is.
My thinking is that while it would be good to be able to expose labs to CI, and I would like to get mine showing up as well it's also important that I be able to keep doing the type I do where I extend the normal defconfig with additional features to test.
Let me know if this doesn't work for you. It is certainly an important use case.
Regards, Simon
On Wed, Aug 28, 2024 at 03:25:00PM -0600, Simon Glass wrote:
Hi Tom,
On Wed, 28 Aug 2024 at 14:19, Tom Rini trini@konsulko.com wrote:
On Wed, Aug 28, 2024 at 10:45:23AM -0600, Simon Glass wrote:
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
Signed-off-by: Simon Glass sjg@chromium.org
I think maybe we call this version "labgrid-buildman" and make sure the documentation emphasizes using buildman to provide U-Boot. This will make it easier for me to provide a follow-up of say "labgrid-external" that requires the binary to be provided, similar to our other hooks.
It can be used in this way. The 'ub-pyt -B' does a bootstrap of the board without doing a build, if that helps to show the path through the code.
If you don't pass --build to test.py then it won't do a build.
If you like, you can pass --build-dir (and with the testb series --build-dir-extra) as you do now. The binary will be written to the board as is.
Right, but this gets back to what I'm most confused about (and need to talk with Kevin Hillman about maybe, but I missed my last meeting where he was as well), you have a bunch of logic for integration of buildman too, and I have a few wrappers around exiting labgrid-client commands. And I don't know what direction upstream is going to want to take.
Hi Tom,
On Wed, 28 Aug 2024 at 15:33, Tom Rini trini@konsulko.com wrote:
On Wed, Aug 28, 2024 at 03:25:00PM -0600, Simon Glass wrote:
Hi Tom,
On Wed, 28 Aug 2024 at 14:19, Tom Rini trini@konsulko.com wrote:
On Wed, Aug 28, 2024 at 10:45:23AM -0600, Simon Glass wrote:
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
Signed-off-by: Simon Glass sjg@chromium.org
I think maybe we call this version "labgrid-buildman" and make sure the documentation emphasizes using buildman to provide U-Boot. This will make it easier for me to provide a follow-up of say "labgrid-external" that requires the binary to be provided, similar to our other hooks.
It can be used in this way. The 'ub-pyt -B' does a bootstrap of the board without doing a build, if that helps to show the path through the code.
If you don't pass --build to test.py then it won't do a build.
If you like, you can pass --build-dir (and with the testb series --build-dir-extra) as you do now. The binary will be written to the board as is.
Right, but this gets back to what I'm most confused about (and need to talk with Kevin Hillman about maybe, but I missed my last meeting where he was as well), you have a bunch of logic for integration of buildman too, and I have a few wrappers around exiting labgrid-client commands. And I don't know what direction upstream is going to want to take.
Hmm, who is Kevin?
At the moment we are a long way from upstream. I have only one reviewed patch and it is pretty trivial. I just updated the stack again. I am hoping to meet some of the people in a few weeks, but they are pretty tied up on other projects, from my understanding.
Anyway, we could have a chat about it if you like. So far I'm pretty comfortable with the approach taken here. At this point it replaces my Labman as well as tbot for my use cases. Without integrated building, everything will be a PITA for me.
I am not sure this much matters on your side though, since you don't intend to use the building features here? How exactly are you building U-Boot? Is it using the --build option with pytest, or something else?
In the interim, one way of threading the needle would be to have test.py get the info from Labgrid using the 'query' and then run in the build in pytest. Or, even worse, have a separate yaml file with the required info and parse that in pytest and run buildman there... but as I say we are a long way from getting this integrated into Labgrid, so it is somewhat academic at present.
My immediate goal is to get all these things merged in U-Boot, so I can get my lab running properly with gitlab, etc. and ideally allow other custodians to push to it. Getting the integration landed in labgrid will likely come a lot later.
Regards, Simon
+Kevin,
On 8/29/24 00:27, Simon Glass wrote:
Hi Tom,
On Wed, 28 Aug 2024 at 15:33, Tom Rini trini@konsulko.com wrote:
On Wed, Aug 28, 2024 at 03:25:00PM -0600, Simon Glass wrote:
Hi Tom,
On Wed, 28 Aug 2024 at 14:19, Tom Rini trini@konsulko.com wrote:
On Wed, Aug 28, 2024 at 10:45:23AM -0600, Simon Glass wrote:
With Labgrid we don't need to specify the various methods, except for the console, which simply calls labgrid-client.
This allows supporting any boards in your lab, without adding per-board configuration to these hooks.
Provide ellesmere files as an example.
Signed-off-by: Simon Glass sjg@chromium.org
I think maybe we call this version "labgrid-buildman" and make sure the documentation emphasizes using buildman to provide U-Boot. This will make it easier for me to provide a follow-up of say "labgrid-external" that requires the binary to be provided, similar to our other hooks.
It can be used in this way. The 'ub-pyt -B' does a bootstrap of the board without doing a build, if that helps to show the path through the code.
If you don't pass --build to test.py then it won't do a build.
If you like, you can pass --build-dir (and with the testb series --build-dir-extra) as you do now. The binary will be written to the board as is.
Right, but this gets back to what I'm most confused about (and need to talk with Kevin Hillman about maybe, but I missed my last meeting where he was as well), you have a bunch of logic for integration of buildman too, and I have a few wrappers around exiting labgrid-client commands. And I don't know what direction upstream is going to want to take.
Hmm, who is Kevin?
Kevin was starting with kernel testing long time ago which become kernelci.
At the moment we are a long way from upstream. I have only one reviewed patch and it is pretty trivial. I just updated the stack again. I am hoping to meet some of the people in a few weeks, but they are pretty tied up on other projects, from my understanding.
Anyway, we could have a chat about it if you like. So far I'm pretty comfortable with the approach taken here. At this point it replaces my Labman as well as tbot for my use cases. Without integrated building, everything will be a PITA for me.
I am not sure this much matters on your side though, since you don't intend to use the building features here? How exactly are you building U-Boot? Is it using the --build option with pytest, or something else?
In the interim, one way of threading the needle would be to have test.py get the info from Labgrid using the 'query' and then run in the build in pytest. Or, even worse, have a separate yaml file with the required info and parse that in pytest and run buildman there... but as I say we are a long way from getting this integrated into Labgrid, so it is somewhat academic at present.
My immediate goal is to get all these things merged in U-Boot, so I can get my lab running properly with gitlab, etc. and ideally allow other custodians to push to it. Getting the integration landed in labgrid will likely come a lot later.
We are using these hooks script for u-boot testing for a while in AMD boardfarm. It is not perfect but we find a way how to get it work with AMD boardfarm.
As is visible there is never going to be one boardfarm solution because it is hard to do switch from existing system to something else and move all infrastructure around.
Recently Linaro announced onelab [1] where the part of it is LAA. Pretty much ready to go box which you can connect your local HW to it and start to schedule jobs to it.
And I think this is a step which I am missing here. I have bunch of HW here and I know how to configure it, boot it, etc but I want to use it and I don't really want to take care about lab SW, maintain it, test it, etc.
It means I have no issue with Labgrind but I don't want to waste my time on making sure that Labgrind itself is working. I just want to configure my board connection once and then do the work not really updating Labgrind instances, deal with dependencies, etc.
It means having generic box, sw with connection to gitlab should get us to situation that u-boot will be tested on more HW and make it available for others.
Thanks, Michal
[1] https://www.linaro.org/blog/onelab-launch-at-linaro-connect-madrid-2024/
participants (5)
-
Michal Simek -
neil.armstrong＠linaro.org -
Peter Robinson -
Simon Glass -
Tom Rini