My labgrid setup
Hey all,
So now that I've posted my u-boot-test-hooks for labgrid: https://patchwork.ozlabs.org/project/uboot/patch/20240829185620.3179866-1-tr... the next relevant parts would be how I use that. There's three scripts, first of which is "uboot-hw-testcycle.sh" : ------------------------ >8 ------------------------ #!/bin/bash set -e
# Save output LOG=$HOME/logs/`git describe --abbrev=0`/`date +"%Y-%m-%d-%H%M"`-hw.txt
# Make a base directory B=`mktemp -d` chmod a+rwX ${B}
cleanup() { # Clean up random outputs [ -d test/py/.pytest_cache ] && find -type d -name __pycache__ | xargs rm -r sudo rm -rf ${B} rm -rf test/py/.pytest_cache }
# Cleanups trap cleanup INT trap cleanup ERR trap cleanup EXIT
# Find our arguments while test $# -ne 0; do if [ "$1" == "--no-32bit" ]; then NO32BIT="$1" shift elif [ "$1" == "--no-64bit" ]; then NO64BIT="$1" shift else echo "Unknown argument $1" exit 1 fi done
# Run the build if [ -f .gitlab-ci.yml ]; then IMG=`grep ci_runner_image: .azure-pipelines.yml | cut -d ' ' -f 4` else IMG=`git grep ci_runner_image: origin/master -- .azure-pipelines.yml | cut -d ' ' -f 4` fi
# Build all platforms touch $LOG && chmod a+rw $LOG sudo docker run --rm -it --privileged \ -v /home/trini/bin:/home/uboot/bin \ -v /home/trini/u-boot:/home/uboot/u-boot \ -v /home/trini/logs:/home/uboot/logs \ -v ${B}:${B} \ -w /home/uboot/u-boot/u-boot \ -e PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/uboot/u-boot/u-boot-test-hooks/bin \ $IMG \ /home/uboot/bin/uboot-local-ci-in-docker.sh ${B} $NO32BIT $NO64BIT --log $LOG
# Test all targets /home/trini/bin/uboot-local-ci-outside-docker.sh $B $NO32BIT $NO64BIT --log $LOG
less $LOG ------------------------ >8 ------------------------
Which is still a little debug-y, but is basically just build everything I want in the same container as CI, and then run pytest outside of that container. Could I do it inside the container? Maybe. This has grown from when I used to build everything on the host and used "flashair" SD cards to update my hardware. With labgrid everything might be abstracted enough that yes, really, I can stop doing that. I'll likely look in to that next just to clean things up more.
Second is "uboot-local-ci-in-docker.sh" as invoked above: ------------------------ >8 ------------------------ #!/bin/bash # Build everything we are going to test. set -e
# Let git stop being mad about permissions. git config --global --add safe.directory /home/uboot/u-boot/u-boot
buildaboard() { # Board is first, all fragments follow. BRD=$1 shift 1
# Second would be to use clang if [ "$1" == "--clang" ]; then echo "Building $BRD with clang" | tee -a $LOG OBJ=${B}/${BRD}_clang export HOSTCC=clang-16 export CC=clang-16 shift 1 else echo "Building $BRD" | tee -a $LOG OBJ=${B}/${BRD} fi # All that remains are fragments. FRAGS=$@
make O=${OBJ} ${BRD}_defconfig -s 2>&1 | tee -a $LOG pushd ${OBJ} >/dev/null [ ! -z "${FRAGS}" ] && ${S}/scripts/kconfig/merge_config.sh ${S}/configs/${BRD}_defconfig ${FRAGS} 2>&1 | tee -a $LOG make -sj$(nproc) 2>&1 | tee -a $LOG popd >/dev/null }
# Prepare virtualenv if [ -f test/py/requirements.txt ]; then virtualenv -p /usr/bin/python3 /tmp/venv . /tmp/venv/bin/activate pip install --quiet -r test/py/requirements.txt pip install --quiet -r doc/sphinx/requirements.txt [ -f tools/buildman/requirements.txt ] && pip install --quiet -r tools/buildman/requirements.txt fi
# Find our arguments while test $# -ne 0; do if [ "$1" == "--no-32bit" ]; then NO32BIT="true" shift elif [ "$1" == "--no-64bit" ]; then NO64BIT="true" shift elif [ "$1" == "--log" ]; then LOG=${2/trini/uboot} shift 2 else B=$1 shift fi done
if [ -z "$LOG" ]; then echo "Must pass --log /path/to/log/file" exit 1 fi
S=`pwd`
# Common to all newer TI platforms export BINMAN_INDIRS=/home/uboot/u-boot/ti-linux-firmware
# Prepare defconfigs with addons echo 'CONFIG_UNIT_TEST=y' > ${B}/unittest.config echo 'CONFIG_CMD_BOOTMENU=y' > ${B}/general.config echo 'CONFIG_CMD_LOG=y' >> ${B}/general.config echo '# CONFIG_CMD_BOOTEFI_SELFTEST is not set' > ${B}/noefist.config echo 'CONFIG_CMD_BOOTEFI_HELLO=y' > ${B}/bootefi.config echo 'CONFIG_CMD_BOOTEFI_SELFTEST=y' >> ${B}/bootefi.config echo 'CONFIG_IPV6=y' > ${B}/net.config echo 'CONFIG_IPV6_ROUTER_DISCOVERY=y' >> ${B}/net.config echo 'CONFIG_CMD_TFTPPUT=y' >> ${B}/net.config echo 'CONFIG_FIT=y' >> ${B}/net.config echo 'CONFIG_FIT_SIGNATURE=y' >> ${B}/net.config echo '# CONFIG_AUTOBOOT_KEYED is not set' > ${B}/noautobootkeyed.config echo 'CONFIG_BOOTSTAGE_STASH_ADDR=0x02400000' > ${B}/bootstage-pi.config echo 'CONFIG_BOOTSTAGE=y' >> ${B}/bootstage-pi.config echo 'CONFIG_BOOTSTAGE_STASH=y' >> ${B}/bootstage-pi.config echo 'CONFIG_CMD_BOOTSTAGE=y' >> ${B}/bootstage-pi.config
## ARM if [ -d /opt/gcc-13.2.0-nolibc/arm-linux-gnueabi ]; then export CROSS_COMPILE=/opt/gcc-13.2.0-nolibc/arm-linux-gnueabi/bin/arm-linux-gnueabi- else echo "Toolchain missing for ARM" exit 1 fi
if [ -z "$NO32BIT" ]; then # Raspberry Pi boards buildaboard rpi_4_32b ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_4_32b --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config fi
if [ -z "$NO64BIT" ]; then ## Build r5 cores for the TI platforms that need both ## Need to backport the kernel fix for cpp with clang and passing the right ## flags. Then need to figure out pip / venv. buildaboard am64x_evm_r5 buildaboard am62x_evm_r5 buildaboard am62x_beagleplay_r5 fi
## AARCH64 if [ -d /opt/gcc-13.2.0-nolibc/aarch64-linux ]; then export CROSS_COMPILE=/opt/gcc-13.2.0-nolibc/aarch64-linux/bin/aarch64-linux- else echo "Toolchain missing for ARM64" exit 1 fi
if [ -z "$NO64BIT" ]; then # Raspberry Pi platforms buildaboard rpi_4 ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_4 --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_arm64 ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_arm64 --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
## These TI platforms require an R5 core and then Axx core ## Need to backport the kernel fix for cpp with clang and passing the right ## flags. export BL31=/home/uboot/u-boot/external-binaries/am64x-u-boot-binaries/bl31.bin export TEE=/home/uboot/u-boot/external-binaries/am64x-u-boot-binaries/tee-raw.bin buildaboard am64x_evm_a53 ${B}/noefist.config ${B}/net.config ${B}/general.config
# The AM62x platforms share BL31/TEE. export BL31=/home/uboot/u-boot/external-binaries/am62x-u-boot-binaries/bl31.bin export TEE=/home/uboot/u-boot/external-binaries/am62x-u-boot-binaries/tee-raw.bin buildaboard am62x_evm_a53 ${B}/noefist.config ${B}/net.config ${B}/general.config buildaboard am62x_beagleplay_a53 ${B}/noautobootkeyed.config ${B}/noefist.config ${B}/net.config fi
echo "Building complete on `git rev-parse --short HEAD`" ------------------------ >8 ------------------------ Which is a little debug-y but more so comments to self. I setup the additional changes to a platform that I want to test with, and then build them. For required additional binaries I point them at known good copies so I don't have to worry if a failure is because of U-Boot or some other project.
Finally we have "uboot-local-ci-outside-docker.sh" which is where pytest is actually run. ------------------------ >8 ------------------------ #!/bin/bash set -e
# Find our arguments while test $# -ne 0; do if [ "$1" == "--no-32bit" ]; then NO32BIT="true" shift elif [ "$1" == "--no-64bit" ]; then NO64BIT="true" shift elif [ "$1" == "--log" ]; then LOG=$2 shift 2 else B=$1 shift fi done
if [ -z "$LOG" ]; then echo "Must pass --log /path/to/log/file" exit 1 fi
# Env export PATH=${PATH}:/home/trini/u-boot/u-boot-test-hooks/bin:/sbin export PYTHONPATH=/home/trini/u-boot/u-boot-test-hooks/py/lootbox:${PYTHONPATH}
# Setup Labgrid variables and use the special virtualenv for now export LG_ENV=/home/trini/u-boot/lg_env.yaml export LG_CROSSBAR=ws://lootbox.k.g:20408/ws export PATH=$PATH:/home/trini/u-boot-test-hooks/bin source /home/trini/labgrid-venv/bin/activate
runtestpy() { BT=$1 shift 1
if [ "$1" == "--clang" ]; then echo "Testing $BT built with clang" shift 1 BD=${B}/${BT} RD=${B}/${BT}_clang_results PD=${B}/${BT}_clang_persistent_data else echo "Testing $BT" BD=${B}/${BT} RD=${B}/${BT}_results PD=${B}/${BT}_persistent_data fi
# If we need special pytest arguments do that here. while getopts "sk:" options; do case $options in k) PA="-k $2" shift 2 ;; esac done
labgrid-client -c $LG_ENV -p $LG_PLACE release >/dev/null 2>&1 || true labgrid-client -c $LG_ENV -p $LG_PLACE acquire ./test/py/test.py -ra --bd $BT --build-dir $BD --result-dir $RD \ --persistent-data-dir $PD --exitfirst $PA >> $LOG 2>&1 labgrid-client -c $LG_ENV -p $LG_PLACE release
}
if [ -z "$NO64BIT" ]; then export LG_PLACE=rpi4 runtestpy rpi_4 runtestpy rpi_arm64 runtestpy rpi_4 --clang runtestpy rpi_arm64 --clang
export LG_PLACE=am64-sk runtestpy am64x_evm_a53
export LG_PLACE=am62-sk runtestpy am62x_evm_a53
export LG_PLACE=beagleplay runtestpy am62x_beagleplay_a53
fi
if [ -z "$NO32BIT" ]; then export LG_PLACE=rpi4 runtestpy rpi_4_32b runtestpy rpi_4_32b --clang fi
echo "Tests complete on `git rev-parse --short HEAD`" ------------------------ >8 ------------------------
I will say, if there's one thing I don't quite like with how my integrations with labgrid work right now it's that acquire/release is done the way it is. This is where I think Simon's adding another hook is the right direction, if there's not some other hook that can be used and I'm just missing.
Hi Tom,
On Thu, 29 Aug 2024 at 13:30, Tom Rini trini@konsulko.com wrote:
Hey all,
So now that I've posted my u-boot-test-hooks for labgrid: https://patchwork.ozlabs.org/project/uboot/patch/20240829185620.3179866-1-tr... the next relevant parts would be how I use that. There's three scripts, first of which is "uboot-hw-testcycle.sh" : ------------------------ >8 ------------------------ #!/bin/bash set -e
# Save output LOG=$HOME/logs/`git describe --abbrev=0`/`date +"%Y-%m-%d-%H%M"`-hw.txt
# Make a base directory B=`mktemp -d` chmod a+rwX ${B}
cleanup() { # Clean up random outputs [ -d test/py/.pytest_cache ] && find -type d -name __pycache__ | xargs rm -r sudo rm -rf ${B} rm -rf test/py/.pytest_cache }
# Cleanups trap cleanup INT trap cleanup ERR trap cleanup EXIT
# Find our arguments while test $# -ne 0; do if [ "$1" == "--no-32bit" ]; then NO32BIT="$1" shift elif [ "$1" == "--no-64bit" ]; then NO64BIT="$1" shift else echo "Unknown argument $1" exit 1 fi done
# Run the build if [ -f .gitlab-ci.yml ]; then IMG=`grep ci_runner_image: .azure-pipelines.yml | cut -d ' ' -f 4` else IMG=`git grep ci_runner_image: origin/master -- .azure-pipelines.yml | cut -d ' ' -f 4` fi
# Build all platforms touch $LOG && chmod a+rw $LOG sudo docker run --rm -it --privileged \ -v /home/trini/bin:/home/uboot/bin \ -v /home/trini/u-boot:/home/uboot/u-boot \ -v /home/trini/logs:/home/uboot/logs \ -v ${B}:${B} \ -w /home/uboot/u-boot/u-boot \ -e PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/uboot/u-boot/u-boot-test-hooks/bin \ $IMG \ /home/uboot/bin/uboot-local-ci-in-docker.sh ${B} $NO32BIT $NO64BIT --log $LOG
# Test all targets /home/trini/bin/uboot-local-ci-outside-docker.sh $B $NO32BIT $NO64BIT --log $LOG
less $LOG ------------------------ >8 ------------------------
Which is still a little debug-y, but is basically just build everything I want in the same container as CI, and then run pytest outside of that container. Could I do it inside the container? Maybe. This has grown from when I used to build everything on the host and used "flashair" SD cards to update my hardware. With labgrid everything might be abstracted enough that yes, really, I can stop doing that. I'll likely look in to that next just to clean things up more.
Second is "uboot-local-ci-in-docker.sh" as invoked above: ------------------------ >8 ------------------------ #!/bin/bash # Build everything we are going to test. set -e
# Let git stop being mad about permissions. git config --global --add safe.directory /home/uboot/u-boot/u-boot
buildaboard() { # Board is first, all fragments follow. BRD=$1 shift 1
# Second would be to use clang if [ "$1" == "--clang" ]; then echo "Building $BRD with clang" | tee -a $LOG OBJ=${B}/${BRD}_clang export HOSTCC=clang-16 export CC=clang-16 shift 1 else echo "Building $BRD" | tee -a $LOG OBJ=${B}/${BRD} fi # All that remains are fragments. FRAGS=$@ make O=${OBJ} ${BRD}_defconfig -s 2>&1 | tee -a $LOG pushd ${OBJ} >/dev/null [ ! -z "${FRAGS}" ] && ${S}/scripts/kconfig/merge_config.sh ${S}/configs/${BRD}_defconfig ${FRAGS} 2>&1 | tee -a $LOG make -sj$(nproc) 2>&1 | tee -a $LOG popd >/dev/null}
# Prepare virtualenv if [ -f test/py/requirements.txt ]; then virtualenv -p /usr/bin/python3 /tmp/venv . /tmp/venv/bin/activate pip install --quiet -r test/py/requirements.txt pip install --quiet -r doc/sphinx/requirements.txt [ -f tools/buildman/requirements.txt ] && pip install --quiet -r tools/buildman/requirements.txt fi
# Find our arguments while test $# -ne 0; do if [ "$1" == "--no-32bit" ]; then NO32BIT="true" shift elif [ "$1" == "--no-64bit" ]; then NO64BIT="true" shift elif [ "$1" == "--log" ]; then LOG=${2/trini/uboot} shift 2 else B=$1 shift fi done
if [ -z "$LOG" ]; then echo "Must pass --log /path/to/log/file" exit 1 fi
S=`pwd`
# Common to all newer TI platforms export BINMAN_INDIRS=/home/uboot/u-boot/ti-linux-firmware
# Prepare defconfigs with addons echo 'CONFIG_UNIT_TEST=y' > ${B}/unittest.config echo 'CONFIG_CMD_BOOTMENU=y' > ${B}/general.config echo 'CONFIG_CMD_LOG=y' >> ${B}/general.config echo '# CONFIG_CMD_BOOTEFI_SELFTEST is not set' > ${B}/noefist.config echo 'CONFIG_CMD_BOOTEFI_HELLO=y' > ${B}/bootefi.config echo 'CONFIG_CMD_BOOTEFI_SELFTEST=y' >> ${B}/bootefi.config echo 'CONFIG_IPV6=y' > ${B}/net.config echo 'CONFIG_IPV6_ROUTER_DISCOVERY=y' >> ${B}/net.config echo 'CONFIG_CMD_TFTPPUT=y' >> ${B}/net.config echo 'CONFIG_FIT=y' >> ${B}/net.config echo 'CONFIG_FIT_SIGNATURE=y' >> ${B}/net.config echo '# CONFIG_AUTOBOOT_KEYED is not set' > ${B}/noautobootkeyed.config echo 'CONFIG_BOOTSTAGE_STASH_ADDR=0x02400000' > ${B}/bootstage-pi.config echo 'CONFIG_BOOTSTAGE=y' >> ${B}/bootstage-pi.config echo 'CONFIG_BOOTSTAGE_STASH=y' >> ${B}/bootstage-pi.config echo 'CONFIG_CMD_BOOTSTAGE=y' >> ${B}/bootstage-pi.config
## ARM if [ -d /opt/gcc-13.2.0-nolibc/arm-linux-gnueabi ]; then export CROSS_COMPILE=/opt/gcc-13.2.0-nolibc/arm-linux-gnueabi/bin/arm-linux-gnueabi- else echo "Toolchain missing for ARM" exit 1 fi
if [ -z "$NO32BIT" ]; then # Raspberry Pi boards buildaboard rpi_4_32b ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_4_32b --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config fi
if [ -z "$NO64BIT" ]; then ## Build r5 cores for the TI platforms that need both ## Need to backport the kernel fix for cpp with clang and passing the right ## flags. Then need to figure out pip / venv. buildaboard am64x_evm_r5 buildaboard am62x_evm_r5 buildaboard am62x_beagleplay_r5 fi
## AARCH64 if [ -d /opt/gcc-13.2.0-nolibc/aarch64-linux ]; then export CROSS_COMPILE=/opt/gcc-13.2.0-nolibc/aarch64-linux/bin/aarch64-linux- else echo "Toolchain missing for ARM64" exit 1 fi
if [ -z "$NO64BIT" ]; then # Raspberry Pi platforms buildaboard rpi_4 ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_4 --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_arm64 ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_arm64 --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
## These TI platforms require an R5 core and then Axx core ## Need to backport the kernel fix for cpp with clang and passing the right ## flags. export BL31=/home/uboot/u-boot/external-binaries/am64x-u-boot-binaries/bl31.bin export TEE=/home/uboot/u-boot/external-binaries/am64x-u-boot-binaries/tee-raw.bin buildaboard am64x_evm_a53 ${B}/noefist.config ${B}/net.config ${B}/general.config # The AM62x platforms share BL31/TEE. export BL31=/home/uboot/u-boot/external-binaries/am62x-u-boot-binaries/bl31.bin export TEE=/home/uboot/u-boot/external-binaries/am62x-u-boot-binaries/tee-raw.bin buildaboard am62x_evm_a53 ${B}/noefist.config ${B}/net.config ${B}/general.config buildaboard am62x_beagleplay_a53 ${B}/noautobootkeyed.config ${B}/noefist.config ${B}/net.configfi
echo "Building complete on `git rev-parse --short HEAD`" ------------------------ >8 ------------------------ Which is a little debug-y but more so comments to self. I setup the additional changes to a platform that I want to test with, and then build them. For required additional binaries I point them at known good copies so I don't have to worry if a failure is because of U-Boot or some other project.
Finally we have "uboot-local-ci-outside-docker.sh" which is where pytest is actually run. ------------------------ >8 ------------------------ #!/bin/bash set -e
# Find our arguments while test $# -ne 0; do if [ "$1" == "--no-32bit" ]; then NO32BIT="true" shift elif [ "$1" == "--no-64bit" ]; then NO64BIT="true" shift elif [ "$1" == "--log" ]; then LOG=$2 shift 2 else B=$1 shift fi done
if [ -z "$LOG" ]; then echo "Must pass --log /path/to/log/file" exit 1 fi
# Env export PATH=${PATH}:/home/trini/u-boot/u-boot-test-hooks/bin:/sbin export PYTHONPATH=/home/trini/u-boot/u-boot-test-hooks/py/lootbox:${PYTHONPATH}
# Setup Labgrid variables and use the special virtualenv for now export LG_ENV=/home/trini/u-boot/lg_env.yaml export LG_CROSSBAR=ws://lootbox.k.g:20408/ws export PATH=$PATH:/home/trini/u-boot-test-hooks/bin source /home/trini/labgrid-venv/bin/activate
runtestpy() { BT=$1 shift 1
if [ "$1" == "--clang" ]; then echo "Testing $BT built with clang" shift 1 BD=${B}/${BT} RD=${B}/${BT}_clang_results PD=${B}/${BT}_clang_persistent_data else echo "Testing $BT" BD=${B}/${BT} RD=${B}/${BT}_results PD=${B}/${BT}_persistent_data fi # If we need special pytest arguments do that here. while getopts "sk:" options; do case $options in k) PA="-k $2" shift 2 ;; esac done labgrid-client -c $LG_ENV -p $LG_PLACE release >/dev/null 2>&1 || true labgrid-client -c $LG_ENV -p $LG_PLACE acquire ./test/py/test.py -ra --bd $BT --build-dir $BD --result-dir $RD \ --persistent-data-dir $PD --exitfirst $PA >> $LOG 2>&1 labgrid-client -c $LG_ENV -p $LG_PLACE release}
if [ -z "$NO64BIT" ]; then export LG_PLACE=rpi4 runtestpy rpi_4 runtestpy rpi_arm64 runtestpy rpi_4 --clang runtestpy rpi_arm64 --clang
export LG_PLACE=am64-sk runtestpy am64x_evm_a53 export LG_PLACE=am62-sk runtestpy am62x_evm_a53 export LG_PLACE=beagleplay runtestpy am62x_beagleplay_a53fi
if [ -z "$NO32BIT" ]; then export LG_PLACE=rpi4 runtestpy rpi_4_32b runtestpy rpi_4_32b --clang fi
echo "Tests complete on `git rev-parse --short HEAD`" ------------------------ >8 ------------------------
I will say, if there's one thing I don't quite like with how my integrations with labgrid work right now it's that acquire/release is done the way it is. This is where I think Simon's adding another hook is the right direction, if there's not some other hook that can be used and I'm just missing.
Thanks for posting this. I know you had described it but this makes a lot of things clearer.
So basically you have set up builds for the different boards and use the hook scripts to write them and do the actual power/reset/console munging.
How do you handle interactive build/run, e.g. to run U-Boot on a particular board quickly? Also how would gitlab work?
Re acquire/release, I added a -a option to auto-acquire the place and release it afterwards. But if the console dies then so does Labgrid so it doesn't release reliably...something to worry about when more progress is made.
Regards, Simon
On Thu, Aug 29, 2024 at 07:06:53PM -0600, Simon Glass wrote:
Hi Tom,
On Thu, 29 Aug 2024 at 13:30, Tom Rini trini@konsulko.com wrote:
Hey all,
So now that I've posted my u-boot-test-hooks for labgrid: https://patchwork.ozlabs.org/project/uboot/patch/20240829185620.3179866-1-tr... the next relevant parts would be how I use that. There's three scripts, first of which is "uboot-hw-testcycle.sh" : ------------------------ >8 ------------------------ #!/bin/bash set -e
# Save output LOG=$HOME/logs/`git describe --abbrev=0`/`date +"%Y-%m-%d-%H%M"`-hw.txt
# Make a base directory B=`mktemp -d` chmod a+rwX ${B}
cleanup() { # Clean up random outputs [ -d test/py/.pytest_cache ] && find -type d -name __pycache__ | xargs rm -r sudo rm -rf ${B} rm -rf test/py/.pytest_cache }
# Cleanups trap cleanup INT trap cleanup ERR trap cleanup EXIT
# Find our arguments while test $# -ne 0; do if [ "$1" == "--no-32bit" ]; then NO32BIT="$1" shift elif [ "$1" == "--no-64bit" ]; then NO64BIT="$1" shift else echo "Unknown argument $1" exit 1 fi done
# Run the build if [ -f .gitlab-ci.yml ]; then IMG=`grep ci_runner_image: .azure-pipelines.yml | cut -d ' ' -f 4` else IMG=`git grep ci_runner_image: origin/master -- .azure-pipelines.yml | cut -d ' ' -f 4` fi
# Build all platforms touch $LOG && chmod a+rw $LOG sudo docker run --rm -it --privileged \ -v /home/trini/bin:/home/uboot/bin \ -v /home/trini/u-boot:/home/uboot/u-boot \ -v /home/trini/logs:/home/uboot/logs \ -v ${B}:${B} \ -w /home/uboot/u-boot/u-boot \ -e PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/uboot/u-boot/u-boot-test-hooks/bin \ $IMG \ /home/uboot/bin/uboot-local-ci-in-docker.sh ${B} $NO32BIT $NO64BIT --log $LOG
# Test all targets /home/trini/bin/uboot-local-ci-outside-docker.sh $B $NO32BIT $NO64BIT --log $LOG
less $LOG ------------------------ >8 ------------------------
Which is still a little debug-y, but is basically just build everything I want in the same container as CI, and then run pytest outside of that container. Could I do it inside the container? Maybe. This has grown from when I used to build everything on the host and used "flashair" SD cards to update my hardware. With labgrid everything might be abstracted enough that yes, really, I can stop doing that. I'll likely look in to that next just to clean things up more.
Second is "uboot-local-ci-in-docker.sh" as invoked above: ------------------------ >8 ------------------------ #!/bin/bash # Build everything we are going to test. set -e
# Let git stop being mad about permissions. git config --global --add safe.directory /home/uboot/u-boot/u-boot
buildaboard() { # Board is first, all fragments follow. BRD=$1 shift 1
# Second would be to use clang if [ "$1" == "--clang" ]; then echo "Building $BRD with clang" | tee -a $LOG OBJ=${B}/${BRD}_clang export HOSTCC=clang-16 export CC=clang-16 shift 1 else echo "Building $BRD" | tee -a $LOG OBJ=${B}/${BRD} fi # All that remains are fragments. FRAGS=$@ make O=${OBJ} ${BRD}_defconfig -s 2>&1 | tee -a $LOG pushd ${OBJ} >/dev/null [ ! -z "${FRAGS}" ] && ${S}/scripts/kconfig/merge_config.sh ${S}/configs/${BRD}_defconfig ${FRAGS} 2>&1 | tee -a $LOG make -sj$(nproc) 2>&1 | tee -a $LOG popd >/dev/null}
# Prepare virtualenv if [ -f test/py/requirements.txt ]; then virtualenv -p /usr/bin/python3 /tmp/venv . /tmp/venv/bin/activate pip install --quiet -r test/py/requirements.txt pip install --quiet -r doc/sphinx/requirements.txt [ -f tools/buildman/requirements.txt ] && pip install --quiet -r tools/buildman/requirements.txt fi
# Find our arguments while test $# -ne 0; do if [ "$1" == "--no-32bit" ]; then NO32BIT="true" shift elif [ "$1" == "--no-64bit" ]; then NO64BIT="true" shift elif [ "$1" == "--log" ]; then LOG=${2/trini/uboot} shift 2 else B=$1 shift fi done
if [ -z "$LOG" ]; then echo "Must pass --log /path/to/log/file" exit 1 fi
S=`pwd`
# Common to all newer TI platforms export BINMAN_INDIRS=/home/uboot/u-boot/ti-linux-firmware
# Prepare defconfigs with addons echo 'CONFIG_UNIT_TEST=y' > ${B}/unittest.config echo 'CONFIG_CMD_BOOTMENU=y' > ${B}/general.config echo 'CONFIG_CMD_LOG=y' >> ${B}/general.config echo '# CONFIG_CMD_BOOTEFI_SELFTEST is not set' > ${B}/noefist.config echo 'CONFIG_CMD_BOOTEFI_HELLO=y' > ${B}/bootefi.config echo 'CONFIG_CMD_BOOTEFI_SELFTEST=y' >> ${B}/bootefi.config echo 'CONFIG_IPV6=y' > ${B}/net.config echo 'CONFIG_IPV6_ROUTER_DISCOVERY=y' >> ${B}/net.config echo 'CONFIG_CMD_TFTPPUT=y' >> ${B}/net.config echo 'CONFIG_FIT=y' >> ${B}/net.config echo 'CONFIG_FIT_SIGNATURE=y' >> ${B}/net.config echo '# CONFIG_AUTOBOOT_KEYED is not set' > ${B}/noautobootkeyed.config echo 'CONFIG_BOOTSTAGE_STASH_ADDR=0x02400000' > ${B}/bootstage-pi.config echo 'CONFIG_BOOTSTAGE=y' >> ${B}/bootstage-pi.config echo 'CONFIG_BOOTSTAGE_STASH=y' >> ${B}/bootstage-pi.config echo 'CONFIG_CMD_BOOTSTAGE=y' >> ${B}/bootstage-pi.config
## ARM if [ -d /opt/gcc-13.2.0-nolibc/arm-linux-gnueabi ]; then export CROSS_COMPILE=/opt/gcc-13.2.0-nolibc/arm-linux-gnueabi/bin/arm-linux-gnueabi- else echo "Toolchain missing for ARM" exit 1 fi
if [ -z "$NO32BIT" ]; then # Raspberry Pi boards buildaboard rpi_4_32b ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_4_32b --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config fi
if [ -z "$NO64BIT" ]; then ## Build r5 cores for the TI platforms that need both ## Need to backport the kernel fix for cpp with clang and passing the right ## flags. Then need to figure out pip / venv. buildaboard am64x_evm_r5 buildaboard am62x_evm_r5 buildaboard am62x_beagleplay_r5 fi
## AARCH64 if [ -d /opt/gcc-13.2.0-nolibc/aarch64-linux ]; then export CROSS_COMPILE=/opt/gcc-13.2.0-nolibc/aarch64-linux/bin/aarch64-linux- else echo "Toolchain missing for ARM64" exit 1 fi
if [ -z "$NO64BIT" ]; then # Raspberry Pi platforms buildaboard rpi_4 ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_4 --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_arm64 ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config buildaboard rpi_arm64 --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
## These TI platforms require an R5 core and then Axx core ## Need to backport the kernel fix for cpp with clang and passing the right ## flags. export BL31=/home/uboot/u-boot/external-binaries/am64x-u-boot-binaries/bl31.bin export TEE=/home/uboot/u-boot/external-binaries/am64x-u-boot-binaries/tee-raw.bin buildaboard am64x_evm_a53 ${B}/noefist.config ${B}/net.config ${B}/general.config # The AM62x platforms share BL31/TEE. export BL31=/home/uboot/u-boot/external-binaries/am62x-u-boot-binaries/bl31.bin export TEE=/home/uboot/u-boot/external-binaries/am62x-u-boot-binaries/tee-raw.bin buildaboard am62x_evm_a53 ${B}/noefist.config ${B}/net.config ${B}/general.config buildaboard am62x_beagleplay_a53 ${B}/noautobootkeyed.config ${B}/noefist.config ${B}/net.configfi
echo "Building complete on `git rev-parse --short HEAD`" ------------------------ >8 ------------------------ Which is a little debug-y but more so comments to self. I setup the additional changes to a platform that I want to test with, and then build them. For required additional binaries I point them at known good copies so I don't have to worry if a failure is because of U-Boot or some other project.
Finally we have "uboot-local-ci-outside-docker.sh" which is where pytest is actually run. ------------------------ >8 ------------------------ #!/bin/bash set -e
# Find our arguments while test $# -ne 0; do if [ "$1" == "--no-32bit" ]; then NO32BIT="true" shift elif [ "$1" == "--no-64bit" ]; then NO64BIT="true" shift elif [ "$1" == "--log" ]; then LOG=$2 shift 2 else B=$1 shift fi done
if [ -z "$LOG" ]; then echo "Must pass --log /path/to/log/file" exit 1 fi
# Env export PATH=${PATH}:/home/trini/u-boot/u-boot-test-hooks/bin:/sbin export PYTHONPATH=/home/trini/u-boot/u-boot-test-hooks/py/lootbox:${PYTHONPATH}
# Setup Labgrid variables and use the special virtualenv for now export LG_ENV=/home/trini/u-boot/lg_env.yaml export LG_CROSSBAR=ws://lootbox.k.g:20408/ws export PATH=$PATH:/home/trini/u-boot-test-hooks/bin source /home/trini/labgrid-venv/bin/activate
runtestpy() { BT=$1 shift 1
if [ "$1" == "--clang" ]; then echo "Testing $BT built with clang" shift 1 BD=${B}/${BT} RD=${B}/${BT}_clang_results PD=${B}/${BT}_clang_persistent_data else echo "Testing $BT" BD=${B}/${BT} RD=${B}/${BT}_results PD=${B}/${BT}_persistent_data fi # If we need special pytest arguments do that here. while getopts "sk:" options; do case $options in k) PA="-k $2" shift 2 ;; esac done labgrid-client -c $LG_ENV -p $LG_PLACE release >/dev/null 2>&1 || true labgrid-client -c $LG_ENV -p $LG_PLACE acquire ./test/py/test.py -ra --bd $BT --build-dir $BD --result-dir $RD \ --persistent-data-dir $PD --exitfirst $PA >> $LOG 2>&1 labgrid-client -c $LG_ENV -p $LG_PLACE release}
if [ -z "$NO64BIT" ]; then export LG_PLACE=rpi4 runtestpy rpi_4 runtestpy rpi_arm64 runtestpy rpi_4 --clang runtestpy rpi_arm64 --clang
export LG_PLACE=am64-sk runtestpy am64x_evm_a53 export LG_PLACE=am62-sk runtestpy am62x_evm_a53 export LG_PLACE=beagleplay runtestpy am62x_beagleplay_a53fi
if [ -z "$NO32BIT" ]; then export LG_PLACE=rpi4 runtestpy rpi_4_32b runtestpy rpi_4_32b --clang fi
echo "Tests complete on `git rev-parse --short HEAD`" ------------------------ >8 ------------------------
I will say, if there's one thing I don't quite like with how my integrations with labgrid work right now it's that acquire/release is done the way it is. This is where I think Simon's adding another hook is the right direction, if there's not some other hook that can be used and I'm just missing.
Thanks for posting this. I know you had described it but this makes a lot of things clearer.
So basically you have set up builds for the different boards and use the hook scripts to write them and do the actual power/reset/console munging.
Yes.
How do you handle interactive build/run, e.g. to run U-Boot on a particular board quickly?
Well, this is the CI lab, so that's not the primary point. But control-O in a shell? Some platforms I have a wrapper for, some I don't.
Also how would gitlab work?
Similar to how it works in your series, but with more "make the container build/include firmware blobs for use", most likely.
Re acquire/release, I added a -a option to auto-acquire the place and release it afterwards. But if the console dies then so does Labgrid so it doesn't release reliably...something to worry about when more progress is made.
Well, that's what pre/post steps are for, IMHO.
Hi Tom,
On Fri, 30 Aug 2024 at 08:30, Tom Rini trini@konsulko.com wrote:
On Thu, Aug 29, 2024 at 07:06:53PM -0600, Simon Glass wrote:
Hi Tom,
On Thu, 29 Aug 2024 at 13:30, Tom Rini trini@konsulko.com wrote:
Hey all,
So now that I've posted my u-boot-test-hooks for labgrid: https://patchwork.ozlabs.org/project/uboot/patch/20240829185620.3179866-1-tr... the next relevant parts would be how I use that. There's three scripts, first of which is "uboot-hw-testcycle.sh" : ------------------------ >8 ------------------------
[..]
------------------------ >8 ------------------------
I will say, if there's one thing I don't quite like with how my integrations with labgrid work right now it's that acquire/release is done the way it is. This is where I think Simon's adding another hook is the right direction, if there's not some other hook that can be used and I'm just missing.
Thanks for posting this. I know you had described it but this makes a lot of things clearer.
So basically you have set up builds for the different boards and use the hook scripts to write them and do the actual power/reset/console munging.
Yes.
How do you handle interactive build/run, e.g. to run U-Boot on a particular board quickly?
Well, this is the CI lab, so that's not the primary point. But control-O in a shell? Some platforms I have a wrapper for, some I don't.
OK. In my case I often want to try things out on particular boards, or at least that is what I have found. Perhaps I will do that less once gitlab is running. Anyway, my point is they use the same mechanism.
Also how would gitlab work?
Similar to how it works in your series, but with more "make the container build/include firmware blobs for use", most likely.
Yes, it should be similar.
Re acquire/release, I added a -a option to auto-acquire the place and release it afterwards. But if the console dies then so does Labgrid so it doesn't release reliably...something to worry about when more progress is made.
Well, that's what pre/post steps are for, IMHO.
Yes that's why I added it. I think it is best to keep that, at least until we find it never goes wrong.
Regards, Simon
participants (2)
-
Simon Glass -
Tom Rini