My labgrid setup

29 Aug 2024


      Hey all,
So now that I've posted my u-boot-test-hooks for labgrid:
https://patchwork.ozlabs.org/project/uboot/patch/20240829185620.3179866-1-tr...
the next relevant parts would be how I use that. There's three scripts,
first of which is "uboot-hw-testcycle.sh" :
------------------------ >8 ------------------------
#!/bin/bash
set -e
# Save output
LOG=$HOME/logs/`git describe --abbrev=0`/`date +"%Y-%m-%d-%H%M"`-hw.txt
# Make a base directory
B=`mktemp -d`
chmod a+rwX ${B}
cleanup() {
    # Clean up random outputs
    [ -d test/py/.pytest_cache ] && find -type d -name __pycache__ | xargs rm -r
    sudo rm -rf ${B}
    rm -rf test/py/.pytest_cache
}
# Cleanups
trap cleanup INT
trap cleanup ERR
trap cleanup EXIT
# Find our arguments
while test $# -ne 0; do
    if [ "$1" == "--no-32bit" ]; then
    	NO32BIT="$1"
    	shift
    elif [ "$1" == "--no-64bit" ]; then
    	NO64BIT="$1"
    	shift
    else
    	echo "Unknown argument $1"
    	exit 1
    fi
done
# Run the build
if [ -f .gitlab-ci.yml ]; then
    IMG=`grep ci_runner_image: .azure-pipelines.yml | cut -d ' ' -f 4`
else
    IMG=`git grep ci_runner_image: origin/master -- .azure-pipelines.yml | cut -d ' ' -f 4`
fi
# Build all platforms
touch $LOG && chmod a+rw $LOG
sudo docker run  --rm -it --privileged \
    -v /home/trini/bin:/home/uboot/bin \
    -v /home/trini/u-boot:/home/uboot/u-boot \
    -v /home/trini/logs:/home/uboot/logs \
    -v ${B}:${B} \
    -w /home/uboot/u-boot/u-boot \
    -e PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/uboot/u-boot/u-boot-test-hooks/bin \
    $IMG \
    /home/uboot/bin/uboot-local-ci-in-docker.sh ${B} $NO32BIT $NO64BIT --log $LOG
# Test all targets
/home/trini/bin/uboot-local-ci-outside-docker.sh $B $NO32BIT $NO64BIT --log $LOG
less $LOG
------------------------ >8 ------------------------
Which is still a little debug-y, but is basically just build everything
I want in the same container as CI, and then run pytest outside of that
container. Could I do it inside the container? Maybe. This has grown
from when I used to build everything on the host and used "flashair" SD
cards to update my hardware. With labgrid everything might be abstracted
enough that yes, really, I can stop doing that. I'll likely look in to
that next just to clean things up more.
Second is "uboot-local-ci-in-docker.sh" as invoked above:
------------------------ >8 ------------------------
#!/bin/bash
# Build everything we are going to test.
set -e
# Let git stop being mad about permissions.
git config --global --add safe.directory /home/uboot/u-boot/u-boot
buildaboard() {
    # Board is first, all fragments follow.
    BRD=$1
    shift 1
# Second would be to use clang
    if [ "$1" == "--clang" ]; then
    	echo "Building $BRD with clang" | tee -a $LOG
    	OBJ=${B}/${BRD}_clang
    	export HOSTCC=clang-16
    	export CC=clang-16
    	shift 1
    else
    	echo "Building $BRD" | tee -a $LOG
    	OBJ=${B}/${BRD}
    fi
    # All that remains are fragments.
    FRAGS=$@
make O=${OBJ} ${BRD}_defconfig -s 2>&1 | tee -a $LOG
    pushd ${OBJ} >/dev/null
    [ ! -z "${FRAGS}" ] && ${S}/scripts/kconfig/merge_config.sh ${S}/configs/${BRD}_defconfig ${FRAGS} 2>&1 | tee -a $LOG
    make -sj$(nproc) 2>&1 | tee -a $LOG
    popd >/dev/null
}
# Prepare virtualenv
if [ -f test/py/requirements.txt ]; then
    virtualenv -p /usr/bin/python3 /tmp/venv
    . /tmp/venv/bin/activate
    pip install --quiet -r test/py/requirements.txt
    pip install --quiet -r doc/sphinx/requirements.txt
    [ -f tools/buildman/requirements.txt ] && pip install --quiet -r tools/buildman/requirements.txt
fi
# Find our arguments
while test $# -ne 0; do
    if [ "$1" == "--no-32bit" ]; then
    	NO32BIT="true"
    	shift
    elif [ "$1" == "--no-64bit" ]; then
    	NO64BIT="true"
    	shift
    elif [ "$1" == "--log" ]; then
    	LOG=${2/trini/uboot}
    	shift 2
    else
    	B=$1
    	shift
    fi
done
if [ -z "$LOG" ]; then
    echo "Must pass --log /path/to/log/file"
    exit 1
fi
S=`pwd`
# Common to all newer TI platforms
export BINMAN_INDIRS=/home/uboot/u-boot/ti-linux-firmware
# Prepare defconfigs with addons
echo 'CONFIG_UNIT_TEST=y' > ${B}/unittest.config
echo 'CONFIG_CMD_BOOTMENU=y' > ${B}/general.config
echo 'CONFIG_CMD_LOG=y' >> ${B}/general.config
echo '# CONFIG_CMD_BOOTEFI_SELFTEST is not set' > ${B}/noefist.config
echo 'CONFIG_CMD_BOOTEFI_HELLO=y' > ${B}/bootefi.config
echo 'CONFIG_CMD_BOOTEFI_SELFTEST=y' >> ${B}/bootefi.config
echo 'CONFIG_IPV6=y' > ${B}/net.config
echo 'CONFIG_IPV6_ROUTER_DISCOVERY=y' >> ${B}/net.config
echo 'CONFIG_CMD_TFTPPUT=y' >> ${B}/net.config
echo 'CONFIG_FIT=y' >> ${B}/net.config
echo 'CONFIG_FIT_SIGNATURE=y' >> ${B}/net.config
echo '# CONFIG_AUTOBOOT_KEYED is not set' > ${B}/noautobootkeyed.config
echo 'CONFIG_BOOTSTAGE_STASH_ADDR=0x02400000' > ${B}/bootstage-pi.config
echo 'CONFIG_BOOTSTAGE=y' >> ${B}/bootstage-pi.config
echo 'CONFIG_BOOTSTAGE_STASH=y' >> ${B}/bootstage-pi.config
echo 'CONFIG_CMD_BOOTSTAGE=y' >> ${B}/bootstage-pi.config
## ARM
if [ -d /opt/gcc-13.2.0-nolibc/arm-linux-gnueabi ]; then
    export CROSS_COMPILE=/opt/gcc-13.2.0-nolibc/arm-linux-gnueabi/bin/arm-linux-gnueabi-
else
    echo "Toolchain missing for ARM"
    exit 1
fi
if [ -z "$NO32BIT" ]; then
    # Raspberry Pi boards
    buildaboard rpi_4_32b ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
    buildaboard rpi_4_32b --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
fi
if [ -z "$NO64BIT" ]; then
    ## Build r5 cores for the TI platforms that need both
    ## Need to backport the kernel fix for cpp with clang and passing the right
    ## flags.  Then need to figure out pip / venv.
    buildaboard am64x_evm_r5
    buildaboard am62x_evm_r5
    buildaboard am62x_beagleplay_r5
fi
## AARCH64
if [ -d /opt/gcc-13.2.0-nolibc/aarch64-linux ]; then
    export CROSS_COMPILE=/opt/gcc-13.2.0-nolibc/aarch64-linux/bin/aarch64-linux-
else
    echo "Toolchain missing for ARM64"
    exit 1
fi
if [ -z "$NO64BIT" ]; then
    # Raspberry Pi platforms
    buildaboard rpi_4 ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
    buildaboard rpi_4 --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
    buildaboard rpi_arm64 ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
    buildaboard rpi_arm64 --clang ${B}/bootefi.config ${B}/net.config ${B}/general.config ${B}/bootstage-pi.config
## These TI platforms require an R5 core and then Axx core
    ## Need to backport the kernel fix for cpp with clang and passing the right
    ## flags.
    export BL31=/home/uboot/u-boot/external-binaries/am64x-u-boot-binaries/bl31.bin
    export TEE=/home/uboot/u-boot/external-binaries/am64x-u-boot-binaries/tee-raw.bin
    buildaboard am64x_evm_a53 ${B}/noefist.config ${B}/net.config ${B}/general.config
# The AM62x platforms share BL31/TEE.
    export BL31=/home/uboot/u-boot/external-binaries/am62x-u-boot-binaries/bl31.bin
    export TEE=/home/uboot/u-boot/external-binaries/am62x-u-boot-binaries/tee-raw.bin
    buildaboard am62x_evm_a53 ${B}/noefist.config ${B}/net.config ${B}/general.config
    buildaboard am62x_beagleplay_a53 ${B}/noautobootkeyed.config ${B}/noefist.config ${B}/net.config
fi
echo "Building complete on `git rev-parse --short HEAD`"
------------------------ >8 ------------------------
Which is a little debug-y but more so comments to self. I setup the
additional changes to a platform that I want to test with, and then
build them. For required additional binaries I point them at known good
copies so I don't have to worry if a failure is because of U-Boot or
some other project.
Finally we have "uboot-local-ci-outside-docker.sh" which is where pytest
is actually run.
------------------------ >8 ------------------------
#!/bin/bash
set -e
# Find our arguments
while test $# -ne 0; do
    if [ "$1" == "--no-32bit" ]; then
    	NO32BIT="true"
    	shift
    elif [ "$1" == "--no-64bit" ]; then
    	NO64BIT="true"
    	shift
    elif [ "$1" == "--log" ]; then
    	LOG=$2
    	shift 2
    else
    	B=$1
    	shift
    fi
done
if [ -z "$LOG" ]; then
    echo "Must pass --log /path/to/log/file"
    exit 1
fi
# Env
export PATH=${PATH}:/home/trini/u-boot/u-boot-test-hooks/bin:/sbin
export PYTHONPATH=/home/trini/u-boot/u-boot-test-hooks/py/lootbox:${PYTHONPATH}
# Setup Labgrid variables and use the special virtualenv for now
export LG_ENV=/home/trini/u-boot/lg_env.yaml
export LG_CROSSBAR=ws://lootbox.k.g:20408/ws
export PATH=$PATH:/home/trini/u-boot-test-hooks/bin
source /home/trini/labgrid-venv/bin/activate
runtestpy() {
    BT=$1
    shift 1
if [ "$1" == "--clang" ]; then
    	echo "Testing $BT built with clang"
    	shift 1
    	BD=${B}/${BT}
    	RD=${B}/${BT}_clang_results
    	PD=${B}/${BT}_clang_persistent_data
    else
    	echo "Testing $BT"
    	BD=${B}/${BT}
    	RD=${B}/${BT}_results
    	PD=${B}/${BT}_persistent_data
    fi
# If we need special pytest arguments do that here.
    while getopts "sk:" options; do
    	case $options in
    		k)
    			PA="-k $2"
    			shift 2
    			;;
    	esac
    done
labgrid-client -c $LG_ENV -p $LG_PLACE release >/dev/null 2>&1 || true
    labgrid-client -c $LG_ENV -p $LG_PLACE acquire
    ./test/py/test.py -ra --bd $BT --build-dir $BD --result-dir $RD \
    	--persistent-data-dir $PD --exitfirst $PA >> $LOG 2>&1
    labgrid-client -c $LG_ENV -p $LG_PLACE release
}
if [ -z "$NO64BIT" ]; then
    export LG_PLACE=rpi4
    runtestpy rpi_4
    runtestpy rpi_arm64
    runtestpy rpi_4 --clang
    runtestpy rpi_arm64 --clang
export LG_PLACE=am64-sk
    runtestpy am64x_evm_a53
export LG_PLACE=am62-sk
    runtestpy am62x_evm_a53
export LG_PLACE=beagleplay
    runtestpy am62x_beagleplay_a53
fi
if [ -z "$NO32BIT" ]; then
    export LG_PLACE=rpi4
    runtestpy rpi_4_32b
    runtestpy rpi_4_32b --clang
fi
echo "Tests complete on `git rev-parse --short HEAD`"
------------------------ >8 ------------------------
I will say, if there's one thing I don't quite like with how my
integrations with labgrid work right now it's that acquire/release is
done the way it is. This is where I think Simon's adding another hook is
the right direction, if there's not some other hook that can be used and
I'm just missing.
-- 
Tom

My labgrid setup

Tom Rini