[PATCH 0/4] Fix CIs skipping filesystem, EFI secure boot and EFI capsule tests
After my previous patch to fix filesystem tests [1] was merged, I noticed the GitLab CI was still skipping them and wanted to figure out why. In short: libguestfs tools (virt-make-fs, guestmount) fail because they need an installed kernel and the host /dev/fuse device, loop mounts need the host /dev/loop* devices, and mounting filesystems (loop and guestmount) fails because Docker containers need extra permissions to mount devices normally disabled for host security.
[1] https://patchwork.ozlabs.org/project/uboot/patch/20210520190947.21773-1-alpe...
Patch #1 is meant to install a kernel into the container image that libguestfs can use, but the image will need to be regenerated manually. The need to regenerate it can be postponed with patch #4.
Patch #2 makes virt-make-fs work, which should make these EFI tests run again. But guestmount doesn't work with this much because it needs more permissions to actually mount a filesystem.
Patch #3 makes mounting filesystems and loop devices work, which should make the filesystem tests run again. This is separate from patch #2 because the parts using guestmount can theoretically be rewritten to use guestfish which would make the filesystem tests work without this patch, and giving mount permissions to the container processes seem to be insecure. So, this patch can be dropped if you think the impact isn't worth it.
Patch #4 is actually for me to test the effects of patch #1 easier on Azure (via U-Boot GitHub repo) and locally with 'gitlab-runner exec docker'. It can be dropped if/when the container image is regenerated with patch #1 applied.
I've pushed this as a GitHub pull request [2] (along with some other fixes to the filesystem test setup I'll send as patches shortly), so it would run on Azure and can be compared to the current master branch.
Before this series the "test.py sandbox" job results in [3]:
[...] SKIPPED [3] /u/test/py/tests/test_efi_capsule/conftest.py:68: Setup failed: virt-make-fs --partition=gpt --size=+1M --type=vfat /tmp/sandbox/persistent-data/test_efi_capsule /tmp/sandbox/persistent-data/test_efi_capsule.img SKIPPED [14] /u/test/py/tests/test_efi_secboot/conftest.py:119: Setup failed: virt-make-fs --partition=gpt --size=+1M --type=vfat /tmp/sandbox/mnt_efisecure /tmp/sandbox/persistent-data/test_efi_secboot.img SKIPPED [3] /u/test/py/tests/test_efi_secboot/conftest.py:235: Setup failed: virt-make-fs --partition=gpt --size=+1M --type=vfat /tmp/sandbox/persistent-data/mnt_efi_secboot_intca /tmp/sandbox/persistent-data/test_efi_secboot_intca.img SKIPPED [13] /u/test/py/tests/test_fs/conftest.py:289: Mounting to folder failed for filesystem: fat16. Command 'guestmount -a /tmp/sandbox/persistent-data/3GB.fat16.img -m /dev/sda /tmp/sandbox/persistent-data/mnt' returned non-zero exit status 1. SKIPPED [13] /u/test/py/tests/test_fs/conftest.py:289: Mounting to folder failed for filesystem: fat32. Command 'guestmount -a /tmp/sandbox/persistent-data/3GB.fat32.img -m /dev/sda /tmp/sandbox/persistent-data/mnt' returned non-zero exit status 1. SKIPPED [13] /u/test/py/tests/test_fs/conftest.py:289: Mounting to folder failed for filesystem: ext4. Command 'guestmount -a /tmp/sandbox/persistent-data/3GB.ext4.img -m /dev/sda /tmp/sandbox/persistent-data/mnt' returned non-zero exit status 1. SKIPPED [11] /u/test/py/tests/test_fs/conftest.py:411: Mounting to folder failed for filesystem: fat16. Command 'guestmount -a /tmp/sandbox/persistent-data/128MB.fat16.img -m /dev/sda /tmp/sandbox/persistent-data/mnt' returned non-zero exit status 1. SKIPPED [11] /u/test/py/tests/test_fs/conftest.py:411: Mounting to folder failed for filesystem: fat32. Command 'guestmount -a /tmp/sandbox/persistent-data/128MB.fat32.img -m /dev/sda /tmp/sandbox/persistent-data/mnt' returned non-zero exit status 1. SKIPPED [4] /u/test/py/tests/test_fs/conftest.py:623: Mounting to folder failed for filesystem: ext4. Command 'guestmount -a /tmp/sandbox/persistent-data/1GB.ext4.img -m /dev/sda /tmp/sandbox/persistent-data/mnt' returned non-zero exit status 1. SKIPPED [7] /u/test/py/tests/test_fs/conftest.py:540: Mounting to folder failed for filesystem: fat16. Command 'guestmount -a /tmp/sandbox/persistent-data/128MB.fat16.img -m /dev/sda /tmp/sandbox/persistent-data/mnt' returned non-zero exit status 1. SKIPPED [7] /u/test/py/tests/test_fs/conftest.py:540: Mounting to folder failed for filesystem: fat32. Command 'guestmount -a /tmp/sandbox/persistent-data/128MB.fat32.img -m /dev/sda /tmp/sandbox/persistent-data/mnt' returned non-zero exit status 1. =========== 650 passed, 162 skipped, 2 warnings in 117.68s (0:01:57) ===========
After this series the tests can be run [4], with the other fixes making all the quoted tests above succeed:
[...] =========== 749 passed, 63 skipped, 2 warnings in 353.09s (0:05:53) ============
[2] https://github.com/u-boot/u-boot/pull/78 [3] https://dev.azure.com/u-boot/u-boot/_build/results?buildId=2322&view=log... [4] https://dev.azure.com/u-boot/u-boot/_build/results?buildId=2333&view=log...
This also exposes the following failure in the "test.py sandbox_clang" job [5], but the CIs have been skipping that test so far:
=> => setenv -e -nv -bs -rt -at -i 4000000:$filesize dbx No EFI system partition Failed to persist EFI variables => => printenv -e -n -guid d719b2cb-3d3a-4596-a3bc-dad00e67656f dbx Error: "dbx" not defined
[...] FAILED test/py/tests/test_efi_secboot/test_authvar.py::TestEfiAuthVar::test_efi_var_auth1 ====== 1 failed, 748 passed, 63 skipped, 3 warnings in 309.82s (0:05:09) =======
I can reproduce the failure locally on the current master branch with the following commands, so it's not due to this series:
$ tools/buildman/buildman -O clang -o build-sandbox -w --boards="sandbox" $ test/py/test.py --bd sandbox -k test_authvar.py
[5] https://dev.azure.com/u-boot/u-boot/_build/results?buildId=2333&view=log...
Similar changes to patch #2, #3 should be applicable to the GitLab CI and probably necessary to get the same effect, but I don't think it's configuration is accessible to me.
Alper Nebi Yasak (4): tools: docker: Install a readable kernel for libguestfs-tools Azure: Add fuse device for sandbox test.py tests Azure: Add loop devices and CAP_SYS_ADMIN for sandbox test.py tests Azure/GitLab: Install a readable kernel for libguestfs-tools
.azure-pipelines.yml | 25 ++++++++++++++++++++++++- .gitlab-ci.yml | 5 +++++ tools/docker/Dockerfile | 4 ++++ 3 files changed, 33 insertions(+), 1 deletion(-)
The filesystem and EFI (capsule and secure boot) test setups try to use guestmount and virt-make-fs respectively to prepare disk images to run tests on. However, these libguestfs tools need a kernel image and fail with the following message (revealed in debug/trace mode) if it can't find one:
supermin: failed to find a suitable kernel (host_cpu=x86_64).
I looked for kernels in /boot and modules in /lib/modules.
If this is a Xen guest, and you only have Xen domU kernels installed, try installing a fullvirt kernel (only for supermin use, you shouldn't boot the Xen guest with it).
This failure then causes these tests to be skipped in CIs. Install a kernel package in the Docker containers so the CIs can run these tests with libguestfs tools again (assuming the container is run with necessary host devices and privileges). As this kernel would be only used for virtualization, we can use the kernel package specialized for that. On Ubuntu systems kernel images are not readable by non-root users, so explicitly add read permissions with chmod as well.
Signed-off-by: Alper Nebi Yasak alpernebiyasak@gmail.com ---
tools/docker/Dockerfile | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index d2f0074ee8a6..563b16639e54 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -71,6 +71,7 @@ RUN apt-get update && apt-get install -y \ libssl-dev \ libudev-dev \ libusb-1.0-0-dev \ + linux-image-kvm \ lzma-alone \ lzop \ mount \ @@ -99,6 +100,9 @@ RUN apt-get update && apt-get install -y \ zip \ && rm -rf /var/lib/apt/lists/*
+# Make kernels readable for libguestfs tools to work correctly +RUN chmod +r /boot/vmlinu* /lib/modules/*/vmlinu* || true + # Manually install libmpfr4 for the toolchains RUN wget http://mirrors.kernel.org/ubuntu/pool/main/m/mpfr4/libmpfr4_3.1.4-1_amd64.de... && dpkg -i libmpfr4_3.1.4-1_amd64.deb && rm libmpfr4_3.1.4-1_amd64.deb
On 6/4/21 9:02 PM, Alper Nebi Yasak wrote:
The filesystem and EFI (capsule and secure boot) test setups try to use guestmount and virt-make-fs respectively to prepare disk images to run tests on. However, these libguestfs tools need a kernel image and fail with the following message (revealed in debug/trace mode) if it can't find one:
supermin: failed to find a suitable kernel (host_cpu=x86_64). I looked for kernels in /boot and modules in /lib/modules. If this is a Xen guest, and you only have Xen domU kernels installed, try installing a fullvirt kernel (only for supermin use, you shouldn't boot the Xen guest with it).This failure then causes these tests to be skipped in CIs. Install a kernel package in the Docker containers so the CIs can run these tests with libguestfs tools again (assuming the container is run with necessary host devices and privileges). As this kernel would be only used for virtualization, we can use the kernel package specialized for that. On Ubuntu systems kernel images are not readable by non-root users, so explicitly add read permissions with chmod as well.
Signed-off-by: Alper Nebi Yasak alpernebiyasak@gmail.com
Acked-by: Heinrich Schuchardt xypron.glpk@gmx.de
tools/docker/Dockerfile | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index d2f0074ee8a6..563b16639e54 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -71,6 +71,7 @@ RUN apt-get update && apt-get install -y \ libssl-dev \ libudev-dev \ libusb-1.0-0-dev \
- linux-image-kvm \ lzma-alone \ lzop \ mount \
@@ -99,6 +100,9 @@ RUN apt-get update && apt-get install -y \ zip \ && rm -rf /var/lib/apt/lists/*
+# Make kernels readable for libguestfs tools to work correctly +RUN chmod +r /boot/vmlinu* /lib/modules/*/vmlinu* || true
- # Manually install libmpfr4 for the toolchains RUN wget http://mirrors.kernel.org/ubuntu/pool/main/m/mpfr4/libmpfr4_3.1.4-1_amd64.de... && dpkg -i libmpfr4_3.1.4-1_amd64.deb && rm libmpfr4_3.1.4-1_amd64.deb
On Fri, Jun 04, 2021 at 10:02:03PM +0300, Alper Nebi Yasak wrote:
The filesystem and EFI (capsule and secure boot) test setups try to use guestmount and virt-make-fs respectively to prepare disk images to run tests on. However, these libguestfs tools need a kernel image and fail with the following message (revealed in debug/trace mode) if it can't find one:
supermin: failed to find a suitable kernel (host_cpu=x86_64). I looked for kernels in /boot and modules in /lib/modules. If this is a Xen guest, and you only have Xen domU kernels installed, try installing a fullvirt kernel (only for supermin use, you shouldn't boot the Xen guest with it).This failure then causes these tests to be skipped in CIs. Install a kernel package in the Docker containers so the CIs can run these tests with libguestfs tools again (assuming the container is run with necessary host devices and privileges). As this kernel would be only used for virtualization, we can use the kernel package specialized for that. On Ubuntu systems kernel images are not readable by non-root users, so explicitly add read permissions with chmod as well.
I thought that I had added linux-image-kvm to Travis CI (and hence other CI loops as well) as part of this patch[1] to fix some test problem. It seems that it has never been merged, though.
Thank you for the reminder.
-Takahiro Akashi
[1] https://lists.denx.de/pipermail/u-boot/2020-July/421810.html
Signed-off-by: Alper Nebi Yasak alpernebiyasak@gmail.com
tools/docker/Dockerfile | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index d2f0074ee8a6..563b16639e54 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -71,6 +71,7 @@ RUN apt-get update && apt-get install -y \ libssl-dev \ libudev-dev \ libusb-1.0-0-dev \
- linux-image-kvm \ lzma-alone \ lzop \ mount \
@@ -99,6 +100,9 @@ RUN apt-get update && apt-get install -y \ zip \ && rm -rf /var/lib/apt/lists/*
+# Make kernels readable for libguestfs tools to work correctly +RUN chmod +r /boot/vmlinu* /lib/modules/*/vmlinu* || true
# Manually install libmpfr4 for the toolchains RUN wget http://mirrors.kernel.org/ubuntu/pool/main/m/mpfr4/libmpfr4_3.1.4-1_amd64.de... && dpkg -i libmpfr4_3.1.4-1_amd64.deb && rm libmpfr4_3.1.4-1_amd64.deb
-- 2.32.0.rc2
The EFI secure boot and capsule test setups need to prepare disk images for their tests using virt-make-fs, which requires access to the host fuse device. This is not exposed to the docker container by default and has to be added explicitly.
Since these tests are marked to run only on the sandbox board, add the fuse device only when testing on sandbox.
Signed-off-by: Alper Nebi Yasak alpernebiyasak@gmail.com ---
.azure-pipelines.yml | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index 35ab7f30b276..a4d796c41895 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -318,7 +318,15 @@ jobs: # as sandbox testing need create files like spi flash images, etc. # (TODO: clean up this in the future) chmod 777 . - docker run -v $PWD:$(work_dir) $(ci_runner_image) /bin/bash $(work_dir)/test.sh + # Some EFI tests need extra docker args to run + set -- + if [[ "${TEST_PY_BD}" == "sandbox" ]]; then + # virt-make-fs needs the fuse device + if modprobe fuse; then + set -- "$@" --device /dev/fuse:/dev/fuse + fi + fi + docker run "$@" -v $PWD:$(work_dir) $(ci_runner_image) /bin/bash $(work_dir)/test.sh
- job: build_the_world displayName: 'Build the World'
On Fri, Jun 04, 2021 at 10:02:04PM +0300, Alper Nebi Yasak wrote:
The EFI secure boot and capsule test setups need to prepare disk images for their tests using virt-make-fs, which requires access to the host fuse device. This is not exposed to the docker container by default and has to be added explicitly.
Since these tests are marked to run only on the sandbox board, add the fuse device only when testing on sandbox.
Signed-off-by: Alper Nebi Yasak alpernebiyasak@gmail.com
.azure-pipelines.yml | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index 35ab7f30b276..a4d796c41895 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -318,7 +318,15 @@ jobs: # as sandbox testing need create files like spi flash images, etc. # (TODO: clean up this in the future) chmod 777 .
docker run -v $PWD:$(work_dir) $(ci_runner_image) /bin/bash $(work_dir)/test.sh
# Some EFI tests need extra docker args to runset --if [[ "${TEST_PY_BD}" == "sandbox" ]]; then# virt-make-fs needs the fuse deviceif modprobe fuse; thenset -- "$@" --device /dev/fuse:/dev/fusefifidocker run "$@" -v $PWD:$(work_dir) $(ci_runner_image) /bin/bash $(work_dir)/test.sh
Thanks for addressing these issues. For this specific patch, I think I would prefer to just always pass in /dev/fuse with a comment about needing it for some tests.
The filesystem test setup needs to prepare disk images for its tests, with either guestmount or loop mounts. The former requires access to the host fuse device (added in a previous patch), the latter requires access to host loop devices. Both mounts also need additional privileges since docker's default configuration prevents the containers from mounting filesystems (for host security).
Add any available loop devices to the container and try to add as few privileges as possible to run these tests, which narrow down to adding SYS_ADMIN capability and disabling apparmor confinement. However, this much still seems to be insecure enough to let malicious container processes escape as root on the host system [1].
[1] https://blog.trailofbits.com/2019/07/19/understanding-docker-container-escap...
Since the mentioned tests are marked to run only on the sandbox board, add these additional devices and privileges only when testing with that.
An alternative to using mounts is modifying the filesystem tests to use virt-make-fs (like some EFI tests do), but it fails to generate a partitionless FAT filesystem image on Debian systems. Other more feasible alternatives are using guestfish or directly using libguestfs Python bindings to create and populate the images, but switching the test setups to these is nontrivial and is left as future work.
Signed-off-by: Alper Nebi Yasak alpernebiyasak@gmail.com ---
.azure-pipelines.yml | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index a4d796c41895..3ec396ae8905 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -318,13 +318,23 @@ jobs: # as sandbox testing need create files like spi flash images, etc. # (TODO: clean up this in the future) chmod 777 . - # Some EFI tests need extra docker args to run + # Filesystem tests and some EFI tests need extra docker args to run set -- if [[ "${TEST_PY_BD}" == "sandbox" ]]; then - # virt-make-fs needs the fuse device + # virt-make-fs, guestmount, etc. need the fuse device if modprobe fuse; then set -- "$@" --device /dev/fuse:/dev/fuse fi + # mount -o loop needs the loop devices + if modprobe loop; then + for d in $(find /dev -maxdepth 1 -name 'loop*'); do + set -- "$@" --device $d:$d + done + fi + # Needed for mount syscall (for guestmount as well) + set -- "$@" --cap-add SYS_ADMIN + # Default apparmor profile denies mounts + set -- "$@" --security-opt apparmor=unconfined fi docker run "$@" -v $PWD:$(work_dir) $(ci_runner_image) /bin/bash $(work_dir)/test.sh
Some sandbox-only test setups use virt-make-fs and guestmount, which require a readable kernel to work. However, no such kernel is currently available on the Docker container image that is used to run the tests on CIs.
Although a previous patch adds a kernel package to the Dockerfile used to build the container, try to explicitly install it in the CI scripts so that it's not immediately necessary to rebuild the container for that change.
Signed-off-by: Alper Nebi Yasak alpernebiyasak@gmail.com ---
.azure-pipelines.yml | 5 +++++ .gitlab-ci.yml | 5 +++++ 2 files changed, 10 insertions(+)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index 3ec396ae8905..fe82cc57feb0 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -293,6 +293,11 @@ jobs: wget -O - https://github.com/riscv/opensbi/releases/download/v0.9/opensbi-0.9-rv-bin.t... | tar -C /tmp -xJ; export OPENSBI=/tmp/opensbi-0.9-rv-bin/share/opensbi/lp64/generic/firmware/fw_dynamic.bin; fi + if [[ "${TEST_PY_BD}" == "sandbox" ]]; then + sudo apt update + sudo apt install -y linux-image-kvm + sudo chmod +r /boot/vmlinu* /lib/modules/*/vmlinu* || true + fi # the below corresponds to .gitlab-ci.yml "script" cd ${WORK_DIR} export UBOOT_TRAVIS_BUILD_DIR=/tmp/${TEST_PY_BD}; diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index d072e833a3de..be3be8e045e5 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -27,6 +27,11 @@ stages: wget -O - https://github.com/riscv/opensbi/releases/download/v0.9/opensbi-0.9-rv-bin.t... | tar -C /tmp -xJ; export OPENSBI=/tmp/opensbi-0.9-rv-bin/share/opensbi/lp64/generic/firmware/fw_dynamic.bin; fi + - if [[ "${TEST_PY_BD}" == "sandbox" ]]; then + sudo apt update; + sudo apt install -y linux-image-kvm; + sudo chmod +r /boot/vmlinu* /lib/modules/*/vmlinu* || true; + fi
after_script: - rm -rf /tmp/uboot-test-hooks /tmp/venv
On 6/4/21 9:02 PM, Alper Nebi Yasak wrote:
Some sandbox-only test setups use virt-make-fs and guestmount, which require a readable kernel to work. However, no such kernel is currently available on the Docker container image that is used to run the tests on CIs.
Although a previous patch adds a kernel package to the Dockerfile used to build the container, try to explicitly install it in the CI scripts so that it's not immediately necessary to rebuild the container for that change.
Signed-off-by: Alper Nebi Yasak alpernebiyasak@gmail.com
As this patch is superfluous after patch 1 I would prefer to remove it.
Tom can take care of updating the Docker image.
Best regards
Heinrich
.azure-pipelines.yml | 5 +++++ .gitlab-ci.yml | 5 +++++ 2 files changed, 10 insertions(+)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml index 3ec396ae8905..fe82cc57feb0 100644 --- a/.azure-pipelines.yml +++ b/.azure-pipelines.yml @@ -293,6 +293,11 @@ jobs: wget -O - https://github.com/riscv/opensbi/releases/download/v0.9/opensbi-0.9-rv-bin.t... | tar -C /tmp -xJ; export OPENSBI=/tmp/opensbi-0.9-rv-bin/share/opensbi/lp64/generic/firmware/fw_dynamic.bin; fi
if [[ "${TEST_PY_BD}" == "sandbox" ]]; thensudo apt updatesudo apt install -y linux-image-kvmsudo chmod +r /boot/vmlinu* /lib/modules/*/vmlinu* || truefi # the below corresponds to .gitlab-ci.yml "script" cd ${WORK_DIR} export UBOOT_TRAVIS_BUILD_DIR=/tmp/${TEST_PY_BD};diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index d072e833a3de..be3be8e045e5 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -27,6 +27,11 @@ stages: wget -O - https://github.com/riscv/opensbi/releases/download/v0.9/opensbi-0.9-rv-bin.t... | tar -C /tmp -xJ; export OPENSBI=/tmp/opensbi-0.9-rv-bin/share/opensbi/lp64/generic/firmware/fw_dynamic.bin; fi
- if [[ "${TEST_PY_BD}" == "sandbox" ]]; then
sudo apt update;sudo apt install -y linux-image-kvm;sudo chmod +r /boot/vmlinu* /lib/modules/*/vmlinu* || true;fiafter_script: - rm -rf /tmp/uboot-test-hooks /tmp/venv
participants (4)
-
AKASHI Takahiro -
Alper Nebi Yasak -
Heinrich Schuchardt -
Tom Rini