Raspberry Pi won’t boot from compressed subvolume (BTRFS)
Hello u-boot team,
I’ve been experimenting for about a year with a NixOS-based Raspberry Pi image, with the end goal of a zstd-compressed image that uses BTRFS subvolumes, having the root filesystem at @ and boot at @boot (with several other subvolumes). The NixOS Raspberry Pi images use u-boot by default. I’ve started a few discussions in the NixOS community as I learn and iterate: https://discourse.nixos.org/t/raspberry-pi-nixos-on-btrfs-root/14081 and I’ve made my code available at: https://github.com/n8henrie/nixos-btrfs-pi . I’m sure it’s obvious (or will be soon) that I’m a hobbyist, a novice with NixOS, and that I understand very little about u-boot and bootloaders in general, so I hope you can be patient as I humbly ask for someone to point me in the right direction.
For reference, the official NixOS code for generating u-boot images is here: https://github.com/NixOS/nixpkgs/blob/3d4e586313d292c9e764a8e88a1fdccb16ffb2...
I’ve made a lot of progress, including an image that reliably boots from @boot to the @ root subvolume with the u-boot configuration:
CONFIG_CMD_BTRFS=y CONFIG_ZSTD=y CONFIG_BOOTCOMMAND="setenv boot_prefixes / /boot/ /@/ /@boot/; run distro_bootcmd;"
with kernel params: "root=LABEL=NIXOS_SD" "rootfstype=btrfs" "rootflags=subvol=@"
Unfortunately, after the initial boot and going through the initial OS installation steps, if I have compression enabled, it reboots into a boot loop with the following error messages:
Scanning mmc 0:2... Found /@boot/extlinux/extlinux.conf Retrieving file: /@boot/extlinux/extlinux.conf 2938 bytes read in 27 ms (105.5 KiB/s)
1 NixOS Default 2 NixOS Configuration 4 (2022-02-24 21:34 22.05pre356135.7f9b6c2babf) 3 NixOS Configuration 3 (2022-02-25 03:59 22.05pre356435.7/9b6eZbabf) 4: NixOS Configuration 2 (2022-02-24 21:34 22.05pre356435.7/9b6c2babr) 5: NixOS Enter choice: 1: NixOS - Default Retrieving file: /@boot/extlinux/..nixos/zhbharzhzqga617uc37uk0g6q1bx891u-initrd-linux-5.10.101-initrd 20588420 bytes read in 1910 ms (10.3 MiB/s) Retrieving file: /@boot/extlinux/..nixos/6c1m6j0k1gmjrmikOyyizariiry?Zlqr-linux-5.10.101-Image 50629120 bytes read in 6863 ms (7 MiB/s) append: init-mix/store/pikl8ju4077iaix36n8mifzu6l6rcajz-nixos-system-nixpi-22.05p re356435-7/9b6eZbabf/init console=tty1 console=ttyAMAO console=ttyS0, 1115200 root= LABEL=-NIXOS_SD rootfstype=btrfs rootflags=subvol=@ loglevel=4 Retrieving file: /@boot/extlinux/../nixos/6c16j0k1gmjrmikOyyizari1ry72lqr-linux-5.10.101-dtbs/broadcon/bcn2837-rpi-3-b.dtb 14310 bytes read in 66 ms (210.9 KiB/s) Moving Image from 0x80000 to 0x200000, end=32f0000 ## Flattened Device Tree blob at 03700000 Booting using the fdt blob at 0x3700000 Using Device Tree in place at 0000000003700000 end 00000000037067e5 fdt_find_or_add_subnode: memory: FDT_ERR_BADSTRUCTURE ERROR: arch-specific fdt fixup failed - must RESET the board to recover.
FDT creation failed! resetting ...
This happens even though I have CONFIG_ZSTD=y. If I do *not* enable compression (in my BTRFS mount options), it reboots normally and everything works!
I may have made some OCR / C&P errors above; I’ve posted some pictures of the error messages (and similar discussion to this email, with no responses) here: https://discourse.nixos.org/t/btrfs-pi-wont-boot-from-compressed-subvolume/1...
I’m not currently using BTRFS RAID, just a single device. One of my other (x86_64) machines uses a similar setup with @, @boot, and zstd-compression with grub and works well, so I’m hoping this is also possible with u-boot.
Seeing that the errors mention the FDT and fixup, I tried CONFIG_ARCH_FIXUP_FDT_MEMORY with both =y and =n without interesting changes to the error messages.
What I’m assuming, based on the above, is that re-writing the @boot subvolume with compression is moving the location of the device tree files and that u-boot is no longer able to find the necessary files (some of which I think may also be in the nix store in the @nix subvolume).
Does that sound like a reasonable guess as to what is going on? If so, are there config options or settings that I might be able to use to help u-boot find the necessary files after things are re-written with compression?
Many thanks in advance for your time and attention, and thanks for your hard work on u-boot!
On 22/05/2022 17:36, Nathan Henrie wrote:
Hello u-boot team,
I’ve been experimenting for about a year with a NixOS-based Raspberry Pi image, with the end goal of a zstd-compressed image that uses BTRFS subvolumes, having the root filesystem at @ and boot at @boot (with several other subvolumes). The NixOS Raspberry Pi images use u-boot by default. I’ve started a few discussions in the NixOS community as I learn and iterate: https://discourse.nixos.org/t/raspberry-pi-nixos-on-btrfs-root/14081 and I’ve made my code available at: https://github.com/n8henrie/nixos-btrfs-pi . I’m sure it’s obvious (or will be soon) that I’m a hobbyist, a novice with NixOS, and that I understand very little about u-boot and bootloaders in general, so I hope you can be patient as I humbly ask for someone to point me in the right direction.
For reference, the official NixOS code for generating u-boot images is here: https://github.com/NixOS/nixpkgs/blob/3d4e586313d292c9e764a8e88a1fdccb16ffb2...
I’ve made a lot of progress, including an image that reliably boots from @boot to the @ root subvolume with the u-boot configuration:
CONFIG_CMD_BTRFS=y CONFIG_ZSTD=y CONFIG_BOOTCOMMAND="setenv boot_prefixes / /boot/ /@/ /@boot/; run distro_bootcmd;"
with kernel params: "root=LABEL=NIXOS_SD" "rootfstype=btrfs" "rootflags=subvol=@"
Unfortunately, after the initial boot and going through the initial OS installation steps, if I have compression enabled, it reboots into a boot loop with the following error messages:
Scanning mmc 0:2... Found /@boot/extlinux/extlinux.conf Retrieving file: /@boot/extlinux/extlinux.conf 2938 bytes read in 27 ms (105.5 KiB/s)
1 NixOS Default 2 NixOS Configuration 4 (2022-02-24 21:34 22.05pre356135.7f9b6c2babf) 3 NixOS Configuration 3 (2022-02-25 03:59 22.05pre356435.7/9b6eZbabf) 4: NixOS Configuration 2 (2022-02-24 21:34 22.05pre356435.7/9b6c2babr) 5: NixOS Enter choice: 1: NixOS - Default Retrieving file: /@boot/extlinux/..nixos/zhbharzhzqga617uc37uk0g6q1bx891u-initrd-linux-5.10.101-initrd 20588420 bytes read in 1910 ms (10.3 MiB/s) Retrieving file: /@boot/extlinux/..nixos/6c1m6j0k1gmjrmikOyyizariiry?Zlqr-linux-5.10.101-Image 50629120 bytes read in 6863 ms (7 MiB/s) append: init-mix/store/pikl8ju4077iaix36n8mifzu6l6rcajz-nixos-system-nixpi-22.05p re356435-7/9b6eZbabf/init console=tty1 console=ttyAMAO console=ttyS0, 1115200 root= LABEL=-NIXOS_SD rootfstype=btrfs rootflags=subvol=@ loglevel=4 Retrieving file: /@boot/extlinux/../nixos/6c16j0k1gmjrmikOyyizari1ry72lqr-linux-5.10.101-dtbs/broadcon/bcn2837-rpi-3-b.dtb
Just a random guess, but bcn2837-rpi-3-b.dtb does look right, the DTB should be called bcm2837-rpi-3-b.dtb
14310 bytes read in 66 ms (210.9 KiB/s) Moving Image from 0x80000 to 0x200000, end=32f0000 ## Flattened Device Tree blob at 03700000 Booting using the fdt blob at 0x3700000 Using Device Tree in place at 0000000003700000 end 00000000037067e5 fdt_find_or_add_subnode: memory: FDT_ERR_BADSTRUCTURE ERROR: arch-specific fdt fixup failed
- must RESET the board to recover.
Then you try to fixup that address which has no valid DTB and fails.
Anyway to me that sounds like a nixOS problem rather then a U-Boot problem. I'm not even sure which version of U-Boot you are using and if it has any patches on top of the upstream version.
Regards, Matthias
FDT creation failed! resetting ...
This happens even though I have CONFIG_ZSTD=y. If I do *not* enable compression (in my BTRFS mount options), it reboots normally and everything works!
I may have made some OCR / C&P errors above; I’ve posted some pictures of the error messages (and similar discussion to this email, with no responses) here: https://discourse.nixos.org/t/btrfs-pi-wont-boot-from-compressed-subvolume/1...
I’m not currently using BTRFS RAID, just a single device. One of my other (x86_64) machines uses a similar setup with @, @boot, and zstd-compression with grub and works well, so I’m hoping this is also possible with u-boot.
Seeing that the errors mention the FDT and fixup, I tried CONFIG_ARCH_FIXUP_FDT_MEMORY with both =y and =n without interesting changes to the error messages.
What I’m assuming, based on the above, is that re-writing the @boot subvolume with compression is moving the location of the device tree files and that u-boot is no longer able to find the necessary files (some of which I think may also be in the nix store in the @nix subvolume).
Does that sound like a reasonable guess as to what is going on? If so, are there config options or settings that I might be able to use to help u-boot find the necessary files after things are re-written with compression?
Many thanks in advance for your time and attention, and thanks for your hard work on u-boot!
Hi Matthias, thanks for your time and response.
Just a random guess, but bcn2837-rpi-3-b.dtb does look right, the DTB should be called bcm2837-rpi-3-b.dtb
Sorry, that was an OCR error (wasn't sure how to screenshot or log to a file in this context, so pictures -> OCR seemed like my best option, pictures available here: https://discourse.nixos.org/t/btrfs-pi-wont-boot-from-compressed-subvolume/1...). It is indeed `bc*m*`... in the actual output.
Then you try to fixup that address which has no valid DTB and fails.
Anyway to me that sounds like a nixOS problem rather then a U-Boot problem. I'm not even sure which version of U-Boot you are using and if it has any patches on top of the upstream version.
Sorry, I included the link to the relevant code (https://github.com/NixOS/nixpkgs/blob/3d4e586313d292c9e764a8e88a1fdccb16ffb2...) but should have included those details:
``` defaultSrc = fetchurl { url = "ftp://ftp.denx.de/pub/u-boot/u-boot-${defaultVersion}.tar.bz2"; hash = "sha256-gbRUMifbIowD+KG/XdvIE7C7j2VVzkYGTvchpvxoBBM="; defaultVersion = "2022.01"; ```
By default it will have two patches: - https://github.com/NixOS/nixpkgs/blob/master/pkgs/misc/uboot/0001-configs-rp... - https://github.com/NixOS/nixpkgs/blob/master/pkgs/misc/uboot/0001-rpi-Copy-p...
The latter looks like it might be relevant.
I was suspicious about it being a u-boot issue because everything works fine by default, it's only when I enable zstd compression that I get the above errors. However I'm pretty new to NixOS and obviously know little about u-boot, so you may well be correct!
Nate
On Wed, May 25, 2022 at 2:58 AM Matthias Brugger matthias.bgg@gmail.com wrote:
On 22/05/2022 17:36, Nathan Henrie wrote:
Hello u-boot team,
I’ve been experimenting for about a year with a NixOS-based Raspberry Pi image, with the end goal of a zstd-compressed image that uses BTRFS subvolumes, having the root filesystem at @ and boot at @boot (with several other subvolumes). The NixOS Raspberry Pi images use u-boot by default. I’ve started a few discussions in the NixOS community as I learn and iterate: https://discourse.nixos.org/t/raspberry-pi-nixos-on-btrfs-root/14081 and I’ve made my code available at: https://github.com/n8henrie/nixos-btrfs-pi . I’m sure it’s obvious (or will be soon) that I’m a hobbyist, a novice with NixOS, and that I understand very little about u-boot and bootloaders in general, so I hope you can be patient as I humbly ask for someone to point me in the right direction.
For reference, the official NixOS code for generating u-boot images is here: https://github.com/NixOS/nixpkgs/blob/3d4e586313d292c9e764a8e88a1fdccb16ffb2...
I’ve made a lot of progress, including an image that reliably boots from @boot to the @ root subvolume with the u-boot configuration:
CONFIG_CMD_BTRFS=y CONFIG_ZSTD=y CONFIG_BOOTCOMMAND="setenv boot_prefixes / /boot/ /@/ /@boot/; run distro_bootcmd;"
with kernel params: "root=LABEL=NIXOS_SD" "rootfstype=btrfs" "rootflags=subvol=@"
Unfortunately, after the initial boot and going through the initial OS installation steps, if I have compression enabled, it reboots into a boot loop with the following error messages:
Scanning mmc 0:2... Found /@boot/extlinux/extlinux.conf Retrieving file: /@boot/extlinux/extlinux.conf 2938 bytes read in 27 ms (105.5 KiB/s)
1 NixOS Default 2 NixOS Configuration 4 (2022-02-24 21:34 22.05pre356135.7f9b6c2babf) 3 NixOS Configuration 3 (2022-02-25 03:59 22.05pre356435.7/9b6eZbabf) 4: NixOS Configuration 2 (2022-02-24 21:34 22.05pre356435.7/9b6c2babr) 5: NixOS Enter choice: 1: NixOS - Default Retrieving file: /@boot/extlinux/..nixos/zhbharzhzqga617uc37uk0g6q1bx891u-initrd-linux-5.10.101-initrd 20588420 bytes read in 1910 ms (10.3 MiB/s) Retrieving file: /@boot/extlinux/..nixos/6c1m6j0k1gmjrmikOyyizariiry?Zlqr-linux-5.10.101-Image 50629120 bytes read in 6863 ms (7 MiB/s) append: init-mix/store/pikl8ju4077iaix36n8mifzu6l6rcajz-nixos-system-nixpi-22.05p re356435-7/9b6eZbabf/init console=tty1 console=ttyAMAO console=ttyS0, 1115200 root= LABEL=-NIXOS_SD rootfstype=btrfs rootflags=subvol=@ loglevel=4 Retrieving file: /@boot/extlinux/../nixos/6c16j0k1gmjrmikOyyizari1ry72lqr-linux-5.10.101-dtbs/broadcon/bcn2837-rpi-3-b.dtb
Just a random guess, but bcn2837-rpi-3-b.dtb does look right, the DTB should be called bcm2837-rpi-3-b.dtb
14310 bytes read in 66 ms (210.9 KiB/s) Moving Image from 0x80000 to 0x200000, end=32f0000 ## Flattened Device Tree blob at 03700000 Booting using the fdt blob at 0x3700000 Using Device Tree in place at 0000000003700000 end 00000000037067e5 fdt_find_or_add_subnode: memory: FDT_ERR_BADSTRUCTURE ERROR: arch-specific fdt fixup failed
- must RESET the board to recover.
Then you try to fixup that address which has no valid DTB and fails.
Anyway to me that sounds like a nixOS problem rather then a U-Boot problem. I'm not even sure which version of U-Boot you are using and if it has any patches on top of the upstream version.
Regards, Matthias
FDT creation failed! resetting ...
This happens even though I have CONFIG_ZSTD=y. If I do *not* enable compression (in my BTRFS mount options), it reboots normally and everything works!
I may have made some OCR / C&P errors above; I’ve posted some pictures of the error messages (and similar discussion to this email, with no responses) here: https://discourse.nixos.org/t/btrfs-pi-wont-boot-from-compressed-subvolume/1...
I’m not currently using BTRFS RAID, just a single device. One of my other (x86_64) machines uses a similar setup with @, @boot, and zstd-compression with grub and works well, so I’m hoping this is also possible with u-boot.
Seeing that the errors mention the FDT and fixup, I tried CONFIG_ARCH_FIXUP_FDT_MEMORY with both =y and =n without interesting changes to the error messages.
What I’m assuming, based on the above, is that re-writing the @boot subvolume with compression is moving the location of the device tree files and that u-boot is no longer able to find the necessary files (some of which I think may also be in the nix store in the @nix subvolume).
Does that sound like a reasonable guess as to what is going on? If so, are there config options or settings that I might be able to use to help u-boot find the necessary files after things are re-written with compression?
Many thanks in advance for your time and attention, and thanks for your hard work on u-boot!
participants (2)
-
Matthias Brugger -
Nathan Henrie