booti usage offset requirements for compressed kernel and initrd with dtb
Hello,
There is data corruption when using booti on a JH7110 (riscv64) board to load (over tftp or ymodem) a compressed kernel, compressed initrd, and device tree blob, when the data are too close together (even though not overlapping). In the adjustments I have tried it seems to be consistent but not clear "why?" by how much for each adjustment is needed to observe the failure versus success.
My question is simple enough, what are the intended offset requirements for loading these data into memory for use with booti ?
For example:
setenv ip 192.168.2.216 setenv loadaddr 0x60000000 # default from Milk-V vendor for Mars CM product setenv kernel_uncomp_size 24193024 # uncompressed size of vmlinuz-5.15.0-gpu117 from file command information setenv kernel_comp_addr_r $loadaddr setenv kernel_addr_r 0x61712800 # $kernel_comp_addr_r + $kernel_uncomp_size dhcp $kernel_addr_r $ip:mars-cm_debian-desktop_sdk-boot/vmlinuz-5.15.0-gpu117 setenv kernel_comp_size 0x$filesize setenv dtb_addr 0x62e25000 # $kernel_addr_r + $kernel_uncomp_size dhcp $dtb_addr $ip:mars-cm_debian-desktop_sdk-boot/dtbs/starfive/jh7110-visionfive-v2.dtb setenv initrd_addr 0x62e31d7f # $dtb_addr + 0x$filesize dhcp $initrd_addr $ip:debian-installer-netboot/initrd.gz; setenv initrd_size 0x$filesize booti $kernel_addr_r $initrd_addr:$initrd_size $dtb_addr
Waiting for root device ...
(and nothing else, this is a failure)
If 0x62e31d7f is replaced by 0x62e31ec6 then the boot is successful and dmesg from debian-installer shows the additional information:
Freeing unused kernel image (initmem) memory: 2196K Run /init as init process with arguments: /init with environment: HOME=/ TERM=linux
So in this the kernel boots and all works normally. Even if that address is 0x62e31ec5 one byte earlier it is a failure.
In other attempts I would change the ordering i.e. so the initrd is first at $loadaddr and then kernel and DTB, or DTB and kernel. If I learned anything it is that compressed initrd must be loaded to a span of memory at least the size of its uncompressed length, and this is also true of the compressed kernel vmlinuz even though booti documentation vaguely infers that there is an area of memory specified for decompression - so a compressed kernel needs two times its uncompressed size is that correct? Also there's no hint anywhere if the device tree blob has special memory constraints but if it is not starting on some 16byte aligned address at least my superstition from observations in testing is that it is not successful... and now, where it ends (in the above example) and initrd begins seems arbitrary.
Sure I could pad things out, they tend to work that way, but I would like to know "why?"
E Shattow (kindly reply-all off-list I am not subscriber)
Hi,
On Thu, Jan 4, 2024 at 5:19 AM E Shattow lucent@gmail.com wrote:
Hello,
There is data corruption when using booti on a JH7110 (riscv64) board to load (over tftp or ymodem) a compressed kernel, compressed initrd, and device tree blob, when the data are too close together (even though not overlapping). In the adjustments I have tried it seems to be consistent but not clear "why?" by how much for each adjustment is needed to observe the failure versus success.
My question is simple enough, what are the intended offset requirements for loading these data into memory for use with booti ?
For example:
setenv ip 192.168.2.216 setenv loadaddr 0x60000000 # default from Milk-V vendor for Mars CM product setenv kernel_uncomp_size 24193024 # uncompressed size of vmlinuz-5.15.0-gpu117 from file command information setenv kernel_comp_addr_r $loadaddr setenv kernel_addr_r 0x61712800 # $kernel_comp_addr_r + $kernel_uncomp_size dhcp $kernel_addr_r $ip:mars-cm_debian-desktop_sdk-boot/vmlinuz-5.15.0-gpu117 setenv kernel_comp_size 0x$filesize setenv dtb_addr 0x62e25000 # $kernel_addr_r + $kernel_uncomp_size dhcp $dtb_addr $ip:mars-cm_debian-desktop_sdk-boot/dtbs/starfive/jh7110-visionfive-v2.dtb setenv initrd_addr 0x62e31d7f # $dtb_addr + 0x$filesize dhcp $initrd_addr $ip:debian-installer-netboot/initrd.gz; setenv initrd_size 0x$filesize booti $kernel_addr_r $initrd_addr:$initrd_size $dtb_addr
Waiting for root device ...
(and nothing else, this is a failure)
If 0x62e31d7f is replaced by 0x62e31ec6 then the boot is successful and dmesg from debian-installer shows the additional information:
Freeing unused kernel image (initmem) memory: 2196K Run /init as init process with arguments: /init with environment: HOME=/ TERM=linux
So in this the kernel boots and all works normally. Even if that address is 0x62e31ec5 one byte earlier it is a failure.
In other attempts I would change the ordering i.e. so the initrd is first at $loadaddr and then kernel and DTB, or DTB and kernel. If I learned anything it is that compressed initrd must be loaded to a span of memory at least the size of its uncompressed length, and this is also true of the compressed kernel vmlinuz even though booti documentation vaguely infers that there is an area of memory specified for decompression - so a compressed kernel needs two times its uncompressed size is that correct? Also there's no hint anywhere if the device tree blob has special memory constraints but if it is not starting on some 16byte aligned address at least my superstition from observations in testing is that it is not successful... and now, where it ends (in the above example) and initrd begins seems arbitrary.
Sure I could pad things out, they tend to work that way, but I would like to know "why?"
I don't know why, but I imagine that Linux finds the initrd in one case and not inthe other...does the kernel log give you any clues?
Also, can you use U-Boot's 'md5sum' command or similar to check that the initrd and DT are the same in each case?
Also, you could use a FIT [1] rather than booti.
Regards, Simon
Hi Simon,
"If this [fdt_high environment variable] is set to the special value 0xffffffff (32-bit machines) or 0xffffffffffffffff (64-bit machines) then the fdt will not be copied at all on boot. For this to work it must reside in writable memory, have sufficient padding on the end of it for U-Boot to add the information it needs into it, and the memory must be accessible by the kernel. This usage is strongly discouraged however as it also stops U-Boot from ensuring the device tree starting address is properly aligned and a misaligned tree will cause OS failures."
https://docs.u-boot.org/en/latest/usage/environment.html
"The device tree blob (dtb) must be placed on an 8-byte boundary"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Docu...
No mention of dtb alignment requirement at the similar documentation for riscv64, but there is this mention: "The RISC-V kernel expects to be placed at a PMD boundary (2MB aligned for rv64 and 4MB aligned for rv32)."
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Docu...
Perhaps then it is both a dtb alignment issue (not mentioned in the documentation?) and the spurious extra requirement after the transferred data length?
If yes, how much is enough to pad after the end of devicetree blob?
Thanks!
E Shattow
On Thu, Jan 4, 2024 at 8:02 AM Simon Glass sjg@chromium.org wrote:
Hi,
On Thu, Jan 4, 2024 at 5:19 AM E Shattow lucent@gmail.com wrote:
Hello,
There is data corruption when using booti on a JH7110 (riscv64) board to load (over tftp or ymodem) a compressed kernel, compressed initrd, and device tree blob, when the data are too close together (even though not overlapping). In the adjustments I have tried it seems to be consistent
but
not clear "why?" by how much for each adjustment is needed to observe the failure versus success.
My question is simple enough, what are the intended offset requirements
for
loading these data into memory for use with booti ?
For example:
setenv ip 192.168.2.216 setenv loadaddr 0x60000000 # default from Milk-V vendor for Mars CM
product
setenv kernel_uncomp_size 24193024 # uncompressed size of vmlinuz-5.15.0-gpu117 from file command information setenv kernel_comp_addr_r $loadaddr setenv kernel_addr_r 0x61712800 # $kernel_comp_addr_r +
$kernel_uncomp_size
dhcp $kernel_addr_r $ip:mars-cm_debian-desktop_sdk-boot/vmlinuz-5.15.0-gpu117 setenv kernel_comp_size 0x$filesize setenv dtb_addr 0x62e25000 # $kernel_addr_r + $kernel_uncomp_size dhcp $dtb_addr
$ip:mars-cm_debian-desktop_sdk-boot/dtbs/starfive/jh7110-visionfive-v2.dtb
setenv initrd_addr 0x62e31d7f # $dtb_addr + 0x$filesize dhcp $initrd_addr $ip:debian-installer-netboot/initrd.gz; setenv initrd_size 0x$filesize booti $kernel_addr_r $initrd_addr:$initrd_size $dtb_addr
Waiting for root device ...
(and nothing else, this is a failure)
If 0x62e31d7f is replaced by 0x62e31ec6 then the boot is successful and dmesg from debian-installer shows the additional information:
Freeing unused kernel image (initmem) memory: 2196K Run /init as init process with arguments: /init with environment: HOME=/ TERM=linux
So in this the kernel boots and all works normally. Even if that address
is
0x62e31ec5 one byte earlier it is a failure.
In other attempts I would change the ordering i.e. so the initrd is first at $loadaddr and then kernel and DTB, or DTB and kernel. If I learned anything it is that compressed initrd must be loaded to a span of memory
at
least the size of its uncompressed length, and this is also true of the compressed kernel vmlinuz even though booti documentation vaguely infers that there is an area of memory specified for decompression - so a compressed kernel needs two times its uncompressed size is that correct? Also there's no hint anywhere if the device tree blob has special memory constraints but if it is not starting on some 16byte aligned address at least my superstition from observations in testing is that it is not successful... and now, where it ends (in the above example) and initrd begins seems arbitrary.
Sure I could pad things out, they tend to work that way, but I would like to know "why?"
I don't know why, but I imagine that Linux finds the initrd in one case and not inthe other...does the kernel log give you any clues?
Also, can you use U-Boot's 'md5sum' command or similar to check that the initrd and DT are the same in each case?
Also, you could use a FIT [1] rather than booti.
Regards, Simon
On Sat, Jan 06, 2024 at 02:45:47PM -0800, E Shattow wrote:
Hi Simon,
"If this [fdt_high environment variable] is set to the special value 0xffffffff (32-bit machines) or 0xffffffffffffffff (64-bit machines) then the fdt will not be copied at all on boot. For this to work it must reside in writable memory, have sufficient padding on the end of it for U-Boot to add the information it needs into it, and the memory must be accessible by the kernel. This usage is strongly discouraged however as it also stops U-Boot from ensuring the device tree starting address is properly aligned and a misaligned tree will cause OS failures."
https://docs.u-boot.org/en/latest/usage/environment.html
"The device tree blob (dtb) must be placed on an 8-byte boundary"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Docu...
No mention of dtb alignment requirement at the similar documentation for riscv64, but there is this mention: "The RISC-V kernel expects to be placed at a PMD boundary (2MB aligned for rv64 and 4MB aligned for rv32)."
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Docu...
Perhaps then it is both a dtb alignment issue (not mentioned in the documentation?) and the spurious extra requirement after the transferred data length?
If yes, how much is enough to pad after the end of devicetree blob?
One should not try and aim for the smallest possible memory region here, but instead pick good defaults. Practically speaking, if I recall, 128MB is an implicit limit in uncompressed kernel image size, so taking one 128MB chunk for the kernel, and the next 128MB chunk for a decompress-to location is reasonable. Then carve a few MB for device tree next, then initrd goes after that.
Thanks!
E Shattow
On Thu, Jan 4, 2024 at 8:02 AM Simon Glass sjg@chromium.org wrote:
Hi,
On Thu, Jan 4, 2024 at 5:19 AM E Shattow lucent@gmail.com wrote:
Hello,
There is data corruption when using booti on a JH7110 (riscv64) board to load (over tftp or ymodem) a compressed kernel, compressed initrd, and device tree blob, when the data are too close together (even though not overlapping). In the adjustments I have tried it seems to be consistent
but
not clear "why?" by how much for each adjustment is needed to observe the failure versus success.
My question is simple enough, what are the intended offset requirements
for
loading these data into memory for use with booti ?
For example:
setenv ip 192.168.2.216 setenv loadaddr 0x60000000 # default from Milk-V vendor for Mars CM
product
setenv kernel_uncomp_size 24193024 # uncompressed size of vmlinuz-5.15.0-gpu117 from file command information setenv kernel_comp_addr_r $loadaddr setenv kernel_addr_r 0x61712800 # $kernel_comp_addr_r +
$kernel_uncomp_size
dhcp $kernel_addr_r $ip:mars-cm_debian-desktop_sdk-boot/vmlinuz-5.15.0-gpu117 setenv kernel_comp_size 0x$filesize setenv dtb_addr 0x62e25000 # $kernel_addr_r + $kernel_uncomp_size dhcp $dtb_addr
$ip:mars-cm_debian-desktop_sdk-boot/dtbs/starfive/jh7110-visionfive-v2.dtb
setenv initrd_addr 0x62e31d7f # $dtb_addr + 0x$filesize dhcp $initrd_addr $ip:debian-installer-netboot/initrd.gz; setenv initrd_size 0x$filesize booti $kernel_addr_r $initrd_addr:$initrd_size $dtb_addr
Waiting for root device ...
(and nothing else, this is a failure)
If 0x62e31d7f is replaced by 0x62e31ec6 then the boot is successful and dmesg from debian-installer shows the additional information:
Freeing unused kernel image (initmem) memory: 2196K Run /init as init process with arguments: /init with environment: HOME=/ TERM=linux
So in this the kernel boots and all works normally. Even if that address
is
0x62e31ec5 one byte earlier it is a failure.
In other attempts I would change the ordering i.e. so the initrd is first at $loadaddr and then kernel and DTB, or DTB and kernel. If I learned anything it is that compressed initrd must be loaded to a span of memory
at
least the size of its uncompressed length, and this is also true of the compressed kernel vmlinuz even though booti documentation vaguely infers that there is an area of memory specified for decompression - so a compressed kernel needs two times its uncompressed size is that correct? Also there's no hint anywhere if the device tree blob has special memory constraints but if it is not starting on some 16byte aligned address at least my superstition from observations in testing is that it is not successful... and now, where it ends (in the above example) and initrd begins seems arbitrary.
Sure I could pad things out, they tend to work that way, but I would like to know "why?"
I don't know why, but I imagine that Linux finds the initrd in one case and not inthe other...does the kernel log give you any clues?
Also, can you use U-Boot's 'md5sum' command or similar to check that the initrd and DT are the same in each case?
Also, you could use a FIT [1] rather than booti.
Regards, Simon
participants (3)
-
E Shattow -
Simon Glass -
Tom Rini