imx8mm memory env in U-Boot
Greetings,
I'm trying to understand what the best memory usage is in U-Boot for IMX8M boards for generic distro configs such as: loadaddr, kernel_addr_r, fdt_addr_r, ramdisk_addr, scriptaddr.
My understanding is that the following is a good rule of thumb: loadaddr = DDR start + 32MB (as FIT images may load kernel at DDR start; but this only allows for a 32MB kernel) kernel_addr_r = $loadaddr fdt_addr_r = $kernel_addr_r + 128MB (allows you up to 128MB for your kernel; handy if you want a kernel with internal ramdisk) ramdisk_addr = fdt_addr_r + 512KB (512KB should be plenty for a dt) scriptaddr = $loadaddr
Looking at the various imx8mm boards upstream they are kind of all over the place but do follow some patterns likely due to some of us just going with what prior boards used.
While I'm at it I've encountered a couple other questions: - why on IMX8MM is CONFIG_LOADADDR is 0x40480000 when DDR starts at 0x40000000. Why the 4608KB offset? - what is CONFIG_SYS_INIT_RAM_SIZE? Most boards are setting this to 2MB but a couple (cl-iot-gate/phycore) set it to 512KB - what are people using for the load address for the kernel within FIT images? I expect start of DDR is appropriate (0x40000000) however for whatever reason I've been using 0x40200000. This plays into the env as you can't overlap where you loaded the FIT image with where you told the FIT image to relocate the kernel to.
Best regards,
Tim
On Thu, Aug 26, 2021 at 09:39:20AM -0700, Tim Harvey wrote:
Greetings,
I'm trying to understand what the best memory usage is in U-Boot for IMX8M boards for generic distro configs such as: loadaddr, kernel_addr_r, fdt_addr_r, ramdisk_addr, scriptaddr.
My understanding is that the following is a good rule of thumb: loadaddr = DDR start + 32MB (as FIT images may load kernel at DDR start; but this only allows for a 32MB kernel) kernel_addr_r = $loadaddr fdt_addr_r = $kernel_addr_r + 128MB (allows you up to 128MB for your kernel; handy if you want a kernel with internal ramdisk) ramdisk_addr = fdt_addr_r + 512KB (512KB should be plenty for a dt) scriptaddr = $loadaddr
Missing from the list here is bootm_size, so that we make sure everything that does need relocation is relocated within a specific size range. Where much of this comes from is (or should be) the huge comment in ti_armv7_common.h that's based off of the Linux kernel arm "booting" document (now converted to rST): /* * We setup defaults based on constraints from the Linux kernel, which should * also be safe elsewhere. We have the default load at 32MB into DDR (for * the kernel), FDT above 128MB (the maximum location for the end of the * kernel), and the ramdisk 512KB above that (allowing for hopefully never * seen large trees). We say all of this must be within the first 256MB * as that will normally be within the kernel lowmem and thus visible via * bootm_size and we only run on platforms with 256MB or more of memory. * * As a temporary storage for DTBO blobs (which should be applied into DTB * blob), we use the location 15.5 MB above the ramdisk. If someone wants to * use ramdisk bigger than 15.5 MB, then DTBO can be loaded and applied to DTB * blob before loading the ramdisk, as DTBO location is only used as a temporary * storage, and can be re-used after 'fdt apply' command is done. */
At this point, re-reading and referencing both: https://www.kernel.org/doc/Documentation/arm64/booting.rst https://www.kernel.org/doc/Documentation/arm/Booting would be good, and note that there's not currently a similar document for RISC-V, they often follow the same guidelines. And I know you're talking about imx8 specifically right now but due to the copy/paste nature of these kinds of values, I like to err on the side of maximal safety. Which means that we should bump the DTB size to 2MB, per arm64.
It also doesn't cover kernel_comp_addr_r / kernel_comp_size for automatic decompression of Image files, but should.
Note that I believe (but would have to think on and re-read a bunch of stuff to be sure), it's not that saying the kernel address is at 32MB from the start of memory limits us to 32MB, but that it makes life easier all around.
Looking at the various imx8mm boards upstream they are kind of all over the place but do follow some patterns likely due to some of us just going with what prior boards used.
While I'm at it I've encountered a couple other questions:
- why on IMX8MM is CONFIG_LOADADDR is 0x40480000 when DDR starts at
0x40000000. Why the 4608KB offset?
- what is CONFIG_SYS_INIT_RAM_SIZE? Most boards are setting this to
2MB but a couple (cl-iot-gate/phycore) set it to 512KB
I feel like it's pretty likely CONFIG_SYS_INIT_RAM_SIZE has been copy-pasted around as part of setting CONFIG_SYS_INIT_SP_OFFSET which is unused. A lot of unused (outside of m68k / PowerPC generally) options in that area.
- what are people using for the load address for the kernel within FIT
images? I expect start of DDR is appropriate (0x40000000) however for whatever reason I've been using 0x40200000. This plays into the env as you can't overlap where you loaded the FIT image with where you told the FIT image to relocate the kernel to.
Getting some documentation under doc/ about both the environment variables and optimal FIT layout would be good. Since we're talking about arm64 here (but this is true for RISC-V too, same header). Reading over booti_setup(), the entry point we set is (so long as 2MB aligned) where we relocate to. So the 0x40200000 would be base + 2MB, and there were points in history on arm64 where it had to be at that offset, I believe.
On Thu, Aug 26, 2021 at 12:41 PM Tom Rini trini@konsulko.com wrote:
On Thu, Aug 26, 2021 at 09:39:20AM -0700, Tim Harvey wrote:
Greetings,
I'm trying to understand what the best memory usage is in U-Boot for IMX8M boards for generic distro configs such as: loadaddr, kernel_addr_r, fdt_addr_r, ramdisk_addr, scriptaddr.
My understanding is that the following is a good rule of thumb: loadaddr = DDR start + 32MB (as FIT images may load kernel at DDR start; but this only allows for a 32MB kernel) kernel_addr_r = $loadaddr fdt_addr_r = $kernel_addr_r + 128MB (allows you up to 128MB for your kernel; handy if you want a kernel with internal ramdisk) ramdisk_addr = fdt_addr_r + 512KB (512KB should be plenty for a dt) scriptaddr = $loadaddr
Hi Tom,
Thanks for the reply.
Missing from the list here is bootm_size, so that we make sure everything that does need relocation is relocated within a specific size range.
I still don't quite understand bootm_size, you say it sets the limit to where things are relocated to. Shouldn't this just be the size of dram then? A few IMX8MM boards set this but most do not.
Where much of this comes from is (or should be) the huge comment in ti_armv7_common.h that's based off of the Linux kernel arm "booting" document (now converted to rST): /*
- We setup defaults based on constraints from the Linux kernel, which should
- also be safe elsewhere. We have the default load at 32MB into DDR (for
- the kernel), FDT above 128MB (the maximum location for the end of the
- kernel), and the ramdisk 512KB above that (allowing for hopefully never
- seen large trees). We say all of this must be within the first 256MB
- as that will normally be within the kernel lowmem and thus visible via
- bootm_size and we only run on platforms with 256MB or more of memory.
- As a temporary storage for DTBO blobs (which should be applied into DTB
- blob), we use the location 15.5 MB above the ramdisk. If someone wants to
- use ramdisk bigger than 15.5 MB, then DTBO can be loaded and applied to DTB
- blob before loading the ramdisk, as DTBO location is only used as a temporary
- storage, and can be re-used after 'fdt apply' command is done.
*/
Right, that's where I got my recommendations and I don't understand the reasoning behind the default loadaddr be 32MB into DDR.
At this point, re-reading and referencing both: https://www.kernel.org/doc/Documentation/arm64/booting.rst https://www.kernel.org/doc/Documentation/arm/Booting would be good, and note that there's not currently a similar document for RISC-V, they often follow the same guidelines. And I know you're talking about imx8 specifically right now but due to the copy/paste nature of these kinds of values, I like to err on the side of maximal safety. Which means that we should bump the DTB size to 2MB, per arm64.
ok, good to know 2MB should be allotted for dtb.
It also doesn't cover kernel_comp_addr_r / kernel_comp_size for automatic decompression of Image files, but should.
interesting... I didn't even realize booti supported compressed images!
I see now commit 414c34ed55: ("image: Add compressed Image parsing support in booti."). I'm not clear why the uncompressed kernel needs to be moved to kernel_addr_r after decompression... why can't it simply be decompressed directly to kernel_addr_r?
I would think kernel_comp_size would typically be set to filesize as currently that is set by tftp as well as fs load commands.
Note that I believe (but would have to think on and re-read a bunch of stuff to be sure), it's not that saying the kernel address is at 32MB from the start of memory limits us to 32MB, but that it makes life easier all around.
Looking at the various imx8mm boards upstream they are kind of all over the place but do follow some patterns likely due to some of us just going with what prior boards used.
While I'm at it I've encountered a couple other questions:
- why on IMX8MM is CONFIG_LOADADDR is 0x40480000 when DDR starts at
0x40000000. Why the 4608KB offset?
any idea why IMX8MM boards are using DDR+4608KB for loadaddr vs just DDR? I am hoping some of the IMX8MM board maintainers I've cc'd here can answer that.
- what is CONFIG_SYS_INIT_RAM_SIZE? Most boards are setting this to
2MB but a couple (cl-iot-gate/phycore) set it to 512KB
I feel like it's pretty likely CONFIG_SYS_INIT_RAM_SIZE has been copy-pasted around as part of setting CONFIG_SYS_INIT_SP_OFFSET which is unused. A lot of unused (outside of m68k / PowerPC generally) options in that area.
- what are people using for the load address for the kernel within FIT
images? I expect start of DDR is appropriate (0x40000000) however for whatever reason I've been using 0x40200000. This plays into the env as you can't overlap where you loaded the FIT image with where you told the FIT image to relocate the kernel to.
Getting some documentation under doc/ about both the environment variables and optimal FIT layout would be good. Since we're talking about arm64 here (but this is true for RISC-V too, same header). Reading over booti_setup(), the entry point we set is (so long as 2MB aligned) where we relocate to. So the 0x40200000 would be base + 2MB, and there were points in history on arm64 where it had to be at that offset, I believe.
I did verify that using 0x40000000 for loadaddr in a FIT image booted a modern kernel just fine so yes I guess the reasoning must have been historical.
Best regards,
Tim
On Fri, Aug 27, 2021 at 02:32:00PM -0700, Tim Harvey wrote:
On Thu, Aug 26, 2021 at 12:41 PM Tom Rini trini@konsulko.com wrote:
On Thu, Aug 26, 2021 at 09:39:20AM -0700, Tim Harvey wrote:
Greetings,
I'm trying to understand what the best memory usage is in U-Boot for IMX8M boards for generic distro configs such as: loadaddr, kernel_addr_r, fdt_addr_r, ramdisk_addr, scriptaddr.
My understanding is that the following is a good rule of thumb: loadaddr = DDR start + 32MB (as FIT images may load kernel at DDR start; but this only allows for a 32MB kernel) kernel_addr_r = $loadaddr fdt_addr_r = $kernel_addr_r + 128MB (allows you up to 128MB for your kernel; handy if you want a kernel with internal ramdisk) ramdisk_addr = fdt_addr_r + 512KB (512KB should be plenty for a dt) scriptaddr = $loadaddr
Hi Tom,
Thanks for the reply.
Missing from the list here is bootm_size, so that we make sure everything that does need relocation is relocated within a specific size range.
I still don't quite understand bootm_size, you say it sets the limit to where things are relocated to. Shouldn't this just be the size of dram then? A few IMX8MM boards set this but most do not.
Both ARM32 and ARM64 define limits on where in memory anything can reside and still be seen / used by the kernel early on. There's certainly a lot of platforms where the whole of DRAM fits in that window, being super safe for copy/paste is a big concern of mine over all. When I wrote out that comment for the TI platforms, I remember seeing just how much arbitrary offsets had been reused over and over and on many platforms without understanding why.
Where much of this comes from is (or should be) the huge comment in ti_armv7_common.h that's based off of the Linux kernel arm "booting" document (now converted to rST): /*
- We setup defaults based on constraints from the Linux kernel, which should
- also be safe elsewhere. We have the default load at 32MB into DDR (for
- the kernel), FDT above 128MB (the maximum location for the end of the
- kernel), and the ramdisk 512KB above that (allowing for hopefully never
- seen large trees). We say all of this must be within the first 256MB
- as that will normally be within the kernel lowmem and thus visible via
- bootm_size and we only run on platforms with 256MB or more of memory.
- As a temporary storage for DTBO blobs (which should be applied into DTB
- blob), we use the location 15.5 MB above the ramdisk. If someone wants to
- use ramdisk bigger than 15.5 MB, then DTBO can be loaded and applied to DTB
- blob before loading the ramdisk, as DTBO location is only used as a temporary
- storage, and can be re-used after 'fdt apply' command is done.
*/
Right, that's where I got my recommendations and I don't understand the reasoning behind the default loadaddr be 32MB into DDR.
I _think_, but would want to re-re-read https://people.kernel.org/linusw/how-the-arm32-linux-kernel-decompresses to make sure I'm remembering it right, if we put the zImage at 32MB offset, that means the decompressor won't have to move things another time, and saves boot time. Not a concern for ARM64 because it's our responsibility to do the decompression.
At this point, re-reading and referencing both: https://www.kernel.org/doc/Documentation/arm64/booting.rst https://www.kernel.org/doc/Documentation/arm/Booting would be good, and note that there's not currently a similar document for RISC-V, they often follow the same guidelines. And I know you're talking about imx8 specifically right now but due to the copy/paste nature of these kinds of values, I like to err on the side of maximal safety. Which means that we should bump the DTB size to 2MB, per arm64.
ok, good to know 2MB should be allotted for dtb.
It also doesn't cover kernel_comp_addr_r / kernel_comp_size for automatic decompression of Image files, but should.
interesting... I didn't even realize booti supported compressed images!
I see now commit 414c34ed55: ("image: Add compressed Image parsing support in booti."). I'm not clear why the uncompressed kernel needs to be moved to kernel_addr_r after decompression... why can't it simply be decompressed directly to kernel_addr_r?
I would think kernel_comp_size would typically be set to filesize as currently that is set by tftp as well as fs load commands.
Off-hand, I'm not sure it couldn't be further optimized. I suspect it's about avoiding overlaps, but there could be some checks made for optimal aligned values and avoid that.
Note that I believe (but would have to think on and re-read a bunch of stuff to be sure), it's not that saying the kernel address is at 32MB from the start of memory limits us to 32MB, but that it makes life easier all around.
Looking at the various imx8mm boards upstream they are kind of all over the place but do follow some patterns likely due to some of us just going with what prior boards used.
While I'm at it I've encountered a couple other questions:
- why on IMX8MM is CONFIG_LOADADDR is 0x40480000 when DDR starts at
0x40000000. Why the 4608KB offset?
any idea why IMX8MM boards are using DDR+4608KB for loadaddr vs just DDR? I am hoping some of the IMX8MM board maintainers I've cc'd here can answer that.
I too would be interested in knowing what's going on there.
- what is CONFIG_SYS_INIT_RAM_SIZE? Most boards are setting this to
2MB but a couple (cl-iot-gate/phycore) set it to 512KB
I feel like it's pretty likely CONFIG_SYS_INIT_RAM_SIZE has been copy-pasted around as part of setting CONFIG_SYS_INIT_SP_OFFSET which is unused. A lot of unused (outside of m68k / PowerPC generally) options in that area.
- what are people using for the load address for the kernel within FIT
images? I expect start of DDR is appropriate (0x40000000) however for whatever reason I've been using 0x40200000. This plays into the env as you can't overlap where you loaded the FIT image with where you told the FIT image to relocate the kernel to.
Getting some documentation under doc/ about both the environment variables and optimal FIT layout would be good. Since we're talking about arm64 here (but this is true for RISC-V too, same header). Reading over booti_setup(), the entry point we set is (so long as 2MB aligned) where we relocate to. So the 0x40200000 would be base + 2MB, and there were points in history on arm64 where it had to be at that offset, I believe.
I did verify that using 0x40000000 for loadaddr in a FIT image booted a modern kernel just fine so yes I guess the reasoning must have been historical.
Good to know. Were there any other relocations shown? One part of these values was also to hope to avoid any avoidable relocations of the kernel (and if possible, initrd, but that can be trickier).
participants (2)
-
Tim Harvey -
Tom Rini