bugs with EFI "reserved" memory and LMB?
Efi loader module have its own memory management that flags as reserved the area between ram_top to ram_end (currently where dt reserved-memory is falling). uboot lmb updates reserved-memory by adding these efi loader module areas (see lmb_reserve_common->efi_lmb_reserve).
On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in the device tree (used for comunication with other microcontroller on the SoC, OPTEE, ...) falling in the DDR range between 456 and 512 MB.
Normally U-Boot relocate itself at the end of DDR so it overlap this reserved-memory and this of course does not work [1].
In this case U-Boot prints the following errors:
ERROR: reserving fdt memory region failed (addr=9c800000 size=300000) ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000) ... ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000)
because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve memories (lmb_flag LMB_NOMAP).
To fix this I moved the U-Boot relocation address, implementing board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value with no overlap (448 MB).
The memory map is: +---------------+ 512 MB |DT reserved-mem| +---------------+ 456 MB |free space | +---------------+ 448 MB (ram_top) | | |uboot | | | +---------------+ 0 MB
This is working fine (the board is able to boot, no memory overlaps) ...
however, in this configuration U-Boot we still have some errors prints while loading linux dtb:
ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9cc00000 size=e00000 flags=4) ERROR: reserving fdt memory region failed (addr=9da00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4) ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4) ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
And the reason of these error is complete different: The efi loader lmb memory reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area.
1) lib/lmb.c:lmb_reserve_common() reserve these memories: ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
2) lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved area. This is not coalesced/merged with the u-boot reserved area because it is not contiguous.
This is the reserved-memory array at this point: ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
3) lib/lmb.c:efi_lmb_reserve() reserve some more areas. In the last two steps efi loader add some reserved area that overlap u-boot and fdt reserved area BUT they are also contiguous to the "efi loader reserved area". The very last step reserve also area between ram_top (448M) to ram_end (512M)
We fall in this condition (where we have overlapping areas!): ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
4) Now, while loading the linux fdt the reserved-memory areas are checked toward efi loader reserved area, they overlap, BUT they have different lmb_flag. So we get the ERROR print.
Hopefully this is clear, I undestand is not that obvious ...
IMO we have two different bugs: - there is nothing that prevent that coalesced area may overlap other areas (while lmb module expect that there aren't overlapping areas) - efi loader (correctly) reserve between ram_top and ram_end BUT this area is reserved by fdt.
One potential solution could be to override efi_add_known_memory() and set ram_top = ram_end, e.g.
- efi_add_conventional_memory_map(ram_start, ram_end, ram_top); + efi_add_conventional_memory_map(ram_start, ram_end, ram_end);
but this does not really seems like something that should ve overridden at the board level.
Any suggestion?
Emanuele
On 27.07.23 13:08, Emanuele Ghidoli wrote:
Efi loader module have its own memory management that flags as reserved the area between ram_top to ram_end (currently where dt reserved-memory is falling). uboot lmb updates reserved-memory by adding these efi loader module areas (see lmb_reserve_common->efi_lmb_reserve).
On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in the device tree (used for comunication with other microcontroller on the SoC, OPTEE, ...) falling in the DDR range between 456 and 512 MB.
Normally U-Boot relocate itself at the end of DDR so it overlap this reserved-memory and this of course does not work [1].
In this case U-Boot prints the following errors:
ERROR: reserving fdt memory region failed (addr=9c800000 size=300000) ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000) ... ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000)
because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve memories (lmb_flag LMB_NOMAP).
To fix this I moved the U-Boot relocation address, implementing board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value with no overlap (448 MB).
The memory map is: +---------------+ 512 MB |DT reserved-mem| +---------------+ 456 MB |free space | +---------------+ 448 MB (ram_top) | | |uboot | | | +---------------+ 0 MB
This is working fine (the board is able to boot, no memory overlaps) ...
however, in this configuration U-Boot we still have some errors prints while loading linux dtb:
ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9cc00000 size=e00000 flags=4) ERROR: reserving fdt memory region failed (addr=9da00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4) ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4) ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
And the reason of these error is complete different: The efi loader lmb memory reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area.
lib/lmb.c:lmb_reserve_common() reserve these memories: ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved
area. This is not coalesced/merged with the u-boot reserved area because it is not contiguous.
This is the reserved-memory array at this point: ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- lib/lmb.c:efi_lmb_reserve() reserve some more areas.
In the last two steps efi loader add some reserved area that overlap u-boot and fdt reserved area BUT they are also contiguous to the "efi loader reserved area". The very last step reserve also area between ram_top (448M) to ram_end (512M)
We fall in this condition (where we have overlapping areas!): ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- Now, while loading the linux fdt the reserved-memory areas are checked
toward efi loader reserved area, they overlap, BUT they have different lmb_flag. So we get the ERROR print.
Hopefully this is clear, I undestand is not that obvious ...
Thanks Emanuele for reporting the issue.
Could you, please, provide the output of
efidebug memmap
and of
bdinfo
Best regards
Heinrich
IMO we have two different bugs:
- there is nothing that prevent that coalesced area may overlap other areas (while lmb module expect that there aren't overlapping areas)
- efi loader (correctly) reserve between ram_top and ram_end BUT this area is reserved by fdt.
One potential solution could be to override efi_add_known_memory() and set ram_top = ram_end, e.g.
- efi_add_conventional_memory_map(ram_start, ram_end, ram_top); + efi_add_conventional_memory_map(ram_start, ram_end, ram_end);
but this does not really seems like something that should ve overridden at the board level.
Any suggestion?
Emanuele
On 27/07/2023 13:21, Heinrich Schuchardt wrote:
On 27.07.23 13:08, Emanuele Ghidoli wrote:
Efi loader module have its own memory management that flags as reserved the area between ram_top to ram_end (currently where dt reserved-memory is falling). uboot lmb updates reserved-memory by adding these efi loader module areas (see lmb_reserve_common->efi_lmb_reserve).
On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in the device tree (used for comunication with other microcontroller on the SoC, OPTEE, ...) falling in the DDR range between 456 and 512 MB.
Normally U-Boot relocate itself at the end of DDR so it overlap this reserved-memory and this of course does not work [1].
In this case U-Boot prints the following errors:
ERROR: reserving fdt memory region failed (addr=9c800000 size=300000) ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000) ... ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000)
because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve memories (lmb_flag LMB_NOMAP).
To fix this I moved the U-Boot relocation address, implementing board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value with no overlap (448 MB).
The memory map is: +---------------+ 512 MB |DT reserved-mem| +---------------+ 456 MB |free space | +---------------+ 448 MB (ram_top) | | |uboot | | | +---------------+ 0 MB
This is working fine (the board is able to boot, no memory overlaps) ...
however, in this configuration U-Boot we still have some errors prints while loading linux dtb:
ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9cc00000 size=e00000 flags=4) ERROR: reserving fdt memory region failed (addr=9da00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4) ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4) ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
And the reason of these error is complete different: The efi loader lmb memory reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area.
- lib/lmb.c:lmb_reserve_common() reserve these memories:
~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved
area. This is not coalesced/merged with the u-boot reserved area because it is not contiguous.
This is the reserved-memory array at this point: ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- lib/lmb.c:efi_lmb_reserve() reserve some more areas.
In the last two steps efi loader add some reserved area that overlap u-boot and fdt reserved area BUT they are also contiguous to the "efi loader reserved area". The very last step reserve also area between ram_top (448M) to ram_end (512M)
We fall in this condition (where we have overlapping areas!): ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- Now, while loading the linux fdt the reserved-memory areas are checked
toward efi loader reserved area, they overlap, BUT they have different lmb_flag. So we get the ERROR print.
Hopefully this is clear, I undestand is not that obvious ...
Thanks Emanuele for reporting the issue.
Could you, please, provide the output of
efidebug memmap
Verdin AM62 # efidebug memmap MMC: no card present No EFI system partition No EFI system partition Failed to persist EFI variables Type Start End Attributes ================ ================ ================ ========== CONVENTIONAL 0000000080000000-0000000098eec000 WB BOOT DATA 0000000098eec000-0000000098eee000 WB RUNTIME DATA 0000000098eee000-0000000098eef000 WB|RT BOOT DATA 0000000098eef000-0000000098ef2000 WB RUNTIME DATA 0000000098ef2000-0000000098ef3000 WB|RT BOOT DATA 0000000098ef3000-0000000098ef4000 WB RUNTIME DATA 0000000098ef4000-0000000098ef6000 WB|RT BOOT DATA 0000000098ef6000-0000000098ef7000 WB RUNTIME DATA 0000000098ef7000-0000000098f07000 WB|RT BOOT DATA 0000000098f07000-0000000098f10000 WB BOOT CODE 0000000098f10000-000000009bf20000 WB RUNTIME CODE 000000009bf20000-000000009bf30000 WB|RT BOOT CODE 000000009bf30000-000000009c000000 WB BOOT DATA 000000009c000000-00000000a0000000 WB
and of
bdinfo
With this "debug" patch I hope giving you a better picture:
diff --git a/lib/lmb.c b/lib/lmb.c index b2c233edb64e..1feb7189936e 100644 --- a/lib/lmb.c +++ b/lib/lmb.c @@ -197,6 +197,9 @@ static void lmb_reserve_common(struct lmb *lmb, void *fdt_blob) if (CONFIG_IS_ENABLED(OF_LIBFDT) && fdt_blob) boot_fdt_add_mem_rsv_regions(lmb, fdt_blob);
+ printf("%s. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve\n", __func__); + lmb_dump_all_force(lmb); + if (CONFIG_IS_ENABLED(EFI_LOADER)) efi_lmb_reserve(lmb); }
Verdin AM62 # bdinfo boot_params = 0x0000000000000000 DRAM bank = 0x0000000000000000 -> start = 0x0000000080000000 -> size = 0x0000000020000000 flashstart = 0x0000000000000000 flashsize = 0x0000000000000000 flashoffset = 0x0000000000000000 baudrate = 115200 bps relocaddr = 0x000000009bf22000 reloc off = 0x000000001b722000 Build = 64-bit current eth = ethernet@8000000port@1 ethaddr = 00:14:2d:e5:73:c1 IP addr = <NULL> fdt_blob = 0x0000000099f10890 new_fdt = 0x0000000099f10890 fdt_size = 0x000000000000f4e0 multi_dtb_fit= 0x0000000000000000 lmb_reserve_common. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve lmb_dump_all: memory.cnt = 0x1 / max = 0x10 memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 reserved.cnt = 0x3 / max = 0x10 reserved[0] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 reserved[1] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 reserved[2] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 lmb_dump_all: memory.cnt = 0x1 / max = 0x10 memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 reserved.cnt = 0x4 / max = 0x10 reserved[0] [0x98f06000-0x9fffffff], 0x070fa000 bytes flags: 0 reserved[1] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 reserved[2] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 reserved[3] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 devicetree = separate serial addr = 0x0000000002800000 width = 0x0000000000000000 shift = 0x0000000000000002 offset = 0x0000000000000000 clock = 0x0000000002dc6c00 arch_number = 0x0000000000000000 TLB addr = 0x000000009bff0000 irq_sp = 0x0000000099f10880 sp start = 0x0000000099f10880 Early malloc usage: 2a88 / 8000
Best regards
Heinrich
IMO we have two different bugs: - there is nothing that prevent that coalesced area may overlap other areas (while lmb module expect that there aren't overlapping areas) - efi loader (correctly) reserve between ram_top and ram_end BUT this area is reserved by fdt.
One potential solution could be to override efi_add_known_memory() and set ram_top = ram_end, e.g.
- efi_add_conventional_memory_map(ram_start, ram_end, ram_top); + efi_add_conventional_memory_map(ram_start, ram_end, ram_end);
but this does not really seems like something that should ve overridden at the board level.
Any suggestion?
Emanuele
Very interesting the efidebug command... Thank you.
Emanuele
On 7/27/23 14:14, Emanuele Ghidoli wrote:
On 27/07/2023 13:21, Heinrich Schuchardt wrote:
On 27.07.23 13:08, Emanuele Ghidoli wrote:
Efi loader module have its own memory management that flags as reserved the area between ram_top to ram_end (currently where dt reserved-memory is falling). uboot lmb updates reserved-memory by adding these efi loader module areas (see lmb_reserve_common->efi_lmb_reserve).
On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in the device tree (used for comunication with other microcontroller on the SoC, OPTEE, ...) falling in the DDR range between 456 and 512 MB.
Normally U-Boot relocate itself at the end of DDR so it overlap this reserved-memory and this of course does not work [1].
In this case U-Boot prints the following errors:
ERROR: reserving fdt memory region failed (addr=9c800000 size=300000) ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000) ... ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000)
because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve memories (lmb_flag LMB_NOMAP).
To fix this I moved the U-Boot relocation address, implementing board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value with no overlap (448 MB).
The memory map is: +---------------+ 512 MB |DT reserved-mem| +---------------+ 456 MB |free space | +---------------+ 448 MB (ram_top) | | |uboot | | | +---------------+ 0 MB
This is working fine (the board is able to boot, no memory overlaps) ...
however, in this configuration U-Boot we still have some errors prints while loading linux dtb:
ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9cc00000 size=e00000 flags=4) ERROR: reserving fdt memory region failed (addr=9da00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4) ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4) ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
And the reason of these error is complete different: The efi loader lmb memory reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area.
- lib/lmb.c:lmb_reserve_common() reserve these memories:
~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved
area. This is not coalesced/merged with the u-boot reserved area because it is not contiguous.
This is the reserved-memory array at this point: ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- lib/lmb.c:efi_lmb_reserve() reserve some more areas.
In the last two steps efi loader add some reserved area that overlap u-boot and fdt reserved area BUT they are also contiguous to the "efi loader reserved area". The very last step reserve also area between ram_top (448M) to ram_end (512M)
We fall in this condition (where we have overlapping areas!): ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- Now, while loading the linux fdt the reserved-memory areas are checked
toward efi loader reserved area, they overlap, BUT they have different lmb_flag. So we get the ERROR print.
Hopefully this is clear, I undestand is not that obvious ...
Thanks Emanuele for reporting the issue.
Could you, please, provide the output of
efidebug memmap
Verdin AM62 # efidebug memmap MMC: no card present No EFI system partition No EFI system partition Failed to persist EFI variables Type Start End Attributes ================ ================ ================ ========== CONVENTIONAL 0000000080000000-0000000098eec000 WB BOOT DATA 0000000098eec000-0000000098eee000 WB RUNTIME DATA 0000000098eee000-0000000098eef000 WB|RT BOOT DATA 0000000098eef000-0000000098ef2000 WB RUNTIME DATA 0000000098ef2000-0000000098ef3000 WB|RT BOOT DATA 0000000098ef3000-0000000098ef4000 WB RUNTIME DATA 0000000098ef4000-0000000098ef6000 WB|RT BOOT DATA 0000000098ef6000-0000000098ef7000 WB RUNTIME DATA 0000000098ef7000-0000000098f07000 WB|RT BOOT DATA 0000000098f07000-0000000098f10000 WB BOOT CODE 0000000098f10000-000000009bf20000 WB RUNTIME CODE 000000009bf20000-000000009bf30000 WB|RT BOOT CODE 000000009bf30000-000000009c000000 WB BOOT DATA 000000009c000000-00000000a0000000 WB
and of
bdinfo
With this "debug" patch I hope giving you a better picture:
diff --git a/lib/lmb.c b/lib/lmb.c index b2c233edb64e..1feb7189936e 100644 --- a/lib/lmb.c +++ b/lib/lmb.c @@ -197,6 +197,9 @@ static void lmb_reserve_common(struct lmb *lmb, void *fdt_blob) if (CONFIG_IS_ENABLED(OF_LIBFDT) && fdt_blob) boot_fdt_add_mem_rsv_regions(lmb, fdt_blob);
printf("%s. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve\n", __func__);lmb_dump_all_force(lmb);if (CONFIG_IS_ENABLED(EFI_LOADER)) efi_lmb_reserve(lmb);Verdin AM62 # bdinfo boot_params = 0x0000000000000000 DRAM bank = 0x0000000000000000 -> start = 0x0000000080000000 -> size = 0x0000000020000000 flashstart = 0x0000000000000000 flashsize = 0x0000000000000000 flashoffset = 0x0000000000000000 baudrate = 115200 bps relocaddr = 0x000000009bf22000 reloc off = 0x000000001b722000 Build = 64-bit current eth = ethernet@8000000port@1 ethaddr = 00:14:2d:e5:73:c1 IP addr = <NULL> fdt_blob = 0x0000000099f10890 new_fdt = 0x0000000099f10890 fdt_size = 0x000000000000f4e0 multi_dtb_fit= 0x0000000000000000 lmb_reserve_common. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve lmb_dump_all: memory.cnt = 0x1 / max = 0x10 memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 reserved.cnt = 0x3 / max = 0x10 reserved[0] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 reserved[1] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 reserved[2] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 lmb_dump_all: memory.cnt = 0x1 / max = 0x10 memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 reserved.cnt = 0x4 / max = 0x10 reserved[0] [0x98f06000-0x9fffffff], 0x070fa000 bytes flags: 0 reserved[1] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 reserved[2] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 reserved[3] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 devicetree = separate serial addr = 0x0000000002800000 width = 0x0000000000000000 shift = 0x0000000000000002 offset = 0x0000000000000000 clock = 0x0000000002dc6c00 arch_number = 0x0000000000000000 TLB addr = 0x000000009bff0000 irq_sp = 0x0000000099f10880 sp start = 0x0000000099f10880 Early malloc usage: 2a88 / 8000
Best regards
Heinrich
IMO we have two different bugs: - there is nothing that prevent that coalesced area may overlap other areas (while lmb module expect that there aren't overlapping areas) - efi loader (correctly) reserve between ram_top and ram_end BUT this area is reserved by fdt.
One potential solution could be to override efi_add_known_memory() and set ram_top = ram_end, e.g.
- efi_add_conventional_memory_map(ram_start, ram_end, ram_top); + efi_add_conventional_memory_map(ram_start, ram_end, ram_end);
but this does not really seems like something that should ve overridden at the board level.
Any suggestion?
Emanuele
Very interesting the efidebug command... Thank you.
Emanuele
Hello Emanuele,
I am able to reproduce the issue on a Pine A64 LTS with this tree: https://source.denx.de/u-boot/custodians/u-boot-efi/-/tree/top_reserved
Best regards
Heinrich
Hello all, re-opening this old email thread ...
On Sat, Aug 12, 2023 at 08:23:32AM +0200, Heinrich Schuchardt wrote:
On 7/27/23 14:14, Emanuele Ghidoli wrote:
On 27/07/2023 13:21, Heinrich Schuchardt wrote:
On 27.07.23 13:08, Emanuele Ghidoli wrote:
Efi loader module have its own memory management that flags as reserved the area between ram_top to ram_end (currently where dt reserved-memory is falling). uboot lmb updates reserved-memory by adding these efi loader module areas (see lmb_reserve_common->efi_lmb_reserve).
On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in the device tree (used for comunication with other microcontroller on the SoC, OPTEE, ...) falling in the DDR range between 456 and 512 MB.
Normally U-Boot relocate itself at the end of DDR so it overlap this reserved-memory and this of course does not work [1].
In this case U-Boot prints the following errors:
ERROR: reserving fdt memory region failed (addr=9c800000 size=300000) ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000) ... ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000)
because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve memories (lmb_flag LMB_NOMAP).
To fix this I moved the U-Boot relocation address, implementing board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value with no overlap (448 MB).
The memory map is: +---------------+ 512 MB |DT reserved-mem| +---------------+ 456 MB |free space | +---------------+ 448 MB (ram_top) | | |uboot | | | +---------------+ 0 MB
This is working fine (the board is able to boot, no memory overlaps) ...
however, in this configuration U-Boot we still have some errors prints while loading linux dtb:
ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9cc00000 size=e00000 flags=4) ERROR: reserving fdt memory region failed (addr=9da00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4) ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4) ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
And the reason of these error is complete different: The efi loader lmb memory reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area.
- lib/lmb.c:lmb_reserve_common() reserve these memories:
~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved
area. This is not coalesced/merged with the u-boot reserved area because it is not contiguous.
This is the reserved-memory array at this point: ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- lib/lmb.c:efi_lmb_reserve() reserve some more areas.
In the last two steps efi loader add some reserved area that overlap u-boot and fdt reserved area BUT they are also contiguous to the "efi loader reserved area". The very last step reserve also area between ram_top (448M) to ram_end (512M)
We fall in this condition (where we have overlapping areas!): ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- Now, while loading the linux fdt the reserved-memory areas are checked
toward efi loader reserved area, they overlap, BUT they have different lmb_flag. So we get the ERROR print.
Hopefully this is clear, I undestand is not that obvious ...
efidebug memmap
Verdin AM62 # efidebug memmap MMC: no card present No EFI system partition No EFI system partition Failed to persist EFI variables Type Start End Attributes ================ ================ ================ ========== CONVENTIONAL 0000000080000000-0000000098eec000 WB BOOT DATA 0000000098eec000-0000000098eee000 WB RUNTIME DATA 0000000098eee000-0000000098eef000 WB|RT BOOT DATA 0000000098eef000-0000000098ef2000 WB RUNTIME DATA 0000000098ef2000-0000000098ef3000 WB|RT BOOT DATA 0000000098ef3000-0000000098ef4000 WB RUNTIME DATA 0000000098ef4000-0000000098ef6000 WB|RT BOOT DATA 0000000098ef6000-0000000098ef7000 WB RUNTIME DATA 0000000098ef7000-0000000098f07000 WB|RT BOOT DATA 0000000098f07000-0000000098f10000 WB BOOT CODE 0000000098f10000-000000009bf20000 WB RUNTIME CODE 000000009bf20000-000000009bf30000 WB|RT BOOT CODE 000000009bf30000-000000009c000000 WB BOOT DATA 000000009c000000-00000000a0000000 WB
and of
bdinfo
With this "debug" patch I hope giving you a better picture:
diff --git a/lib/lmb.c b/lib/lmb.c index b2c233edb64e..1feb7189936e 100644 --- a/lib/lmb.c +++ b/lib/lmb.c @@ -197,6 +197,9 @@ static void lmb_reserve_common(struct lmb *lmb, void *fdt_blob) if (CONFIG_IS_ENABLED(OF_LIBFDT) && fdt_blob) boot_fdt_add_mem_rsv_regions(lmb, fdt_blob);
printf("%s. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve\n", __func__);lmb_dump_all_force(lmb);if (CONFIG_IS_ENABLED(EFI_LOADER)) efi_lmb_reserve(lmb);Verdin AM62 # bdinfo boot_params = 0x0000000000000000 DRAM bank = 0x0000000000000000 -> start = 0x0000000080000000 -> size = 0x0000000020000000 flashstart = 0x0000000000000000 flashsize = 0x0000000000000000 flashoffset = 0x0000000000000000 baudrate = 115200 bps relocaddr = 0x000000009bf22000 reloc off = 0x000000001b722000 Build = 64-bit current eth = ethernet@8000000port@1 ethaddr = 00:14:2d:e5:73:c1 IP addr = <NULL> fdt_blob = 0x0000000099f10890 new_fdt = 0x0000000099f10890 fdt_size = 0x000000000000f4e0 multi_dtb_fit= 0x0000000000000000 lmb_reserve_common. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve lmb_dump_all: memory.cnt = 0x1 / max = 0x10 memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 reserved.cnt = 0x3 / max = 0x10 reserved[0] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 reserved[1] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 reserved[2] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 lmb_dump_all: memory.cnt = 0x1 / max = 0x10 memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 reserved.cnt = 0x4 / max = 0x10 reserved[0] [0x98f06000-0x9fffffff], 0x070fa000 bytes flags: 0 reserved[1] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 reserved[2] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 reserved[3] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 devicetree = separate serial addr = 0x0000000002800000 width = 0x0000000000000000 shift = 0x0000000000000002 offset = 0x0000000000000000 clock = 0x0000000002dc6c00 arch_number = 0x0000000000000000 TLB addr = 0x000000009bff0000 irq_sp = 0x0000000099f10880 sp start = 0x0000000099f10880 Early malloc usage: 2a88 / 8000
IMO we have two different bugs: - there is nothing that prevent that coalesced area may overlap other areas (while lmb module expect that there aren't overlapping areas) - efi loader (correctly) reserve between ram_top and ram_end BUT this area is reserved by fdt.
One potential solution could be to override efi_add_known_memory() and set ram_top = ram_end, e.g.
- efi_add_conventional_memory_map(ram_start, ram_end, ram_top); + efi_add_conventional_memory_map(ram_start, ram_end, ram_end);
but this does not really seems like something that should ve overridden at the board level.
I am able to reproduce the issue on a Pine A64 LTS with this tree: https://source.denx.de/u-boot/custodians/u-boot-efi/-/tree/top_reserved
Patrice send this patch https://lore.kernel.org/all/20240308101230.2595220-1-patrice.chotard@foss.st... to fix what it looks like the same issue on STM32MP1.
Should we have a similar patch for TI K3? What do you think?
Francesco
participants (3)
-
Emanuele Ghidoli -
Francesco Dolcini -
Heinrich Schuchardt