On 27/07/2023 13:21, Heinrich Schuchardt wrote:
On 27.07.23 13:08, Emanuele Ghidoli wrote:
Efi loader module have its own memory management that flags as reserved the area between ram_top to ram_end (currently where dt reserved-memory is falling). uboot lmb updates reserved-memory by adding these efi loader module areas (see lmb_reserve_common->efi_lmb_reserve).
On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in the device tree (used for comunication with other microcontroller on the SoC, OPTEE, ...) falling in the DDR range between 456 and 512 MB.
Normally U-Boot relocate itself at the end of DDR so it overlap this reserved-memory and this of course does not work [1].
In this case U-Boot prints the following errors:
ERROR: reserving fdt memory region failed (addr=9c800000 size=300000) ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000) ... ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000)
because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve memories (lmb_flag LMB_NOMAP).
To fix this I moved the U-Boot relocation address, implementing board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value with no overlap (448 MB).
The memory map is: +---------------+ 512 MB |DT reserved-mem| +---------------+ 456 MB |free space | +---------------+ 448 MB (ram_top) | | |uboot | | | +---------------+ 0 MB
This is working fine (the board is able to boot, no memory overlaps) ...
however, in this configuration U-Boot we still have some errors prints while loading linux dtb:
ERROR: reserving fdt memory region failed (addr=9cb00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9cc00000 size=e00000 flags=4) ERROR: reserving fdt memory region failed (addr=9da00000 size=100000 flags=4) ERROR: reserving fdt memory region failed (addr=9db00000 size=c00000 flags=4) ERROR: reserving fdt memory region failed (addr=9e780000 size=80000 flags=4) ERROR: reserving fdt memory region failed (addr=9e800000 size=1800000 flags=4)
And the reason of these error is complete different: The efi loader lmb memory reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area.
- lib/lmb.c:lmb_reserve_common() reserve these memories:
~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved
area. This is not coalesced/merged with the u-boot reserved area because it is not contiguous.
This is the reserved-memory array at this point: ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- lib/lmb.c:efi_lmb_reserve() reserve some more areas.
In the last two steps efi loader add some reserved area that overlap u-boot and fdt reserved area BUT they are also contiguous to the "efi loader reserved area". The very last step reserve also area between ram_top (448M) to ram_end (512M)
We fall in this condition (where we have overlapping areas!): ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP)
- Now, while loading the linux fdt the reserved-memory areas are checked
toward efi loader reserved area, they overlap, BUT they have different lmb_flag. So we get the ERROR print.
Hopefully this is clear, I undestand is not that obvious ...
Thanks Emanuele for reporting the issue.
Could you, please, provide the output of
efidebug memmap
Verdin AM62 # efidebug memmap MMC: no card present No EFI system partition No EFI system partition Failed to persist EFI variables Type Start End Attributes ================ ================ ================ ========== CONVENTIONAL 0000000080000000-0000000098eec000 WB BOOT DATA 0000000098eec000-0000000098eee000 WB RUNTIME DATA 0000000098eee000-0000000098eef000 WB|RT BOOT DATA 0000000098eef000-0000000098ef2000 WB RUNTIME DATA 0000000098ef2000-0000000098ef3000 WB|RT BOOT DATA 0000000098ef3000-0000000098ef4000 WB RUNTIME DATA 0000000098ef4000-0000000098ef6000 WB|RT BOOT DATA 0000000098ef6000-0000000098ef7000 WB RUNTIME DATA 0000000098ef7000-0000000098f07000 WB|RT BOOT DATA 0000000098f07000-0000000098f10000 WB BOOT CODE 0000000098f10000-000000009bf20000 WB RUNTIME CODE 000000009bf20000-000000009bf30000 WB|RT BOOT CODE 000000009bf30000-000000009c000000 WB BOOT DATA 000000009c000000-00000000a0000000 WB
and of
bdinfo
With this "debug" patch I hope giving you a better picture:
diff --git a/lib/lmb.c b/lib/lmb.c index b2c233edb64e..1feb7189936e 100644 --- a/lib/lmb.c +++ b/lib/lmb.c @@ -197,6 +197,9 @@ static void lmb_reserve_common(struct lmb *lmb, void *fdt_blob) if (CONFIG_IS_ENABLED(OF_LIBFDT) && fdt_blob) boot_fdt_add_mem_rsv_regions(lmb, fdt_blob);
+ printf("%s. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve\n", __func__); + lmb_dump_all_force(lmb); + if (CONFIG_IS_ENABLED(EFI_LOADER)) efi_lmb_reserve(lmb); }
Verdin AM62 # bdinfo boot_params = 0x0000000000000000 DRAM bank = 0x0000000000000000 -> start = 0x0000000080000000 -> size = 0x0000000020000000 flashstart = 0x0000000000000000 flashsize = 0x0000000000000000 flashoffset = 0x0000000000000000 baudrate = 115200 bps relocaddr = 0x000000009bf22000 reloc off = 0x000000001b722000 Build = 64-bit current eth = ethernet@8000000port@1 ethaddr = 00:14:2d:e5:73:c1 IP addr = <NULL> fdt_blob = 0x0000000099f10890 new_fdt = 0x0000000099f10890 fdt_size = 0x000000000000f4e0 multi_dtb_fit= 0x0000000000000000 lmb_reserve_common. Calling lmb_dump_all_force between boot_fdt_add_mem_rsv_regions and efi_lmb_reserve lmb_dump_all: memory.cnt = 0x1 / max = 0x10 memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 reserved.cnt = 0x3 / max = 0x10 reserved[0] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 reserved[1] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 reserved[2] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 lmb_dump_all: memory.cnt = 0x1 / max = 0x10 memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0 reserved.cnt = 0x4 / max = 0x10 reserved[0] [0x98f06000-0x9fffffff], 0x070fa000 bytes flags: 0 reserved[1] [0x99f0c280-0x9bffffff], 0x020f3d80 bytes flags: 0 reserved[2] [0x9db00000-0x9e6fffff], 0x00c00000 bytes flags: 4 reserved[3] [0x9e780000-0x9fffffff], 0x01880000 bytes flags: 4 devicetree = separate serial addr = 0x0000000002800000 width = 0x0000000000000000 shift = 0x0000000000000002 offset = 0x0000000000000000 clock = 0x0000000002dc6c00 arch_number = 0x0000000000000000 TLB addr = 0x000000009bff0000 irq_sp = 0x0000000099f10880 sp start = 0x0000000099f10880 Early malloc usage: 2a88 / 8000
Best regards
Heinrich
IMO we have two different bugs: - there is nothing that prevent that coalesced area may overlap other areas (while lmb module expect that there aren't overlapping areas) - efi loader (correctly) reserve between ram_top and ram_end BUT this area is reserved by fdt.
One potential solution could be to override efi_add_known_memory() and set ram_top = ram_end, e.g.
- efi_add_conventional_memory_map(ram_start, ram_end, ram_top); + efi_add_conventional_memory_map(ram_start, ram_end, ram_end);
but this does not really seems like something that should ve overridden at the board level.
Any suggestion?
Emanuele
Very interesting the efidebug command... Thank you.
Emanuele