On Wed, 7 Jul 2021 at 05:59, Heinrich Schuchardt xypron.glpk@gmx.de wrote:
Am 7. Juli 2021 05:18:20 MESZ schrieb Heinrich Schuchardt xypron.glpk@gmx.de:
Am 7. Juli 2021 03:44:35 MESZ schrieb AKASHI Takahiro takahiro.akashi@linaro.org:
François,
On Tue, Jul 06, 2021 at 08:10:08PM +0200, Heinrich Schuchardt wrote:
On 7/6/21 6:13 PM, François Ozog wrote:
Hi Heinrich, U-Boot 2021-07rc5 does not take into account memory description when using Qemu 5.2 NUMA configuration to adapt memory
map
(kernel_addr_r...):
-smp 4 \ -m 8G,slots=2,maxmem=16G \ -object memory-backend-ram,size=4G,id=m0 \ -object memory-backend-ram,size=4G,id=m1 \ -numa node,cpus=0-1,nodeid=0,memdev=m0 \ -numa node,cpus=2-3,nodeid=1,memdev=m1kernel_addr_r is still 0x4040000 and thus you can't use it to
bootefi.
fdt addr 0x13ede6de0; fdt print
Displays fdt while I think it should not.
If I load the kernel at dram.start, the load works but not boot
U-Boot 2021.07 (Jul 06 2021 - 13:26:43 +0000)
DRAM:4 GiB
Flash: 64 MiB
Loading Environment from Flash... OK
In:pl011@9000000
Out: pl011@9000000
Err: pl011@9000000
Net: eth0: virtio-net#32
Hit any key to stop autoboot:0
=>
=> bdinfo
boot_params = 0x0000000000000000
DRAM bank = 0x0000000000000000
-> start= 0x0000000140000000
-> size = 0x0000000100000000
flashstart= 0x0000000000000000
flashsize = 0x0000000004000000
flashoffset = 0x00000000000bc990
baudrate= 115200 bps
relocaddr = 0x000000013ff27000
reloc off = 0x000000013ff27000
Build = 64-bit
current eth = virtio-net#32
ethaddr = 52:52:52:52:52:52
IP addr = <NULL>
fdt_blob= 0x000000013ede6de0
new_fdt = 0x000000013ede6de0
fdt_size= 0x0000000000100000
lmb_dump_all:
memory.cnt= 0x1
memory.reg[0x0].base = 0x140000000
.size = 0x100000000
reserved.cnt= 0x0
arch_number = 0x0000000000000000
TLB addr= 0x000000013fff0000
irq_sp= 0x000000013ede6dd0
sp start= 0x000000013ede6dd0
Early malloc usage: 3a8 / 2000
=> load virtio 0:1 0x140000000 /oskit.efi
853424 bytes read in 1 ms (813.9 MiB/s)
=> bootefi0x140000000 0x13ede6dd0
ERROR: Failed to register WaitForKey event
Setting OsIndications failed
Error: Cannot initialize UEFI sub-system, r = 9
I think there is a need to calculate memory map based on previous firmware (TFA, QEMU can be considered as previous frimware)
information
(DT or blob_list).
What do you think ?
Cheers
FF
--
François-Frédéric Ozog | /Director Business Development/ T: +33.67221.6485 francois.ozog@linaro.org mailto:francois.ozog@linaro.org
| Skype: ffozog
The kernel load address is hard coded here: include/configs/qemu-arm.h:41: "kernel_addr_r=0x40400000\0" \
bdinfo shows: DRAM start = 0x140000000 DRAM size = 0x100000000
fdt addr $fdt_addr fdt printf
shows two memory areas. One at 40000000, one at 140000000.
(This shows that U-Boot receives a correct memory map via dtb.)
Is this a NUMA machine, isn't it? Why should we care of which memory region be used here? Please note that this is a virtual
machine,
there is no practical difference between two regions.
The root problem is that U-Boot did not recognize there were two memory regions. We can fix this issue in either way:
diff --git a/configs/qemu_arm64_defconfig b/configs/qemu_arm64_defconfig index f6e586627a8e..b70ffae8bf6e 100644 --- a/configs/qemu_arm64_defconfig +++ b/configs/qemu_arm64_defconfig @@ -1,7 +1,7 @@ CONFIG_ARM=y CONFIG_POSITION_INDEPENDENT=y CONFIG_ARCH_QEMU=y -CONFIG_NR_DRAM_BANKS=1 +CONFIG_NR_DRAM_BANKS=2 CONFIG_ENV_SIZE=0x40000 CONFIG_ENV_SECT_SIZE=0x40000 CONFIG_AHCI=y
diff --git a/lib/fdtdec.c b/lib/fdtdec.c index 4b097fb588ed..4067ea2dead6 100644 --- a/lib/fdtdec.c +++ b/lib/fdtdec.c @@ -1111,7 +1111,7 @@ int fdtdec_setup_memory_banksize(void) return -EINVAL; }
for (bank = 0; bank < CONFIG_NR_DRAM_BANKS; bank++) {
for (bank = 0; ; bank++) { ret = ofnode_read_resource(mem, reg++, &res); if (ret < 0) { reg = 0;(fdtdec_setup_memory_banksize() is called in dram_init_banksize().)
(2) seems much better, but I don't know why we had to use CONFIG_NR_DRAM_BANKS here.
2) alone does not work as other places in the code refer to CONFIG_NR_DRAM_BANKS. Setting ...BANKS to 32 makes my code work and bdinfo seems now correct:
=> bdinfo boot_params = 0x0000000000000000 DRAM bank = 0x0000000000000000 -> start = 0x0000000140000000 -> size = 0x0000000100000000 DRAM bank = 0x0000000000000001 -> start = 0x0000000040000000 -> size = 0x0000000100000000 flashstart = 0x0000000000000000 flashsize = 0x0000000004000000 flashoffset = 0x00000000000bcb88 baudrate = 115200 bps relocaddr = 0x000000013ff27000 reloc off = 0x000000013ff27000 Build = 64-bit current eth = virtio-net#32 ethaddr = 52:52:52:52:52:52 IP addr = <NULL> fdt_blob = 0x000000013ede6cf0 new_fdt = 0x000000013ede6cf0 fdt_size = 0x0000000000100000 lmb_dump_all: memory.cnt = 0x1 memory.reg[0x0].base = 0x40000000 .size = 0x200000000 reserved.cnt = 0x1 reserved.reg[0x0].base = 0x13ede58f0 .size = 0x121a710 arch_number = 0x0000000000000000 TLB addr = 0x000000013fff0000 irq_sp = 0x000000013ede6ce0 sp start = 0x000000013ede6ce0 Early malloc usage: 3a8 / 2000
May I suggest you propose a combined patch Akashi-san? If we assume NUMA systems to be tested up to 8 nodes to mimic real existing enterprise hardware and up to 4 memory slots (say for memory hot plugging tests) what about a default value of 32? Alternatively, we could set this value to a much higher one if the costs are negligible.
In this case, other occurrences of CONFIG_NR_DRAM_BANKS in this file should be replaced with a variable for it.
Your use case is well beyond the typical U-Boot usage. So I guess it will be up to Linaro to provide the necessary patches:
- determine the active CPU
- determine the RAM assigned to the active CPU according to the numa-node-id in the device-tree
- make sure that U-Boot only uses the memory of the active CPU internally
- make sure that the UEFI memory map contains a compliant
description
- possibly, dynamically set up the environment variables
+CC Tuomas Tynkkynen (maintainer for qemu_arm64_defconfig)
For (1), we'd better have a different config, or increase the value of CONFIG_NR_DRAM_BANKS to a bigger number?
Is the system configured such that each CPU can access the others CPU's RAM when entering U-Boot?
Best regards
Heinrich
At least the comments for this patch sound as if on a physical system cross NUMA node memory access is only available after full SMP initialization:
https://patchwork.kernel.org/project/linux-acpi/patch/20180625130552.5636-1-...
QEMU may be less restrictive.
QEMU allows the node distance to be 255 indicating that cross node access is infeasible.
Best regards
Heinrich
-Takahiro Akashi
Best regards
Heinrich