On 9/5/22 3:47 AM, Nikita Shubin wrote:
Hi Rick!
On Mon, 5 Sep 2022 14:22:41 +0800 Rick Chen rickchen36@gmail.com wrote:
Hi,
When I free-run a SMP system, I once hit a failure case where some harts didn't boot to the kernel shell successfully. However it can't be duplicated anymore even if I try many times.
But when I set a break during debugging with GDB, it can trigger the failure case each time.
If hart fails to register itself to available_harts before send_ipi_many is hit by the main hart: https://elixir.bootlin.com/u-boot/v2022.10-rc3/source/arch/riscv/lib/smp.c#L...
it won't exit the secondary_hart_loop: https://elixir.bootlin.com/u-boot/v2022.10-rc3/source/arch/riscv/cpu/start.S... As no ipi will be sent to it.
This might be exactly your case.
When working on the IPI mechanism, I considered this possibility. However, there's really no way to know how long to wait. On normal systems, the boot hart is going to do a lot of work before calling send_ipi_many, and the other harts just have to make it through ~100 instructions. So I figured we would never run into this issue.
We might not even need the mask... the only direct reason we might is for OpenSBI, as spl_invoke_opensbi is the only function which uses the wait parameter.
I think the mechanism of available_harts does not provide a method that guarantees the success of the SMP system. Maybe we shall think of a better way for the SMP booting or just remove it ?
I haven't experienced any unexplained problem with hart_lottery or available_harts_lock unless:
- harts are started non-simultaneously
- SPL/U-Boot is in some kind of TCM, OCRAM, etc... which is not cleared
on reset which leaves available_harts dirty
XIP, of course, has this problem every time and just doesn't use the mask. I remember thinking a lot about how to deal with this, but I never ended up sending a patch because I didn't have a XIP system.
--Sean
- something is wrong with atomics
Also there might be something wrong with IPI send/recieve.
Thread 8 hit Breakpoint 1, harts_early_init ()
(gdb) c Continuing. [Switching to Thread 7]
Thread 7 hit Breakpoint 1, harts_early_init ()
(gdb) Continuing. [Switching to Thread 6]
Thread 6 hit Breakpoint 1, harts_early_init ()
(gdb) Continuing. [Switching to Thread 5]
Thread 5 hit Breakpoint 1, harts_early_init ()
(gdb) Continuing. [Switching to Thread 4]
Thread 4 hit Breakpoint 1, harts_early_init ()
(gdb) Continuing. [Switching to Thread 3]
Thread 3 hit Breakpoint 1, harts_early_init () (gdb) Continuing. [Switching to Thread 2]
Thread 2 hit Breakpoint 1, harts_early_init () (gdb) Continuing. [Switching to Thread 1]
Thread 1 hit Breakpoint 1, harts_early_init () (gdb) Continuing. [Switching to Thread 5]
Thread 5 hit Breakpoint 3, 0x0000000001200000 in ?? () (gdb) info threads Id Target Id Frame 1 Thread 1 (hart 1) secondary_hart_loop () at arch/riscv/cpu/start.S:436 2 Thread 2 (hart 2) secondary_hart_loop () at arch/riscv/cpu/start.S:436 3 Thread 3 (hart 3) secondary_hart_loop () at arch/riscv/cpu/start.S:436 4 Thread 4 (hart 4) secondary_hart_loop () at arch/riscv/cpu/start.S:436
- 5 Thread 5 (hart 5) 0x0000000001200000 in ?? () 6 Thread 6 (hart 6) 0x000000000000b650 in ?? () 7 Thread 7 (hart 7) 0x000000000000b650 in ?? () 8 Thread 8 (hart 8) 0x0000000000005fa0 in ?? ()
(gdb) c Continuing.
Do they all "offline" harts remain in SPL/U-Boot secondary_hart_loop ?
[ 0.175619] smp: Bringing up secondary CPUs ... [ 1.230474] CPU1: failed to come online [ 2.282349] CPU2: failed to come online [ 3.334394] CPU3: failed to come online [ 4.386783] CPU4: failed to come online [ 4.427829] smp: Brought up 1 node, 4 CPUs
/root # cat /proc/cpuinfo processor : 0 hart : 4 isa : rv64i2p0m2p0a2p0c2p0xv5-1p1 mmu : sv39
processor : 5 hart : 5 isa : rv64i2p0m2p0a2p0c2p0xv5-1p1 mmu : sv39
processor : 6 hart : 6 isa : rv64i2p0m2p0a2p0c2p0xv5-1p1 mmu : sv39
processor : 7 hart : 7 isa : rv64i2p0m2p0a2p0c2p0xv5-1p1 mmu : sv39
/root #
Thanks, Rick