OpenBSI and U-Boot
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S. Here like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to an enless loop (hang).
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang() in lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI payload.
@Ard: Does Linux take care of the hanging harts and redirect them to its own routine before SetVirtualAddressMap()? Otherwise anything could happen.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that we have to implement the hart lottery at the entry point of main U-Boot in this case?
Best regards
Heinrich
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
I think the real start is in firmware/fw_base.S. From there, secondary harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage when SMP is enabled. smp_call_function has all harts execute that function.
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang() in lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI payload.
@Ard: Does Linux take care of the hanging harts and redirect them to its own routine before SetVirtualAddressMap()? Otherwise anything could happen.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that we have to implement the hart lottery at the entry point of main U-Boot in this case?
Isn't the hart lottery already implemented for U-Boot? E.g. around line 100 of arch/riscv/cpu/start.S.
On another note, does Linux support S-Mode NOMMU? I was under the impression that NOMMU implied M-Mode (or the other way around).
--Sean
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
I think the real start is in firmware/fw_base.S. From there, secondary harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage when SMP is enabled. smp_call_function has all harts execute that function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()? How do we ensure that an UEFI payload does not overwrite this memory location?
spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to secondary_hart_relocate() fails.
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang() in lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI payload.
@Ard: Does Linux take care of the hanging harts and redirect them to its own routine before SetVirtualAddressMap()? Otherwise anything could happen.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that we have to implement the hart lottery at the entry point of main U-Boot in this case?
Isn't the hart lottery already implemented for U-Boot? E.g. around line 100 of arch/riscv/cpu/start.S.
Thanks for the hint.
On another note, does Linux support S-Mode NOMMU? I was under the impression that NOMMU implied M-Mode (or the other way around).
Have a look at
https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/...
Best regards
Heinrich
On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
I think the real start is in firmware/fw_base.S. From there, secondary harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage when SMP is enabled. smp_call_function has all harts execute that function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()?
Yes. They also need handle_ipi, and by extension riscv_clear_ipi. This latter function currently requires that gd_t be valid, and may require other structures (e.g. a struct udevice) to be valid in the future.
How do we ensure that an UEFI payload does not overwrite this memory location?
The most foolproof is probably to wait for all harts to start running UEFI code before making any modifications to ram outside the binary. One easy way to do this is to use amoadd instead of amoswap (e.g. a semaphor and not a mutex) in the standard boot lottery code. Whichever hart gets to it first then waits for the value of hart_lottery to reach the expected number of harts.
spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to secondary_hart_relocate() fails.
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang() in lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI payload.
@Ard: Does Linux take care of the hanging harts and redirect them to its own routine before SetVirtualAddressMap()? Otherwise anything could happen.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that we have to implement the hart lottery at the entry point of main U-Boot in this case?
Isn't the hart lottery already implemented for U-Boot? E.g. around line 100 of arch/riscv/cpu/start.S.
Thanks for the hint.
On another note, does Linux support S-Mode NOMMU? I was under the impression that NOMMU implied M-Mode (or the other way around).
Have a look at
https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/...
Ok, so NOMMU implies M-Mode. Then use of OpenSBI should preclude Linux.
--Sean
On 8/8/20 7:22 PM, Sean Anderson wrote:
On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
I think the real start is in firmware/fw_base.S. From there, secondary harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage when SMP is enabled. smp_call_function has all harts execute that function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()?
Yes. They also need handle_ipi, and by extension riscv_clear_ipi. This latter function currently requires that gd_t be valid, and may require other structures (e.g. a struct udevice) to be valid in the future.
How do we ensure that an UEFI payload does not overwrite this memory location?
The most foolproof is probably to wait for all harts to start running UEFI code before making any modifications to ram outside the binary. One easy way to do this is to use amoadd instead of amoswap (e.g. a semaphor and not a mutex) in the standard boot lottery code. Whichever hart gets to it first then waits for the value of hart_lottery to reach the expected number of harts.
There is no such requirement in the UEFI specification.
The way to tell which memory should not be overwritten by the UEFI payload is an entry in the memory map that the payload can read via GetMemoryMap(). So we have to make reservations in the memory map, by calling efi_add_memory_map() or by putting the code into the __efi_runtime section.
Do I understand it correctly that the secondary harts stay in the unrelocated secondary_hart_loop()? In this case __efi_runtime would not be the right section, because that memory section is also relocated and only the relocated code is reserved in the memory map.
We would also have to consider the location of secondary_hart_loop() when defining the load address of any payload be it UEFI or not.
Best regards
Heinrich
spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to secondary_hart_relocate() fails.
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang() in lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI payload.
@Ard: Does Linux take care of the hanging harts and redirect them to its own routine before SetVirtualAddressMap()? Otherwise anything could happen.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that we have to implement the hart lottery at the entry point of main U-Boot in this case?
Isn't the hart lottery already implemented for U-Boot? E.g. around line 100 of arch/riscv/cpu/start.S.
Thanks for the hint.
On another note, does Linux support S-Mode NOMMU? I was under the impression that NOMMU implied M-Mode (or the other way around).
Have a look at
https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/...
Ok, so NOMMU implies M-Mode. Then use of OpenSBI should preclude Linux.
--Sean
On 8/8/20 2:56 PM, Heinrich Schuchardt wrote:
On 8/8/20 7:22 PM, Sean Anderson wrote:
On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
I think the real start is in firmware/fw_base.S. From there, secondary harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage when SMP is enabled. smp_call_function has all harts execute that function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()?
Yes. They also need handle_ipi, and by extension riscv_clear_ipi. This latter function currently requires that gd_t be valid, and may require other structures (e.g. a struct udevice) to be valid in the future.
How do we ensure that an UEFI payload does not overwrite this memory location?
The most foolproof is probably to wait for all harts to start running UEFI code before making any modifications to ram outside the binary. One easy way to do this is to use amoadd instead of amoswap (e.g. a semaphor and not a mutex) in the standard boot lottery code. Whichever hart gets to it first then waits for the value of hart_lottery to reach the expected number of harts.
There is no such requirement in the UEFI specification.
Hm, well perhaps there should be a shim (or patch to U-Boot) which implements such a requirement. AFAIK (and please correct me if there is another option) there is no way to communicate between harts except by interrupt or shared memory. Interrupts may require substantial code to handle properly (which could be difficult to trace the requirements of). However, shared memory only requires one or two functions to be valid, plus the memory location itself. I think that solution has been mostly avoided by U-Boot since the rest of U-Boot is not designed for SMP, so there is not much infrastructure for atomics, etc.
The way to tell which memory should not be overwritten by the UEFI payload is an entry in the memory map that the payload can read via GetMemoryMap(). So we have to make reservations in the memory map, by calling efi_add_memory_map() or by putting the code into the __efi_runtime section.
Do I understand it correctly that the secondary harts stay in the unrelocated secondary_hart_loop()? In this case __efi_runtime would not be the right section, because that memory section is also relocated and only the relocated code is reserved in the memory map.
The other harts get relocated in relocate_secondary_harts.
We would also have to consider the location of secondary_hart_loop() when defining the load address of any payload be it UEFI or not.
Yes. I was looking at your patch to add load addresses [1], and I'm concerned that 80400000 may be too high for the average kernel. Since U-Boot relocates to just below 80600000, that leaves only 1.5M or so for the kernel to be loaded into.
--Sean
[1] https://patchwork.ozlabs.org/project/uboot/patch/20200729154235.90766-4-xypr...
Am 9. August 2020 03:48:19 MESZ schrieb Sean Anderson seanga2@gmail.com:
On 8/8/20 2:56 PM, Heinrich Schuchardt wrote:
On 8/8/20 7:22 PM, Sean Anderson wrote:
On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S.
Here
I think the real start is in firmware/fw_base.S. From there,
secondary
harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one
hart in to
an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage
when
SMP is enabled. smp_call_function has all harts execute that
function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()?
Yes. They also need handle_ipi, and by extension riscv_clear_ipi.
This
latter function currently requires that gd_t be valid, and may
require
other structures (e.g. a struct udevice) to be valid in the future.
How do we ensure that an UEFI payload does not overwrite this
memory location?
The most foolproof is probably to wait for all harts to start
running
UEFI code before making any modifications to ram outside the binary.
One
easy way to do this is to use amoadd instead of amoswap (e.g. a
semaphor
and not a mutex) in the standard boot lottery code. Whichever hart
gets
to it first then waits for the value of hart_lottery to reach the expected number of harts.
There is no such requirement in the UEFI specification.
Hm, well perhaps there should be a shim (or patch to U-Boot) which implements such a requirement. AFAIK (and please correct me if there is another option) there is no way to communicate between harts except by interrupt or shared memory. Interrupts may require substantial code to handle properly (which could be difficult to trace the requirements of). However, shared memory only requires one or two functions to be valid, plus the memory location itself. I think that solution has been mostly avoided by U-Boot since the rest of U-Boot is not designed for SMP, so there is not much infrastructure for atomics, etc.
The way to tell which memory should not be overwritten by the UEFI payload is an entry in the memory map that the payload can read via GetMemoryMap(). So we have to make reservations in the memory map, by calling efi_add_memory_map() or by putting the code into the __efi_runtime section.
Do I understand it correctly that the secondary harts stay in the unrelocated secondary_hart_loop()? In this case __efi_runtime would
not
be the right section, because that memory section is also relocated
and
only the relocated code is reserved in the memory map.
The other harts get relocated in relocate_secondary_harts.
We would also have to consider the location of secondary_hart_loop() when defining the load address of any payload be it UEFI or not.
Yes. I was looking at your patch to add load addresses [1], and I'm concerned that 80400000 may be too high for the average kernel. Since U-Boot relocates to just below 80600000, that leaves only 1.5M or so for the kernel to be loaded into.
The patch has:
kernel_addr_r=0x80040000
You somehow missed the position of "4".
--Sean
[1] https://patchwork.ozlabs.org/project/uboot/patch/20200729154235.90766-4-xypr...
On 8/9/20 12:23 AM, Heinrich Schuchardt wrote:
Am 9. August 2020 03:48:19 MESZ schrieb Sean Anderson seanga2@gmail.com:
On 8/8/20 2:56 PM, Heinrich Schuchardt wrote:
On 8/8/20 7:22 PM, Sean Anderson wrote:
On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote: > Hello Anup, > > I have looking at you OpenSBI code firmware/payloads/test_head.S.
Here
I think the real start is in firmware/fw_base.S. From there,
secondary
harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
> like in U-Boot's common/spl/spl_opensbi.c you put all but one
hart in to
> an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage
when
SMP is enabled. smp_call_function has all harts execute that
function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()?
Yes. They also need handle_ipi, and by extension riscv_clear_ipi.
This
latter function currently requires that gd_t be valid, and may
require
other structures (e.g. a struct udevice) to be valid in the future.
How do we ensure that an UEFI payload does not overwrite this
memory location?
The most foolproof is probably to wait for all harts to start
running
UEFI code before making any modifications to ram outside the binary.
One
easy way to do this is to use amoadd instead of amoswap (e.g. a
semaphor
and not a mutex) in the standard boot lottery code. Whichever hart
gets
to it first then waits for the value of hart_lottery to reach the expected number of harts.
There is no such requirement in the UEFI specification.
Hm, well perhaps there should be a shim (or patch to U-Boot) which implements such a requirement. AFAIK (and please correct me if there is another option) there is no way to communicate between harts except by interrupt or shared memory. Interrupts may require substantial code to handle properly (which could be difficult to trace the requirements of). However, shared memory only requires one or two functions to be valid, plus the memory location itself. I think that solution has been mostly avoided by U-Boot since the rest of U-Boot is not designed for SMP, so there is not much infrastructure for atomics, etc.
The way to tell which memory should not be overwritten by the UEFI payload is an entry in the memory map that the payload can read via GetMemoryMap(). So we have to make reservations in the memory map, by calling efi_add_memory_map() or by putting the code into the __efi_runtime section.
Do I understand it correctly that the secondary harts stay in the unrelocated secondary_hart_loop()? In this case __efi_runtime would
not
be the right section, because that memory section is also relocated
and
only the relocated code is reserved in the memory map.
The other harts get relocated in relocate_secondary_harts.
We would also have to consider the location of secondary_hart_loop() when defining the load address of any payload be it UEFI or not.
Yes. I was looking at your patch to add load addresses [1], and I'm concerned that 80400000 may be too high for the average kernel. Since U-Boot relocates to just below 80600000, that leaves only 1.5M or so for the kernel to be loaded into.
The patch has:
kernel_addr_r=0x80040000
You somehow missed the position of "4".
Ah, so I did. That looks fine then.
--Sean
On 8/9/20 3:48 AM, Sean Anderson wrote:
On 8/8/20 2:56 PM, Heinrich Schuchardt wrote:
On 8/8/20 7:22 PM, Sean Anderson wrote:
On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
I think the real start is in firmware/fw_base.S. From there, secondary harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage when SMP is enabled. smp_call_function has all harts execute that function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()?
Yes. They also need handle_ipi, and by extension riscv_clear_ipi. This latter function currently requires that gd_t be valid, and may require other structures (e.g. a struct udevice) to be valid in the future.
How do we ensure that an UEFI payload does not overwrite this memory location?
The most foolproof is probably to wait for all harts to start running UEFI code before making any modifications to ram outside the binary. One easy way to do this is to use amoadd instead of amoswap (e.g. a semaphor and not a mutex) in the standard boot lottery code. Whichever hart gets to it first then waits for the value of hart_lottery to reach the expected number of harts.
There is no such requirement in the UEFI specification.
Hm, well perhaps there should be a shim (or patch to U-Boot) which implements such a requirement. AFAIK (and please correct me if there is another option) there is no way to communicate between harts except by interrupt or shared memory. Interrupts may require substantial code to handle properly (which could be difficult to trace the requirements of). However, shared memory only requires one or two functions to be valid, plus the memory location itself. I think that solution has been mostly avoided by U-Boot since the rest of U-Boot is not designed for SMP, so there is not much infrastructure for atomics, etc.
The way to tell which memory should not be overwritten by the UEFI payload is an entry in the memory map that the payload can read via GetMemoryMap(). So we have to make reservations in the memory map, by calling efi_add_memory_map() or by putting the code into the __efi_runtime section.
Do I understand it correctly that the secondary harts stay in the unrelocated secondary_hart_loop()? In this case __efi_runtime would not be the right section, because that memory section is also relocated and only the relocated code is reserved in the memory map.
The other harts get relocated in relocate_secondary_harts.
I added the following:
secondary_hart_loop: + + lui a5,0x20040 + addi a5,a5,0x48c + slli a5,a5,0x2 + /* a5 = 80101230 */ + auipc a4, 0 + sw a4,0(a5) + lw a4,8(a5) + addw a4,a4,1 + sw a4,8(a5) +
With the mm.w command I can see what is stored at 0x80101230 - 0x8010123f.
On the MaixDuino secondary_hart_loop() is executed 1 to 2 times. Shouldn't it be running all the time?
relocate_secondary_harts() is called once.
hang() is never executed.
Any idea what the secondary hart is executing after relocate_secondary_harts()?
Best regards
Heinrich
We would also have to consider the location of secondary_hart_loop() when defining the load address of any payload be it UEFI or not.
Yes. I was looking at your patch to add load addresses [1], and I'm concerned that 80400000 may be too high for the average kernel. Since U-Boot relocates to just below 80600000, that leaves only 1.5M or so for the kernel to be loaded into.
--Sean
[1] https://patchwork.ozlabs.org/project/uboot/patch/20200729154235.90766-4-xypr...
On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
I think the real start is in firmware/fw_base.S. From there, secondary harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage when SMP is enabled. smp_call_function has all harts execute that function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()? How do we ensure that an UEFI payload does not overwrite this memory location?
If you are booting Linux, U-Boot runs in S-mode and SMP is disabled. All memory regions containing OpenSBI code are reserved. Thus, U-Boot won't touch that. U-Boot also marks the run time services memory region as reserved as well. Thus, Linux kernel won't touch any of those memory regions either.
spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to secondary_hart_relocate() fails.
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang() in lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI payload.
@Ard: Does Linux take care of the hanging harts and redirect them to its own routine before SetVirtualAddressMap()? Otherwise anything could happen.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that we have to implement the hart lottery at the entry point of main U-Boot in this case?
Isn't the hart lottery already implemented for U-Boot? E.g. around line 100 of arch/riscv/cpu/start.S.
Thanks for the hint.
On another note, does Linux support S-Mode NOMMU? I was under the impression that NOMMU implied M-Mode (or the other way around).
Have a look at
https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/...
Best regards
Heinrich
Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra atishp@atishpatra.org:
On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S.
Here
I think the real start is in firmware/fw_base.S. From there,
secondary
harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
in to
an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage
when
SMP is enabled. smp_call_function has all harts execute that
function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()? How do we ensure that an UEFI payload does not overwrite this memory location?
If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
I am wondering if that endless loop needs to be marked as reserved memory for Linux.
All memory regions containing OpenSBI code are reserved. Thus, U-Boot won't touch that. U-Boot also marks the run time services memory region as reserved as well. Thus, Linux kernel won't touch any of those memory regions either.
spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to secondary_hart_relocate() fails.
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang()
in
lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
EFI
payload.
@Ard: Does Linux take care of the hanging harts and redirect them to its
own
routine before SetVirtualAddressMap()? Otherwise anything could
happen.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
that
we have to implement the hart lottery at the entry point of main
U-Boot
in this case?
Isn't the hart lottery already implemented for U-Boot? E.g. around
line
100 of arch/riscv/cpu/start.S.
Thanks for the hint.
On another note, does Linux support S-Mode NOMMU? I was under the impression that NOMMU implied M-Mode (or the other way around).
Have a look at
https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/...
Best regards
Heinrich
On Sun, Aug 9, 2020 at 1:28 PM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra atishp@atishpatra.org:
On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S.
Here
I think the real start is in firmware/fw_base.S. From there,
secondary
harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
in to
an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage
when
SMP is enabled. smp_call_function has all harts execute that
function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()? How do we ensure that an UEFI payload does not overwrite this memory location?
If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
I am wondering if that endless loop needs to be marked as reserved memory for Linux.
Is U-boot running in S-mode or M-mode ? Are you trying to boot Linux on a Kendryte board using OpenSBI ->U-Boot -> Linux? (all in M-mode ?)
All memory regions containing OpenSBI code are reserved. Thus, U-Boot won't touch that. U-Boot also marks the run time services memory region as reserved as well. Thus, Linux kernel won't touch any of those memory regions either.
spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to secondary_hart_relocate() fails.
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang()
in
lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
EFI
payload.
@Ard: Does Linux take care of the hanging harts and redirect them to its
own
routine before SetVirtualAddressMap()? Otherwise anything could
happen.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
that
we have to implement the hart lottery at the entry point of main
U-Boot
in this case?
Isn't the hart lottery already implemented for U-Boot? E.g. around
line
100 of arch/riscv/cpu/start.S.
Thanks for the hint.
On another note, does Linux support S-Mode NOMMU? I was under the impression that NOMMU implied M-Mode (or the other way around).
Have a look at
https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/...
Best regards
Heinrich
Hi Heinrich
Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra atishp@atishpatra.org:
On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S.
Here
I think the real start is in firmware/fw_base.S. From there,
secondary
harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
in to
an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage
when
SMP is enabled. smp_call_function has all harts execute that
function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()? How do we ensure that an UEFI payload does not overwrite this memory location?
If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
I am wondering if that endless loop needs to be marked as reserved memory for Linux.
Currently if U-Boot run in S-Mode, SMP is disable, there will exist OpenSBI version compatible issue. You shall use OpenSBI v0.7 with HSM, thus it will trap the other harts in OpenSBI and only main hart will jump to kernel from U-Boot proper.
Thanks, Rick
All memory regions containing OpenSBI code are reserved. Thus, U-Boot won't touch that. U-Boot also marks the run time services memory region as reserved as well. Thus, Linux kernel won't touch any of those memory regions either.
spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to secondary_hart_relocate() fails.
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang()
in
lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
EFI
payload.
@Ard: Does Linux take care of the hanging harts and redirect them to its
own
routine before SetVirtualAddressMap()? Otherwise anything could
happen.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
that
we have to implement the hart lottery at the entry point of main
U-Boot
in this case?
Isn't the hart lottery already implemented for U-Boot? E.g. around
line
100 of arch/riscv/cpu/start.S.
Thanks for the hint.
On another note, does Linux support S-Mode NOMMU? I was under the impression that NOMMU implied M-Mode (or the other way around).
Have a look at
https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/...
Best regards
Heinrich
On 8/11/20 3:55 AM, Rick Chen wrote:
Hi Heinrich
Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra atishp@atishpatra.org:
On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S.
Here
I think the real start is in firmware/fw_base.S. From there,
secondary
harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
in to
an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage
when
SMP is enabled. smp_call_function has all harts execute that
function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()? How do we ensure that an UEFI payload does not overwrite this memory location?
If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
I am wondering if that endless loop needs to be marked as reserved memory for Linux.
Currently if U-Boot run in S-Mode, SMP is disable, there will exist OpenSBI version compatible issue. You shall use OpenSBI v0.7 with HSM, thus it will trap the other harts in OpenSBI and only main hart will jump to kernel from U-Boot proper.
Thanks, Rick
HSM is described here: https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.adoc#hart-state...
Currently the Kendryte K210 has no SPL. So OpenSBI would be running before U-Boot.
Will OpenSBI v0.7 by itself stop the other harts or do we need to call sbi_hart_stop() in U-Boot? Unfortunately opensbi/README.md and riscv-sbi.adoc both leave this open.
Best regards
Heinrich
All memory regions containing OpenSBI code are reserved. Thus, U-Boot won't touch that. U-Boot also marks the run time services memory region as reserved as well. Thus, Linux kernel won't touch any of those memory regions either.
spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to secondary_hart_relocate() fails.
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang()
in
lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
EFI
payload.
@Ard: Does Linux take care of the hanging harts and redirect them to its
own
routine before SetVirtualAddressMap()? Otherwise anything could
happen.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
that
we have to implement the hart lottery at the entry point of main
U-Boot
in this case?
Isn't the hart lottery already implemented for U-Boot? E.g. around
line
100 of arch/riscv/cpu/start.S.
Thanks for the hint.
On another note, does Linux support S-Mode NOMMU? I was under the impression that NOMMU implied M-Mode (or the other way around).
Have a look at
https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/...
Best regards
Heinrich
On Mon, Aug 10, 2020 at 10:30 PM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 8/11/20 3:55 AM, Rick Chen wrote:
Hi Heinrich
Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra atishp@atishpatra.org:
On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote:
On 8/8/20 10:59 AM, Heinrich Schuchardt wrote: > Hello Anup, > > I have looking at you OpenSBI code firmware/payloads/test_head.S.
Here
I think the real start is in firmware/fw_base.S. From there,
secondary
harts loop first in _wait_relocate_copy_done, and then in _wait_for_boot_hart, and then they execute the next stage via _start_warm and sbi_init.
> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
in to
> an enless loop (hang).
As far as I can tell, U-Boot has all harts execute the next stage
when
SMP is enabled. smp_call_function has all harts execute that
function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()? How do we ensure that an UEFI payload does not overwrite this memory location?
If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
I am wondering if that endless loop needs to be marked as reserved memory for Linux.
Currently if U-Boot run in S-Mode, SMP is disable, there will exist OpenSBI version compatible issue. You shall use OpenSBI v0.7 with HSM, thus it will trap the other harts in OpenSBI and only main hart will jump to kernel from U-Boot proper.
Thanks, Rick
HSM is described here: https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.adoc#hart-state...
Currently the Kendryte K210 has no SPL. So OpenSBI would be running before U-Boot.
Will OpenSBI v0.7 by itself stop the other harts or do we need to call sbi_hart_stop() in U-Boot? Unfortunately opensbi/README.md and riscv-sbi.adoc both leave this open.
OpenSBI will put all non-booting harts into wfi in sbi_hsm_wait https://github.com/riscv/opensbi/blob/master/lib/sbi/sbi_hsm.c#L119
The S-mode OS is required to call sbi_hsm_start for all non-booting harts so that they enter S-mode.
Are you planning to add HSM extension just for Kendryte ?
Best regards
Heinrich
All memory regions containing OpenSBI code are reserved. Thus, U-Boot won't touch that. U-Boot also marks the run time services memory region as reserved as well. Thus, Linux kernel won't touch any of those memory regions either.
spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to secondary_hart_relocate() fails.
> > When Linux boots via UEFI it will wake up the extra harts after > ExitBootServices(). So I assume we should define function hang()
in
> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
EFI
> payload. > > @Ard: > Does Linux take care of the hanging harts and redirect them to its
own
> routine before SetVirtualAddressMap()? Otherwise anything could
happen.
> > On the Kendryte K210 we don't have SPL. So we will not boot in the > sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
that
> we have to implement the hart lottery at the entry point of main
U-Boot
> in this case?
Isn't the hart lottery already implemented for U-Boot? E.g. around
line
100 of arch/riscv/cpu/start.S.
Thanks for the hint.
On another note, does Linux support S-Mode NOMMU? I was under the impression that NOMMU implied M-Mode (or the other way around).
Have a look at
https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/...
Best regards
Heinrich
Hi Atish
On Mon, Aug 10, 2020 at 10:30 PM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 8/11/20 3:55 AM, Rick Chen wrote:
Hi Heinrich
Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra atishp@atishpatra.org:
On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 8/8/20 5:32 PM, Sean Anderson wrote: > On 8/8/20 10:59 AM, Heinrich Schuchardt wrote: >> Hello Anup, >> >> I have looking at you OpenSBI code firmware/payloads/test_head.S.
Here
> > I think the real start is in firmware/fw_base.S. From there,
secondary
> harts loop first in _wait_relocate_copy_done, and then in > _wait_for_boot_hart, and then they execute the next stage via > _start_warm and sbi_init. > >> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
in to
>> an enless loop (hang). > > As far as I can tell, U-Boot has all harts execute the next stage
when
> SMP is enabled. smp_call_function has all harts execute that
function.
U-Boot can only run on one hart. Are the other harts trapped in secondary_hart_loop()? How do we ensure that an UEFI payload does not overwrite this memory location?
If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
I am wondering if that endless loop needs to be marked as reserved memory for Linux.
Currently if U-Boot run in S-Mode, SMP is disable, there will exist OpenSBI version compatible issue. You shall use OpenSBI v0.7 with HSM, thus it will trap the other harts in OpenSBI and only main hart will jump to kernel from U-Boot proper.
Thanks, Rick
HSM is described here: https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.adoc#hart-state...
Currently the Kendryte K210 has no SPL. So OpenSBI would be running before U-Boot.
Will OpenSBI v0.7 by itself stop the other harts or do we need to call sbi_hart_stop() in U-Boot? Unfortunately opensbi/README.md and riscv-sbi.adoc both leave this open.
OpenSBI will put all non-booting harts into wfi in sbi_hsm_wait https://github.com/riscv/opensbi/blob/master/lib/sbi/sbi_hsm.c#L119
The S-mode OS is required to call sbi_hsm_start for all non-booting harts so that they enter S-mode.
Are you planning to add HSM extension just for Kendryte ?
I have finished this part of jobs internally but still need some time to do code cleanup and prepare patchs.
Thanks, Rick
Best regards
Heinrich
All memory regions containing OpenSBI code are reserved. Thus, U-Boot won't touch that. U-Boot also marks the run time services memory region as reserved as well. Thus, Linux kernel won't touch any of those memory regions either.
spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to secondary_hart_relocate() fails.
> >> >> When Linux boots via UEFI it will wake up the extra harts after >> ExitBootServices(). So I assume we should define function hang()
in
>> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
EFI
>> payload. >> >> @Ard: >> Does Linux take care of the hanging harts and redirect them to its
own
>> routine before SetVirtualAddressMap()? Otherwise anything could
happen.
>> >> On the Kendryte K210 we don't have SPL. So we will not boot in the >> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
that
>> we have to implement the hart lottery at the entry point of main
U-Boot
>> in this case? > > Isn't the hart lottery already implemented for U-Boot? E.g. around
line
> 100 of arch/riscv/cpu/start.S.
Thanks for the hint.
> > On another note, does Linux support S-Mode NOMMU? I was under the > impression that NOMMU implied M-Mode (or the other way around).
Have a look at
https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/...
Best regards
Heinrich
-- Regards, Atish
On Sat, Aug 8, 2020 at 7:59 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
Hello Anup,
I have looking at you OpenSBI code firmware/payloads/test_head.S. Here like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to an enless loop (hang).
When Linux boots via UEFI it will wake up the extra harts after ExitBootServices(). So I assume we should define function hang() in lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI payload.
Proper Linux will wake up non-boot harts through SBI HSM APIs. Until then all other harts are held on OpenSBI in sbi_hsm_wait() [1].
[1] https://github.com/riscv/opensbi/blob/master/lib/sbi/sbi_hsm.c#L133
Here are the details on SBI HSM extension if you are interested. https://fosdem.org/2020/schedule/event/riscv_bootflow/
@Ard: Does Linux take care of the hanging harts and redirect them to its own routine before SetVirtualAddressMap()? Otherwise anything could happen.
Linux doesn't need to as SetVirtualAddressMap is part of efistub which is only executed in the booting hart.
On the Kendryte K210 we don't have SPL. So we will not boot in the sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that we have to implement the hart lottery at the entry point of main U-Boot in this case?
Do you want OpenSBI->U-Boot or OpenSBI->U-Boot->Linux for kendryte ? Is this related to testing U-Boot EFI on Kendryte right ?
Best regards
Heinrich
participants (4)
-
Atish Patra -
Heinrich Schuchardt -
Rick Chen -
Sean Anderson