Re: [U-Boot] Hart lottery and CONFIG_XIP
On Jun 2, 2019, at 9:22 PM, Rick Chen rickchen36@gmail.com wrote:
Hi Troy
From: Troy Benjegerdes [mailto:troy.benjegerdes@sifive.com] Sent: Saturday, June 01, 2019 12:24 AM To: Auer, Lukas Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn@gmail.com; palmer@sifive.com Subject: Re: Hart lottery and CONFIG_XIP
On May 31, 2019, at 9:13 AM, Auer, Lukas lukas.auer@aisec.fraunhofer.de
wrote:
Hi Troy,
On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote:
Wouldn't the following line in head.S fail when running from flash (although maybe not in a way that prevents booting)
/* save the boot hart id to global_data */ SREG tp, GD_BOOT_HART(gp)Shouldn’t this be protected by CONFIG_XIP?
The boot hart ID is stored in global data, which is allocated from the stack (in board_init_f_alloc_reserve() ). It is therefore writable and won't cause any issues when running from flash.
Sorry about the confusion, I was reading it wrong earlier.
I’m hoping to have a couple of patches ready later today to change the CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to remove the potential indeterminism of which hart wins the lottery when doing board bringup and debugging.
I am OK with that. Actually my preliminary patch about [PATCH 0/4] AE350 support SMP boot from flash did as your wish.
You can refer it : [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable https://www.mail-archive.com/u-boot@lists.denx.de/msg323419.html
Rick
To follow up on https://www.mail-archive.com/u-boot@lists.denx.de/msg323526.html
It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
From an system testing and validation point of view, I would find it much better (especially at very early boot stages, where U-boot might be the first non-ROM code running) to have a deterministic process to determine what core runs U-boot. This necessarily seems as it will require a CONFIG_BOOT_HART option, as we are not in a position to parse a device tree in head.S
+Anup
Hi Troy,
On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes troy.benjegerdes@sifive.com wrote:
On Jun 2, 2019, at 9:22 PM, Rick Chen rickchen36@gmail.com wrote:
Hi Troy
From: Troy Benjegerdes [mailto:troy.benjegerdes@sifive.com] Sent: Saturday, June 01, 2019 12:24 AM To: Auer, Lukas Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn@gmail.com; palmer@sifive.com Subject: Re: Hart lottery and CONFIG_XIP
On May 31, 2019, at 9:13 AM, Auer, Lukas lukas.auer@aisec.fraunhofer.de
wrote:
Hi Troy,
On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote:
Wouldn't the following line in head.S fail when running from flash (although maybe not in a way that prevents booting)
/* save the boot hart id to global_data */ SREG tp, GD_BOOT_HART(gp)Shouldn’t this be protected by CONFIG_XIP?
The boot hart ID is stored in global data, which is allocated from the stack (in board_init_f_alloc_reserve() ). It is therefore writable and won't cause any issues when running from flash.
Sorry about the confusion, I was reading it wrong earlier.
I’m hoping to have a couple of patches ready later today to change the CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to remove the potential indeterminism of which hart wins the lottery when doing board bringup and debugging.
I am OK with that. Actually my preliminary patch about [PATCH 0/4] AE350 support SMP boot from flash did as your wish.
You can refer it : [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable https://www.mail-archive.com/u-boot@lists.denx.de/msg323419.html
Rick
To follow up on https://www.mail-archive.com/u-boot@lists.denx.de/msg323526.html
It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
I am not sure what caused the confusion. CONFIG_XIP was added to support U-Boot executing from ROM directly.
From an system testing and validation point of view, I would find it much better (especially at very early boot stages, where U-boot might be the first non-ROM code running) to have a deterministic process to determine what core runs U-boot. This
I remember when SMP patches were submitted by Lukas for the first time it was deterministic using some macros like CONFIG_BOOT_HART, however per Anup's request, the hart lottery feature, similar to what Linux has, was added.
necessarily seems as it will require a CONFIG_BOOT_HART option, as we are not in a position to parse a device tree in head.S
Yes, this is understood. So the CONFIG_XIP was added to support such case, for which the lottery feature relies on a memory location which isn't writable yet.
Regards, Bin
On Jun 3, 2019, at 9:19 AM, Bin Meng bmeng.cn@gmail.com wrote:
+Anup
Hi Troy,
On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes troy.benjegerdes@sifive.com wrote:
On Jun 2, 2019, at 9:22 PM, Rick Chen rickchen36@gmail.com wrote:
Hi Troy
From: Troy Benjegerdes [mailto:troy.benjegerdes@sifive.com] Sent: Saturday, June 01, 2019 12:24 AM To: Auer, Lukas Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn@gmail.com; palmer@sifive.com Subject: Re: Hart lottery and CONFIG_XIP
On May 31, 2019, at 9:13 AM, Auer, Lukas lukas.auer@aisec.fraunhofer.de
wrote:
Hi Troy,
On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote: > Wouldn't the following line in head.S fail when running from flash > (although maybe not in a way that prevents booting) > > /* save the boot hart id to global_data */ > SREG tp, GD_BOOT_HART(gp) > > Shouldn’t this be protected by CONFIG_XIP?
The boot hart ID is stored in global data, which is allocated from the stack (in board_init_f_alloc_reserve() ). It is therefore writable and won't cause any issues when running from flash.
Sorry about the confusion, I was reading it wrong earlier.
I’m hoping to have a couple of patches ready later today to change the CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to remove the potential indeterminism of which hart wins the lottery when doing board bringup and debugging.
I am OK with that. Actually my preliminary patch about [PATCH 0/4] AE350 support SMP boot from flash did as your wish.
You can refer it : [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable https://www.mail-archive.com/u-boot@lists.denx.de/msg323419.html
Rick
To follow up on https://www.mail-archive.com/u-boot@lists.denx.de/msg323526.html
It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
I am not sure what caused the confusion. CONFIG_XIP was added to support U-Boot executing from ROM directly.
The confusion is use cases where you don’t necessarily need CONFIG_XIP, but you do want deterministic SMP booting, and the code is a lot more understandable with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP
From an system testing and validation point of view, I would find it much better (especially at very early boot stages, where U-boot might be the first non-ROM code running) to have a deterministic process to determine what core runs U-boot. This
I remember when SMP patches were submitted by Lukas for the first time it was deterministic using some macros like CONFIG_BOOT_HART, however per Anup's request, the hart lottery feature, similar to what Linux has, was added.
I’d like to have CONFIG_BOOT_HART available as a config option as well, particularly for system validation testing. Also along those lines, if we are going to use a lottery to determine what CPU boots the system, how do we know afterwards which one it was?
necessarily seems as it will require a CONFIG_BOOT_HART option, as we are not in a position to parse a device tree in head.S
Yes, this is understood. So the CONFIG_XIP was added to support such case, for which the lottery feature relies on a memory location which isn't writable yet.
Regards, Bin
Hi Troy,
On Mon, Jun 3, 2019 at 10:53 PM Troy Benjegerdes troy.benjegerdes@sifive.com wrote:
On Jun 3, 2019, at 9:19 AM, Bin Meng bmeng.cn@gmail.com wrote:
+Anup
Hi Troy,
On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes troy.benjegerdes@sifive.com wrote:
On Jun 2, 2019, at 9:22 PM, Rick Chen rickchen36@gmail.com wrote:
Hi Troy
From: Troy Benjegerdes [mailto:troy.benjegerdes@sifive.com] Sent: Saturday, June 01, 2019 12:24 AM To: Auer, Lukas Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn@gmail.com; palmer@sifive.com Subject: Re: Hart lottery and CONFIG_XIP
> On May 31, 2019, at 9:13 AM, Auer, Lukas lukas.auer@aisec.fraunhofer.de wrote: > > Hi Troy, > > On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote: >> Wouldn't the following line in head.S fail when running from flash >> (although maybe not in a way that prevents booting) >> >> /* save the boot hart id to global_data */ >> SREG tp, GD_BOOT_HART(gp) >> >> Shouldn’t this be protected by CONFIG_XIP? > > The boot hart ID is stored in global data, which is allocated from the > stack (in board_init_f_alloc_reserve() ). It is therefore writable and > won't cause any issues when running from flash.
Sorry about the confusion, I was reading it wrong earlier.
I’m hoping to have a couple of patches ready later today to change the CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to remove the potential indeterminism of which hart wins the lottery when doing board bringup and debugging.
I am OK with that. Actually my preliminary patch about [PATCH 0/4] AE350 support SMP boot from flash did as your wish.
You can refer it : [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable https://www.mail-archive.com/u-boot@lists.denx.de/msg323419.html
Rick
To follow up on https://www.mail-archive.com/u-boot@lists.denx.de/msg323526.html
It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
I am not sure what caused the confusion. CONFIG_XIP was added to support U-Boot executing from ROM directly.
The confusion is use cases where you don’t necessarily need CONFIG_XIP, but you do want deterministic SMP booting, and the code is a lot more understandable with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP
OK, as I pointed out, the deterministic SMP booting was originally posted, but later was changed to the lottery mechanism per Anup's review comments. I am personally fine with both.
From an system testing and validation point of view, I would find it much better (especially at very early boot stages, where U-boot might be the first non-ROM code running) to have a deterministic process to determine what core runs U-boot. This
I remember when SMP patches were submitted by Lukas for the first time it was deterministic using some macros like CONFIG_BOOT_HART, however per Anup's request, the hart lottery feature, similar to what Linux has, was added.
I’d like to have CONFIG_BOOT_HART available as a config option as well, particularly for system validation testing. Also along those lines, if we are going to use a lottery to determine what CPU boots the system, how do we know afterwards which one it was?
Yes, there is a way to know. Type 'dm tree' from U-Boot shell, then look at which cpu node has a child node of "riscv_timer".
Regards, Bin
Hi Troy
Bin Meng bmeng.cn@gmail.com 於 2019年6月3日 週一 下午10:56寫道：
Hi Troy,
On Mon, Jun 3, 2019 at 10:53 PM Troy Benjegerdes troy.benjegerdes@sifive.com wrote:
On Jun 3, 2019, at 9:19 AM, Bin Meng bmeng.cn@gmail.com wrote:
+Anup
Hi Troy,
On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes troy.benjegerdes@sifive.com wrote:
On Jun 2, 2019, at 9:22 PM, Rick Chen rickchen36@gmail.com wrote:
Hi Troy
> From: Troy Benjegerdes [mailto:troy.benjegerdes@sifive.com] > Sent: Saturday, June 01, 2019 12:24 AM > To: Auer, Lukas > Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn@gmail.com; palmer@sifive.com > Subject: Re: Hart lottery and CONFIG_XIP > > > >> On May 31, 2019, at 9:13 AM, Auer, Lukas lukas.auer@aisec.fraunhofer.de > wrote: >> >> Hi Troy, >> >> On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote: >>> Wouldn't the following line in head.S fail when running from flash >>> (although maybe not in a way that prevents booting) >>> >>> /* save the boot hart id to global_data */ >>> SREG tp, GD_BOOT_HART(gp) >>> >>> Shouldn’t this be protected by CONFIG_XIP? >> >> The boot hart ID is stored in global data, which is allocated from the >> stack (in board_init_f_alloc_reserve() ). It is therefore writable and >> won't cause any issues when running from flash. > > Sorry about the confusion, I was reading it wrong earlier. > > I’m hoping to have a couple of patches ready later today to change the > CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to > remove the potential indeterminism of which hart wins the lottery when doing > board bringup and debugging.
I am OK with that. Actually my preliminary patch about [PATCH 0/4] AE350 support SMP boot from flash did as your wish.
You can refer it : [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable https://www.mail-archive.com/u-boot@lists.denx.de/msg323419.html
Rick
To follow up on https://www.mail-archive.com/u-boot@lists.denx.de/msg323526.html
It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
I am not sure what caused the confusion. CONFIG_XIP was added to support U-Boot executing from ROM directly.
The confusion is use cases where you don’t necessarily need CONFIG_XIP, but you do want deterministic SMP booting, and the code is a lot more understandable with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP
OK, as I pointed out, the deterministic SMP booting was originally posted, but later was changed to the lottery mechanism per Anup's review comments. I am personally fine with both.
Yes. I am agree with that. But it shall also support booting form ROM and RAM as now.
From an system testing and validation point of view, I would find it much better (especially at very early boot stages, where U-boot might be the first non-ROM code running) to have a deterministic process to determine what core runs U-boot. This
I remember when SMP patches were submitted by Lukas for the first time it was deterministic using some macros like CONFIG_BOOT_HART, however per Anup's request, the hart lottery feature, similar to what Linux has, was added.
I’d like to have CONFIG_BOOT_HART available as a config option as well, particularly for system validation testing. Also along those lines, if we are going to use a lottery to determine what CPU boots the system, how do we know afterwards which one it was?
Yes, there is a way to know. Type 'dm tree' from U-Boot shell, then look at which cpu node has a child node of "riscv_timer".
Or you can check gd->arch.boot_hart by gdb
(gdb) p/x gd->arch.boot_hart $3 = 0x0
B.R Rick
Regards, Bin
On Tue, 2019-06-04 at 09:42 +0800, Rick Chen wrote:
Hi Troy
Bin Meng bmeng.cn@gmail.com 於 2019年6月3日 週一 下午10:56寫道：
Hi Troy,
On Mon, Jun 3, 2019 at 10:53 PM Troy Benjegerdes troy.benjegerdes@sifive.com wrote:
On Jun 3, 2019, at 9:19 AM, Bin Meng bmeng.cn@gmail.com wrote:
+Anup
Hi Troy,
On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes troy.benjegerdes@sifive.com wrote:
On Jun 2, 2019, at 9:22 PM, Rick Chen rickchen36@gmail.com wrote:
Hi Troy
> > From: Troy Benjegerdes [mailto:troy.benjegerdes@sifive.com] > > Sent: Saturday, June 01, 2019 12:24 AM > > To: Auer, Lukas > > Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn@gmail.com; palmer@sifive.com > > Subject: Re: Hart lottery and CONFIG_XIP > > > > > > > > > On May 31, 2019, at 9:13 AM, Auer, Lukas lukas.auer@aisec.fraunhofer.de > > wrote: > > > Hi Troy, > > > > > > On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote: > > > > Wouldn't the following line in head.S fail when running from flash > > > > (although maybe not in a way that prevents booting) > > > > > > > > /* save the boot hart id to global_data */ > > > > SREG tp, GD_BOOT_HART(gp) > > > > > > > > Shouldn’t this be protected by CONFIG_XIP? > > > > > > The boot hart ID is stored in global data, which is allocated from the > > > stack (in board_init_f_alloc_reserve() ). It is therefore writable and > > > won't cause any issues when running from flash. > > > > Sorry about the confusion, I was reading it wrong earlier. > > > > I’m hoping to have a couple of patches ready later today to change the > > CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to > > remove the potential indeterminism of which hart wins the lottery when doing > > board bringup and debugging.
I am OK with that. Actually my preliminary patch about [PATCH 0/4] AE350 support SMP boot from flash did as your wish.
You can refer it : [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable https://www.mail-archive.com/u-boot@lists.denx.de/msg323419.html
Rick
To follow up on https://www.mail-archive.com/u-boot@lists.denx.de/msg323526.html
It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
I am not sure what caused the confusion. CONFIG_XIP was added to support U-Boot executing from ROM directly.
The confusion is use cases where you don’t necessarily need CONFIG_XIP, but you do want deterministic SMP booting, and the code is a lot more understandable with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP
OK, as I pointed out, the deterministic SMP booting was originally posted, but later was changed to the lottery mechanism per Anup's review comments. I am personally fine with both.
Yes. I am agree with that. But it shall also support booting form ROM and RAM as now.
Just to give a bit of context information. Anup's reasoning was that with a fixed boot hart, we have to rely on it to work. If there is a hardware failure in just the boot hart, the system won't be able to boot anymore. With a hart lottery that is not a problem.
With that in mind, it makes sense to have to option of a fixed boot hart, but I would not enable that option in any of the upstream defconfigs. That way it can be manually enabled if required for the application.
Thanks, Lukas
From an system testing and validation point of view, I would find it much better (especially at very early boot stages, where U-boot might be the first non-ROM code running) to have a deterministic process to determine what core runs U-boot. This
I remember when SMP patches were submitted by Lukas for the first time it was deterministic using some macros like CONFIG_BOOT_HART, however per Anup's request, the hart lottery feature, similar to what Linux has, was added.
I’d like to have CONFIG_BOOT_HART available as a config option as well, particularly for system validation testing. Also along those lines, if we are going to use a lottery to determine what CPU boots the system, how do we know afterwards which one it was?
Yes, there is a way to know. Type 'dm tree' from U-Boot shell, then look at which cpu node has a child node of "riscv_timer".
Or you can check gd->arch.boot_hart by gdb
(gdb) p/x gd->arch.boot_hart $3 = 0x0
B.R Rick
Regards, Bin
On Tue, Jun 4, 2019 at 3:00 PM Auer, Lukas lukas.auer@aisec.fraunhofer.de wrote:
On Tue, 2019-06-04 at 09:42 +0800, Rick Chen wrote:
Hi Troy
Bin Meng bmeng.cn@gmail.com 於 2019年6月3日 週一 下午10:56寫道：
Hi Troy,
On Mon, Jun 3, 2019 at 10:53 PM Troy Benjegerdes troy.benjegerdes@sifive.com wrote:
On Jun 3, 2019, at 9:19 AM, Bin Meng bmeng.cn@gmail.com wrote:
+Anup
Hi Troy,
On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes troy.benjegerdes@sifive.com wrote:
> On Jun 2, 2019, at 9:22 PM, Rick Chen rickchen36@gmail.com wrote: > > Hi Troy > > > > From: Troy Benjegerdes [mailto:troy.benjegerdes@sifive.com] > > > Sent: Saturday, June 01, 2019 12:24 AM > > > To: Auer, Lukas > > > Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn@gmail.com; palmer@sifive.com > > > Subject: Re: Hart lottery and CONFIG_XIP > > > > > > > > > > > > > On May 31, 2019, at 9:13 AM, Auer, Lukas lukas.auer@aisec.fraunhofer.de > > > wrote: > > > > Hi Troy, > > > > > > > > On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote: > > > > > Wouldn't the following line in head.S fail when running from flash > > > > > (although maybe not in a way that prevents booting) > > > > > > > > > > /* save the boot hart id to global_data */ > > > > > SREG tp, GD_BOOT_HART(gp) > > > > > > > > > > Shouldn’t this be protected by CONFIG_XIP? > > > > > > > > The boot hart ID is stored in global data, which is allocated from the > > > > stack (in board_init_f_alloc_reserve() ). It is therefore writable and > > > > won't cause any issues when running from flash. > > > > > > Sorry about the confusion, I was reading it wrong earlier. > > > > > > I’m hoping to have a couple of patches ready later today to change the > > > CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to > > > remove the potential indeterminism of which hart wins the lottery when doing > > > board bringup and debugging. > > I am OK with that. > Actually my preliminary patch about > [PATCH 0/4] AE350 support SMP boot from flash > did as your wish. > > You can refer it : > [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable > https://www.mail-archive.com/u-boot@lists.denx.de/msg323419.html > > Rick
To follow up on https://www.mail-archive.com/u-boot@lists.denx.de/msg323526.html
It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
I am not sure what caused the confusion. CONFIG_XIP was added to support U-Boot executing from ROM directly.
The confusion is use cases where you don’t necessarily need CONFIG_XIP, but you do want deterministic SMP booting, and the code is a lot more understandable with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP
OK, as I pointed out, the deterministic SMP booting was originally posted, but later was changed to the lottery mechanism per Anup's review comments. I am personally fine with both.
Yes. I am agree with that. But it shall also support booting form ROM and RAM as now.
Just to give a bit of context information. Anup's reasoning was that with a fixed boot hart, we have to rely on it to work. If there is a hardware failure in just the boot hart, the system won't be able to boot anymore. With a hart lottery that is not a problem.
With that in mind, it makes sense to have to option of a fixed boot hart, but I would not enable that option in any of the upstream defconfigs. That way it can be manually enabled if required for the application.
Sorry for pitching in late.
Apart from hardware failure on fixed boot hart, another reason in support of lottery based system is that it will work nicely with upcoming SBI HART power managment where more HARTs/CPUs are kept in power-off state by OpenSBI (RUNTIME firmware) and Linux will power-on HARTs/CPUs as required at boot-time/run-time.
Regards, Anup
participants (5)
-
Anup Patel -
Auer, Lukas -
Bin Meng -
Rick Chen -
Troy Benjegerdes