[U-Boot] UBoot running UBoot - is it possible?
Hello, bods.
We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.
However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.
The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.
1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot: mmcboot=run check_uboot; <rest of original mmcboot> other_uboot=0
2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
3/ We defined check_uboot as: if test ${other_uboot} -eq 1; then setenv other_uboot 0 saveenv ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg go 0x17800000 fi The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.
I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.
So I guess my questions are as follows:
a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field? b/ Does anyone have any ideas on how I could achieve this?
I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.
Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.
Cheers, Pax
Allan Chandler | Software Engineer DTI Group Ltd | Integrated Transit Technologies 31 Affleck Road, Perth Airport, WA 6105, Australia P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler@dti.com.aumailto:allan.chandler@dti.com.au Visit our website www.dti.com.auhttp://www.dti.com.au The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments.
On Tue, Nov 13, 2018 at 1:16 PM Allan Chandler allan.chandler@dti.com.au wrote:
Hello, bods.
We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.
However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.
The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.
1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot: mmcboot=run check_uboot; <rest of original mmcboot> other_uboot=0
2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
3/ We defined check_uboot as: if test ${other_uboot} -eq 1; then setenv other_uboot 0 saveenv ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg go 0x17800000 fi The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.
I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.
So I guess my questions are as follows:
a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field? b/ Does anyone have any ideas on how I could achieve this?
My idea was to let SPL implement a dedicated boot counter/watchdog that detects problems starting U-Boot and using a backup copy if it fails multiple times. Of course you need an SPL on your board to do this. Plus you end up with the same problem for updating SPL, but I'm lucky here that my platform (socfpga gen5) has redundant storage for SPL and implements a startup watchdog mechanism for the SPL.
Simon
I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.
Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.
Cheers, Pax
Allan Chandler | Software Engineer DTI Group Ltd | Integrated Transit Technologies 31 Affleck Road, Perth Airport, WA 6105, Australia P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler@dti.com.aumailto:allan.chandler@dti.com.au Visit our website www.dti.com.auhttp://www.dti.com.au The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments. _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de https://lists.denx.de/listinfo/u-boot
Dear Simon,
In message CAAh8qsxB3YNoDruZnmcPvkygc7HXVOcf=PNdA4xRp=PjmQANsw@mail.gmail.com you wrote:
My idea was to let SPL implement a dedicated boot counter/watchdog that detects problems starting U-Boot and using a backup copy if it fails multiple times. Of course you need an SPL on your board to do this. Plus you end up with the same problem for updating SPL, but I'm lucky here that my platform (socfpga gen5) has redundant storage for SPL and implements a startup watchdog mechanism for the SPL.
In other words, you are just adding enough additional complexity to SPL to make it reasonably likely that there will be bugs that need to be fixed later, i. e. you have to update the SPL.
And then?
You are just moving the problem, not solving it.
Best regards,
Wolfgang Denk
On 13.11.2018 17:00, Wolfgang Denk wrote:
Dear Simon,
In message CAAh8qsxB3YNoDruZnmcPvkygc7HXVOcf=PNdA4xRp=PjmQANsw@mail.gmail.com you wrote:
My idea was to let SPL implement a dedicated boot counter/watchdog that detects problems starting U-Boot and using a backup copy if it fails multiple times. Of course you need an SPL on your board to do this. Plus you end up with the same problem for updating SPL, but I'm lucky here that my platform (socfpga gen5) has redundant storage for SPL and implements a startup watchdog mechanism for the SPL.
In other words, you are just adding enough additional complexity to SPL to make it reasonably likely that there will be bugs that need to be fixed later, i. e. you have to update the SPL.
And then?
You are just moving the problem, not solving it.
That's not how I see it. As I see it, I have to implement an upgrade option for SPL. This is partly due to bad design of the socfpga_gen5 platform. But also I have read multiple times on this list that you should use SPL from the same version as U-Boot as they might work combined and U-Boot might depend on SPL to do things that might change over time. So is it really a good idea to upgrade U-Boot without upgrading SPL at the same time? It seems to me this would require thorough testing of different version mixes...
So given that SPL must be upgradable, how is it more complex to detect U-Boot failure from SPL than from U-Boot itself?
Regards, Simon
On 13/11/18 17:07, Simon Goldschmidt wrote:
On 13.11.2018 17:00, Wolfgang Denk wrote:
Dear Simon,
In message CAAh8qsxB3YNoDruZnmcPvkygc7HXVOcf=PNdA4xRp=PjmQANsw@mail.gmail.com you wrote:
My idea was to let SPL implement a dedicated boot counter/watchdog that detects problems starting U-Boot and using a backup copy if it fails multiple times. Of course you need an SPL on your board to do this. Plus you end up with the same problem for updating SPL, but I'm lucky here that my platform (socfpga gen5) has redundant storage for SPL and implements a startup watchdog mechanism for the SPL.
In other words, you are just adding enough additional complexity to SPL to make it reasonably likely that there will be bugs that need to be fixed later, i. e. you have to update the SPL.
And then?
You are just moving the problem, not solving it.
That's not how I see it. As I see it, I have to implement an upgrade option for SPL. This is partly due to bad design of the socfpga_gen5 platform. But also I have read multiple times on this list that you should use SPL from the same version as U-Boot as they might work combined and U-Boot might depend on SPL to do things that might change over time.
Right. It is not guaranteed that mixing versions works.
So is it really a good idea to upgrade U-Boot without upgrading SPL at the same time?
IMHO it is a bad idea, and it forgets that the bootloader is really SPL + u-boot.img else just u-boot.img. It is also questionable if it makes sense to provide an update mechanism for u-boot when most of critical parts like clocks, DDR initialisation, etc. are in SPL.
It seems to me this would require thorough testing of different version mixes...
Agree, and this becomes a mess.
So given that SPL must be upgradable, how is it more complex to detect U-Boot failure from SPL than from U-Boot itself?
I do not get the question - if SPL fails, it can be at any time before you get the control. There sould be a mechanism to switch to a previous copy of SPL, generally this is not available in hw.
Best regards, Stefano
Am Di., 13. Nov. 2018, 17:53 hat Stefano Babic sbabic@denx.de geschrieben:
On 13/11/18 17:07, Simon Goldschmidt wrote:
On 13.11.2018 17:00, Wolfgang Denk wrote:
Dear Simon,
In message CAAh8qsxB3YNoDruZnmcPvkygc7HXVOcf=PNdA4xRp=PjmQANsw@mail.gmail.com you wrote:
My idea was to let SPL implement a dedicated boot counter/watchdog that detects problems starting U-Boot and using a backup copy if it fails multiple times. Of course you need an SPL on your board to do this. Plus you end up with the same problem for updating SPL, but I'm lucky here that my platform (socfpga gen5) has redundant storage for SPL and implements a startup watchdog mechanism for the SPL.
In other words, you are just adding enough additional complexity to SPL to make it reasonably likely that there will be bugs that need to be fixed later, i. e. you have to update the SPL.
And then?
You are just moving the problem, not solving it.
That's not how I see it. As I see it, I have to implement an upgrade option for SPL. This is partly due to bad design of the socfpga_gen5 platform. But also I have read multiple times on this list that you should use SPL from the same version as U-Boot as they might work combined and U-Boot might depend on SPL to do things that might change over time.
Right. It is not guaranteed that mixing versions works.
So is it really a good idea to upgrade U-Boot without upgrading SPL at the same time?
IMHO it is a bad idea, and it forgets that the bootloader is really SPL
- u-boot.img else just u-boot.img. It is also questionable if it makes
sense to provide an update mechanism for u-boot when most of critical parts like clocks, DDR initialisation, etc. are in SPL.
It seems to me this would require thorough testing of different version mixes...
Agree, and this becomes a mess.
So given that SPL must be upgradable, how is it more complex to detect U-Boot failure from SPL than from U-Boot itself?
I do not get the question - if SPL fails, it can be at any time before you get the control. There sould be a mechanism to switch to a previous copy of SPL, generally this is not available in hw.
Unless I'm mistaken, the socfpga gen5 bootrom implements a timer and loads SPL from secondary storage of it does not register as successful after some time.
That's my arch only, of course...
Simon
Hi Simon,
On 13/11/18 18:43, Simon Goldschmidt wrote:
Am Di., 13. Nov. 2018, 17:53 hat Stefano Babic <sbabic@denx.de mailto:sbabic@denx.de> geschrieben:
On 13/11/18 17:07, Simon Goldschmidt wrote: > On 13.11.2018 17:00, Wolfgang Denk wrote: >> Dear Simon, >> >> In message >> <CAAh8qsxB3YNoDruZnmcPvkygc7HXVOcf=PNdA4xRp=PjmQANsw@mail.gmail.com <mailto:PjmQANsw@mail.gmail.com>> >> you wrote: >>> My idea was to let SPL implement a dedicated boot counter/watchdog >>> that detects problems starting U-Boot and using a backup copy if it >>> fails multiple times. Of course you need an SPL on your board to do >>> this. Plus you end up with the same problem for updating SPL, but I'm >>> lucky here that my platform (socfpga gen5) has redundant storage for >>> SPL and implements a startup watchdog mechanism for the SPL. >> In other words, you are just adding enough additional complexity to >> SPL to make it reasonably likely that there will be bugs that need >> to be fixed later, i. e. you have to update the SPL. >> >> And then? >> >> You are just moving the problem, not solving it. > > That's not how I see it. As I see it, I have to implement an upgrade > option for SPL. This is partly due to bad design of the socfpga_gen5 > platform. But also I have read multiple times on this list that you > should use SPL from the same version as U-Boot as they might work > combined and U-Boot might depend on SPL to do things that might change > over time. Right. It is not guaranteed that mixing versions works. > So is it really a good idea to upgrade U-Boot without > upgrading SPL at the same time? IMHO it is a bad idea, and it forgets that the bootloader is really SPL + u-boot.img else just u-boot.img. It is also questionable if it makes sense to provide an update mechanism for u-boot when most of critical parts like clocks, DDR initialisation, etc. are in SPL. > It seems to me this would require > thorough testing of different version mixes... > Agree, and this becomes a mess. > So given that SPL must be upgradable, how is it more complex to detect > U-Boot failure from SPL than from U-Boot itself? I do not get the question - if SPL fails, it can be at any time before you get the control. There sould be a mechanism to switch to a previous copy of SPL, generally this is not available in hw.Unless I'm mistaken, the socfpga gen5 bootrom implements a timer and loads SPL from secondary storage of it does not register as successful after some time.
That means you *have* hardware support, then.
I am not aware there is such as feature on i.MX6.
That's my arch only, of course...
Regards, Stefano
Hi Allan,
On 13/11/18 02:38, Allan Chandler wrote:
Hello, bods.
We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.
ok - think about that a *safe* update of the bootloader with all components (SPL, u-boot.img, environment) must be considered and designed at the early beginning because it could require hardware support.
The i.MX6 can boot from several devices in a sequence, and this can be used to update the bootloader. For example, it can boot from a SPI NOR flash if booting from eMMC fails. Of course, you have added a SPI NOR (or another second source) in your design.
Even in this case, please consider that the BootROM searches for another source if the first one is definely wroing, that means that the i.MX header is corrupted. If the image (SPL) is correct, the BootROM cannot detect it.
This means that with multiple devices where to boot you cannot be sure, because a valid SPL was written but board does not boot for some reasons.
You could use the EXT CSD register in the eMMC (it looks like you have just eMMC on board) to switch the boot partition. This is transparent to the i.MX and eMMC since eMMC 4.3 supports two partitions. This can allow to write the second bootloader and to be power-off safe, but there is no fallback. Anyway, the rick can be mitigated with testing, but cannot be dropped at all. If the second bootloader is valid, the BootROM will always try to boot from it and there is no fallback if it goes wrong.
To be absolutely sure, a (hw) mechanism that switch between two SPLs, maybe driven if watchdog expires, could be implemented. Of course, complexity increases.
The thing is that each project must balance risks again costs. In some cases, it is also accepted that bootloader is updated with a percentage of risks that the board could be returned because broken. This can be cheaper as implementing a 100% safe update for the bootloader, but of course it cannot be used in critical applications.
You could also plan to update U-Boot, that is u-boot.img on i.MX, without updating SPL. This is also a used path, with SPL fixed on the board and checking for two copies of u-boot.img.
However, this is broken. SPL is responsible for the first initialization and first of all of the DRAM controller on i.MX. If something was broken, it cannot be fixed just by updating the u-boot.img image. I do not know if it is worth to have it, it looks like a half-upate where just a very limited number of issues can be fixed.
However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.
Check if version differs is a *must*. I do this in SWUpdate and the bootloader is always part of each delivery, but it is updated only if version differs.
The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.
Think again: you are updating the bootloader, but you rely on u-boot scripts that are part of u-boot itself. Scripts can be corrupted, too, or changed by a previous u-boot version, or...
It sounds like a hack, but of course, it could work with a percentage of risks that things can go wrong. I mean, a percentage without taking into account Murphy's law :-D
1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot: mmcboot=run check_uboot; <rest of original mmcboot> other_uboot=0
2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
3/ We defined check_uboot as: if test ${other_uboot} -eq 1; then setenv other_uboot 0 saveenv ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg go 0x17800000 fi
This means that the *old* U-Boot works pretty well, because it can go until this point and load something else...
There are a lot of assumption here, because you are not really updating the bootloader, but simply adding a new actor in the chain. And then, both can be buggy.....
The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.
The second bootloader must be fine tuned and changed from original because it is not allowed to set again the DRAM controller, and maybe even some closks, and maybe...
I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.
So I guess my questions are as follows:
a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field?
Define safely. 100% error prone, 97%, ..
100% safe could mean additional hardware and additional complexity, and at the end, additional risk that something can go wrong.
That means: some projects decide to add hw support or more to be absolutely sure that nothing can happen, some other projects take the risk and accept the possibility to have a low RTB. You decide, at the end...
b/ Does anyone have any ideas on how I could achieve this?
I think that each solution is quite project specific. Other SOCs allow to switch automatically SPL (that means, the source for the bootloader). On i.MX6, it can be reached using a sequence of devices (see BOOT_CFG in manual), but there is no fallback.
I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.
Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.
Best regards, Stefano Babic
Cheers, Pax
Allan Chandler | Software Engineer DTI Group Ltd | Integrated Transit Technologies 31 Affleck Road, Perth Airport, WA 6105, Australia P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler@dti.com.aumailto:allan.chandler@dti.com.au Visit our website www.dti.com.auhttp://www.dti.com.au The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments. _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de https://lists.denx.de/listinfo/u-boot
Hi Allan,
On 12 November 2018 at 18:38, Allan Chandler allan.chandler@dti.com.au wrote:
Hello, bods.
We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.
However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.
The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.
1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot: mmcboot=run check_uboot; <rest of original mmcboot> other_uboot=0
2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
3/ We defined check_uboot as: if test ${other_uboot} -eq 1; then setenv other_uboot 0 saveenv ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg go 0x17800000 fi The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.
I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.
You may need to flush the cache area that you read into, or use 'dcache off' before jumping to the second U-Boot.
So I guess my questions are as follows:
a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field? b/ Does anyone have any ideas on how I could achieve this?
I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.
Chromium OS and Android verified boot (which is in mainline - see avb) use an A/B system. You can have SPL chosen whether to boot from A or B.
I'm actually working on integrating Chromium OS vboot into U-Boot again. It uses TPL to select the SPL to use, which then boots into UBoot proper. This is so we can update the SDRAM code. Am hoping to have something going in January although a lot of the required patches have been sent.
Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.
You really need to sign it so you can send updated U-Boot versions and the device in the field can verified the U-Boot image in the field.
Regards, Simon
Cheers, Pax
Allan Chandler | Software Engineer DTI Group Ltd | Integrated Transit Technologies 31 Affleck Road, Perth Airport, WA 6105, Australia P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler@dti.com.aumailto:allan.chandler@dti.com.au Visit our website www.dti.com.auhttp://www.dti.com.au The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments. _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de https://lists.denx.de/listinfo/u-boot
On 13.11.2018 20:53, Simon Glass wrote:
Hi Allan,
On 12 November 2018 at 18:38, Allan Chandler allan.chandler@dti.com.au wrote:
Hello, bods.
We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.
However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.
The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.
1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot: mmcboot=run check_uboot; <rest of original mmcboot> other_uboot=0
2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
3/ We defined check_uboot as: if test ${other_uboot} -eq 1; then setenv other_uboot 0 saveenv ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg go 0x17800000 fi The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.
I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.
You may need to flush the cache area that you read into, or use 'dcache off' before jumping to the second U-Boot.
So I guess my questions are as follows:
a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field? b/ Does anyone have any ideas on how I could achieve this?
I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.
Chromium OS and Android verified boot (which is in mainline - see avb) use an A/B system. You can have SPL chosen whether to boot from A or B.
I'm actually working on integrating Chromium OS vboot into U-Boot again. It uses TPL to select the SPL to use, which then boots into UBoot proper. This is so we can update the SDRAM code. Am hoping to have something going in January although a lot of the required patches have been sent.
That sounds interesting, but would it be safe to use TPL from 2019 and letting it boot a, say, 2025 SPL + U-Boot? I don't have experience with TPL, can it check signatures for secure boot? I'd suspect Chromium OS would need that?
Simon
Also, we actually do quite a bit of checking to ensure the image we install is correctly written to the UBoot partition - it has an MD5 distributed with the package and a mismatch will prevent activation. It also checks certain other things like version info and a tag at the end of the partition to ensure the write was complete. So it may be we're just being too paranoid here. If so, let me know, I'm sure I could convince the customer with some cogent arguments.
You really need to sign it so you can send updated U-Boot versions and the device in the field can verified the U-Boot image in the field.
Regards, Simon
Cheers, Pax
Allan Chandler | Software Engineer DTI Group Ltd | Integrated Transit Technologies 31 Affleck Road, Perth Airport, WA 6105, Australia P +61 8 9373 2905, 182 | F +61 8 9479 1190 | allan.chandler@dti.com.aumailto:allan.chandler@dti.com.au Visit our website www.dti.com.auhttp://www.dti.com.au The information contained in this email is confidential. If you receive this email in error, please inform DTI Group Ltd via the above contact details. If you are not the intended recipient, you may not use or disclose the information contained in this email or attachments. _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de https://lists.denx.de/listinfo/u-boot
U-Boot mailing list U-Boot@lists.denx.de https://lists.denx.de/listinfo/u-boot
Hi Simon,
On 13 November 2018 at 12:17, Simon Goldschmidt simon.k.r.goldschmidt@gmail.com wrote:
On 13.11.2018 20:53, Simon Glass wrote:
Hi Allan,
On 12 November 2018 at 18:38, Allan Chandler allan.chandler@dti.com.au wrote:
Hello, bods.
We're trying to architect a relatively safe solution for updating UBoot in the field. What we have at the moment is an iMX6-based board with two UBoot partitions and two system partitions but we only use the first UBoot one. Switching system partitions is covered well by UBoot since we use what seems like a fairly standard method involving upgrade_available, bootlimit and bootcount variables, along with bootcmd/altbootcmd scripts to try and load a new system partition while falling back if it fails.
However, I'm wondering how people handle the need to update UBoot itself. Although this will happen FAR less often than normal software updates, these product have a multi-decade lifespan and it's hard to imagine we'll get through that without some update being needed. Especially since we need one now (three years in) due to a bug (with our added stuff, not UBoot baseline). With the iMX6 boards we have, once we tell it to start using the second UBoot partition, there's no UBoot/watchdog combo that will revert that change and reprogramming requires a rather expensive RTB (return to base) to fix via the serial interface.
What we had hoped to do was to be able to soft-boot an alternate UBoot (i.e., without first telling the iMX6 board to commit to the change). The scenario would go like this:
1/ Have an installer package (it runs under control of the system partition) that just writes the new UBoot image into the alternate UBoot partition then soft-boots it somehow (so now running same system partition but started from the new UBoot image).
2/ While running from the system partition started from that alternate UBoot partition, have an updater package that tells the iMX6 board to commit to the changeover. This updater package would only run if it detected that the bootable UBoot and currently-used UBoot were different.
The advantage of this is that, unless the new UBoot is FULLY capable of running our system partition (and also running an installer from there), no commit is possible, hence a simple power cycle would return to the previous working state.
We originally tried kexec from the system partition but it seemed to want to run a Linux kernel rather than loading and executing some arbitrary boot code.
So we then turned to the UBoot scripts themselves and thought we'd found a way we could do it.
1/ We changed the mmcboot script by prefixing a special check and introduced a variable for handling soft-boot: mmcboot=run check_uboot; <rest of original mmcboot> other_uboot=0
2/ We extracted the boot image uboot.bootimg from the IMX file by stripping off the first 0xc00 bytes (we had to put this into the /boot file system since I don't yet know how to get at raw partition data from UBoot scripts.
3/ We defined check_uboot as: if test ${other_uboot} -eq 1; then setenv other_uboot 0 saveenv ext4load mmc ${mmcdev}:${mmcpart} 0x17800000 /boot/uboot.bootimg go 0x17800000 fi The way this should work is that, if alternate boot is flagged, it immediately unflags it (for recovery if the alternate fails) then loads and executes the other UBoot image. If it's not flagged, check_uboot will return without trying to soft-boot the alternate.
4/ After installing the new UBoot to the alternate partition, we set a UBoot variable (other_uboot) to 1 and rebooted.
Now this seemed to work inasmuch as the alternate UBoot program started pumping out console messages but, unfortunately, it seemed to hang partway through the boot process.
I suspect this is because, having already been through a portion of that boot process in the primary UBoot, it's not keen on having to do it again.
You may need to flush the cache area that you read into, or use 'dcache off' before jumping to the second U-Boot.
So I guess my questions are as follows:
a/ How do people currently handle (if they do) the requirement that UBoot be safely updatable in the field? b/ Does anyone have any ideas on how I could achieve this?
I've been told that Google does something like this for Android booting but had to heavily modify UBoot to do it. I haven't yet investigated this possibility.
Chromium OS and Android verified boot (which is in mainline - see avb) use an A/B system. You can have SPL chosen whether to boot from A or B.
I'm actually working on integrating Chromium OS vboot into U-Boot again. It uses TPL to select the SPL to use, which then boots into UBoot proper. This is so we can update the SDRAM code. Am hoping to have something going in January although a lot of the required patches have been sent.
That sounds interesting, but would it be safe to use TPL from 2019 and letting it boot a, say, 2025 SPL + U-Boot? I don't have experience with TPL, can it check signatures for secure boot? I'd suspect Chromium OS would need that?
This needs to be tested with each build.
The bloblist feature is designed to handle this, although it is update to individual users to include version info at present. It should land in the next release.
Regards, Simon
participants (5)
-
Allan Chandler -
Simon Glass -
Simon Goldschmidt -
Stefano Babic -
Wolfgang Denk