[U-Boot] [PATCH 0/2] Fix CAAM for TrustZone enable for warp7
This series is the u-boot fix to a problem we encountered when enabling OPTEE/TrustZone on the WaRP7. The symptom is once TrustZone is activated the first page of CAAM registers becomes read-only, read-zero from the perspective of Linux and other non TrustZone contexts.
Offlining the problem with Peng Fan[1] we eventually came to realise the problem could be worked around by
1. Making Linux skip RNG initialisation - a set of patches should be hitting LKML to do just that.
2. Initialising the RNG either from u-boot or OPTEE. In this case u-boot is the right place to-do that because there's upstream code in u-boot that just works. Patch #2 does that for the WaRP7.
3. Ensuring the job-ring registers are assigned to the non TrustZone mode. On the i.MX7 after the BootROM runs the job-ring registers are assigned to TrustZone. Patch #1 does that for all CAAM hardware.
On point #3 this ordinarily isn't a problem because unless TrustZone is activated the restrictions on the job-ring registers don't kick in, its only after enabling TrustZone that Linux will loose access to the job-ring registers.
Finally should OPTEE or another TEE want to do things with the job-ring registers it will have sufficient privilege to assign whichever job-ring registers it wants to OPTEE/TEE but will naturally then have to arbitrate with Linux to inform the Kernel CAAM driver which job-ring registers it can and cannot access.
That arbitration process is for a future putative OPTEE/TEE CAAM driver to solve and is out of scope of this patchset.
[1] Thanks for all of your help BTW - Peng, there's no way this would be working without you giving direction on how.
Bryan O'Donoghue (2): drivers/crypto/fsl: assign job-rings to non-TrustZone warp7 : run sec_init for CAAM RNG
board/warp7/warp7.c | 6 +++++- drivers/crypto/fsl/jr.c | 9 +++++++++ drivers/crypto/fsl/jr.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)
After enabling TrustZone various parts of the CAAM silicon become inaccessible to non TrustZone contexts. The job-ring registers are designed to allow non TrustZone contexts like Linux to still submit jobs to CAAM even after TrustZone has been enabled.
The default job-ring permissions after the BootROM look like this for job-ring zero.
ms=0x00008001 ls=0x00008001
The MS field is JRaMIDR_MS (job ring MID most significant).
Referring to "Security Reference Manual for i.MX 7Dual and 7Solo Applications Processors, Rev. 0, 03/2017" section 8.10.4 we see that JROWN_NS controls whether or not a job-ring is accessible from non TrustZone.
Bit 15 (TrustZone) is the logical inverse of bit 3 hence the above value of 0x8001 shows that JROWN_NS=0 and TrustZone=1.
Clearly then as soon as TrustZone becomes active the job-ring registers are no longer accessible from Linux, which is not what we want.
This patch explicitly sets all job-ring registers to JROWN_NS=1 (non TrustZone) by default. If a piece of TrustZone firmware requires ownership of job-ring registers it can unset the JROWN_NS bit itself.
This patch in conjunction with a modification of the Linux kernel to skip HWRNG initialisation makes CAAM usable to Linux with TrustZone enabled.
Signed-off-by: Bryan O'Donoghue bryan.odonoghue@linaro.org Cc: Fabio Estevam fabio.estevam@nxp.com Cc: Peng Fan peng.fan@nxp.com Cc: Alex Porosanu alexandru.porosanu@nxp.com Cc: Ruchika Gupta ruchika.gupta@nxp.com Cc: Aneesh Bansal aneesh.bansal@nxp.com Link: https://github.com/OP-TEE/optee_os/issues/1408 Link: https://tinyurl.com/yam5gv9a --- drivers/crypto/fsl/jr.c | 9 +++++++++ drivers/crypto/fsl/jr.h | 1 + 2 files changed, 10 insertions(+)
diff --git a/drivers/crypto/fsl/jr.c b/drivers/crypto/fsl/jr.c index 986eabf..cbf570d 100644 --- a/drivers/crypto/fsl/jr.c +++ b/drivers/crypto/fsl/jr.c @@ -566,6 +566,8 @@ int sec_init_idx(uint8_t sec_idx) { ccsr_sec_t *sec = (void *)SEC_ADDR(sec_idx); uint32_t mcr = sec_in32(&sec->mcfgr); + uint32_t jrown_ns; + int i; int ret = 0;
#ifdef CONFIG_FSL_CORENET @@ -621,6 +623,13 @@ int sec_init_idx(uint8_t sec_idx) #endif #endif
+ /* Set ownership of job rings to non-TrustZone mode by default */ + for (i = 0; i < ARRAY_SIZE(sec->jrliodnr); i++) { + jrown_ns = sec_in32(&sec->jrliodnr[i].ms); + jrown_ns |= JROWN_NS; + sec_out32(&sec->jrliodnr[i].ms, jrown_ns); + } + ret = jr_init(sec_idx); if (ret < 0) { printf("SEC initialization failed\n"); diff --git a/drivers/crypto/fsl/jr.h b/drivers/crypto/fsl/jr.h index d897e57..72f7501 100644 --- a/drivers/crypto/fsl/jr.h +++ b/drivers/crypto/fsl/jr.h @@ -34,6 +34,7 @@ #define JRNSLIODN_MASK 0x0fff0000 #define JRSLIODN_SHIFT 0 #define JRSLIODN_MASK 0x00000fff +#define JROWN_NS 0x00000008
#define JQ_DEQ_ERR -1 #define JQ_DEQ_TO_ERR -2
This patch adds a sec_init call into board_init. Doing so in conjunction with the patch "drivers/crypto/fsl: assign job-rings to non-TrustZone" enables use of the CAAM in Linux when OPTEE/TrustZone is active.
u-boot will initialise the RNG and assign ownership of the job-ring registers to a non-TrustZone context. Linux then simply has to detect or be told to skip RNG initialisation.
This change is safe both for the OPTEE/TrustZone boot path and the regular non-OPTEE/TrustZone boot path.
Signed-off-by: Bryan O'Donoghue bryan.odonoghue@linaro.org Cc: Fabio Estevam fabio.estevam@nxp.com Cc: Peng Fan peng.fan@nxp.com Cc: Marco Franchi marco.franchi@nxp.com Cc: Vanessa Maegima vanessa.maegima@nxp.com Cc: Stefano Babic sbabic@denx.de --- board/warp7/warp7.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/board/warp7/warp7.c b/board/warp7/warp7.c index 337e76b..219ab6f 100644 --- a/board/warp7/warp7.c +++ b/board/warp7/warp7.c @@ -16,6 +16,7 @@ #include <asm/io.h> #include <common.h> #include <fsl_esdhc.h> +#include <fsl_sec.h> #include <i2c.h> #include <mmc.h> #include <asm/arch/crm_regs.h> @@ -225,6 +226,10 @@ int board_init(void) setup_i2c(0, CONFIG_SYS_I2C_SPEED, 0x7f, &i2c_pad_info1); #endif
+ #ifdef CONFIG_FSL_CAAM + sec_init(); + #endif + return 0; }
@@ -366,5 +371,4 @@ int g_dnl_bind_fixup(struct usb_device_descriptor *dev, const char *name)
return 0; } - #endif /* ifdef CONFIG_USB_GADGET */
On Tue, 2018-01-23 at 21:10 +0000, Bryan O'Donoghue wrote:
This series is the u-boot fix to a problem we encountered when enabling OPTEE/TrustZone on the WaRP7. The symptom is once TrustZone is activated the first page of CAAM registers becomes read-only, read-zero from the perspective of Linux and other non TrustZone contexts.
Offlining the problem with Peng Fan[1] we eventually came to realise the problem could be worked around by
Making Linux skip RNG initialisation - a set of patches should be hitting LKML to do just that.
Initialising the RNG either from u-boot or OPTEE. In this case u-
boot is the right place to-do that because there's upstream code in u-boot that just works. Patch #2 does that for the WaRP7.
- Ensuring the job-ring registers are assigned to the non TrustZone
mode. On the i.MX7 after the BootROM runs the job-ring registers are assigned to TrustZone. Patch #1 does that for all CAAM hardware.
I did have the same issue, not with booting OPTEE, but booting Linux in non-secure mode. #2 and #3 are required to handle this problem, but #1 is not needed.
The CAAM kernel driver always tries to instantiate all RNG state handles directly using DEC0 (which is only accessible from secure mode). The u-boot driver however only instantiates the first state handle, which is why the kernel driver then goes on and tries to instantiate the second one. This is solved by simply instantiating all RNG state handles in the CAAM u-boot driver.
I can prepare a patch to implement this, if there is interest. This is tested to work with the mainline kernel and I assume that it would work with the NXP kernel as well. Further patches are however needed to support the imx7 in the CAAM kernel driver.
On point #3 this ordinarily isn't a problem because unless TrustZone is activated the restrictions on the job-ring registers don't kick in, its only after enabling TrustZone that Linux will loose access to the job-ring registers.
Finally should OPTEE or another TEE want to do things with the job- ring registers it will have sufficient privilege to assign whichever job- ring registers it wants to OPTEE/TEE but will naturally then have to arbitrate with Linux to inform the Kernel CAAM driver which job-ring registers it can and cannot access.
That arbitration process is for a future putative OPTEE/TEE CAAM driver to solve and is out of scope of this patchset.
This is actually quite simple to solve, since each job ring has a separate device tree node. Simply disabling all job rings used by OPTEE / secure world software should be sufficient.
Thanks, Lukas
[1] Thanks for all of your help BTW - Peng, there's no way this would be working without you giving direction on how.
Bryan O'Donoghue (2): drivers/crypto/fsl: assign job-rings to non-TrustZone warp7 : run sec_init for CAAM RNG
board/warp7/warp7.c | 6 +++++- drivers/crypto/fsl/jr.c | 9 +++++++++ drivers/crypto/fsl/jr.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)
On 24/01/18 12:52, Auer, Lukas wrote:
On Tue, 2018-01-23 at 21:10 +0000, Bryan O'Donoghue wrote:
This series is the u-boot fix to a problem we encountered when enabling OPTEE/TrustZone on the WaRP7. The symptom is once TrustZone is activated the first page of CAAM registers becomes read-only, read-zero from the perspective of Linux and other non TrustZone contexts.
Offlining the problem with Peng Fan[1] we eventually came to realise the problem could be worked around by
Making Linux skip RNG initialisation - a set of patches should be hitting LKML to do just that.
Initialising the RNG either from u-boot or OPTEE. In this case u-
boot is the right place to-do that because there's upstream code in u-boot that just works. Patch #2 does that for the WaRP7.
- Ensuring the job-ring registers are assigned to the non TrustZone
mode. On the i.MX7 after the BootROM runs the job-ring registers are assigned to TrustZone. Patch #1 does that for all CAAM hardware.
I did have the same issue, not with booting OPTEE, but booting Linux in non-secure mode. #2 and #3 are required to handle this problem, but #1 is not needed.
The CAAM kernel driver always tries to instantiate all RNG state handles directly using DEC0 (which is only accessible from secure mode). The u-boot driver however only instantiates the first state handle, which is why the kernel driver then goes on and tries to instantiate the second one. This is solved by simply instantiating all RNG state handles in the CAAM u-boot driver.
I can prepare a patch to implement this, if there is interest. This is tested to work with the mainline kernel and I assume that it would work with the NXP kernel as well. Further patches are however needed to support the imx7 in the CAAM kernel driver.
So I just sent out a patch-set for i.MX7 covering this - I added you to the CC list for that.
The guidance from NXP (and my experience) is that the current kernel driver bugs-out when it can't touch the deco registers.
I have a bunch of fixes around that.
On point #3 this ordinarily isn't a problem because unless TrustZone is activated the restrictions on the job-ring registers don't kick in, its only after enabling TrustZone that Linux will loose access to the job-ring registers.
Finally should OPTEE or another TEE want to do things with the job- ring registers it will have sufficient privilege to assign whichever job- ring registers it wants to OPTEE/TEE but will naturally then have to arbitrate with Linux to inform the Kernel CAAM driver which job-ring registers it can and cannot access.
That arbitration process is for a future putative OPTEE/TEE CAAM driver to solve and is out of scope of this patchset.
This is actually quite simple to solve, since each job ring has a separate device tree node. Simply disabling all job rings used by OPTEE / secure world software should be sufficient.
Yes I agree. Then again there is currently no CAAM/OPTEE driver so nothing to-do.
On Wed, 2018-01-24 at 14:35 +0000, Bryan O'Donoghue wrote:
On 24/01/18 12:52, Auer, Lukas wrote:
On Tue, 2018-01-23 at 21:10 +0000, Bryan O'Donoghue wrote:
This series is the u-boot fix to a problem we encountered when enabling OPTEE/TrustZone on the WaRP7. The symptom is once TrustZone is activated the first page of CAAM registers becomes read-only, read-zero from the perspective of Linux and other non TrustZone contexts.
Offlining the problem with Peng Fan[1] we eventually came to realise the problem could be worked around by
- Making Linux skip RNG initialisation - a set of patches should
be hitting LKML to do just that.
- Initialising the RNG either from u-boot or OPTEE. In this case
u- boot is the right place to-do that because there's upstream code in u-boot that just works. Patch #2 does that for the WaRP7.
- Ensuring the job-ring registers are assigned to the non
TrustZone mode. On the i.MX7 after the BootROM runs the job-ring registers are assigned to TrustZone. Patch #1 does that for all CAAM hardware.
I did have the same issue, not with booting OPTEE, but booting Linux in non-secure mode. #2 and #3 are required to handle this problem, but #1 is not needed.
The CAAM kernel driver always tries to instantiate all RNG state handles directly using DEC0 (which is only accessible from secure mode). The u-boot driver however only instantiates the first state handle, which is why the kernel driver then goes on and tries to instantiate the second one. This is solved by simply instantiating all RNG state handles in the CAAM u-boot driver.
I can prepare a patch to implement this, if there is interest. This is tested to work with the mainline kernel and I assume that it would work with the NXP kernel as well. Further patches are however needed to support the imx7 in the CAAM kernel driver.
So I just sent out a patch-set for i.MX7 covering this - I added you to the CC list for that.
The guidance from NXP (and my experience) is that the current kernel driver bugs-out when it can't touch the deco registers.
I have a bunch of fixes around that.
Thanks for adding me to the CC list.
I have experienced the same thing regarding the dec0 registers. However, I don't understand why you want to detect secure mode in the kernel driver to skip RNG instantiation instead of instantiating all RNG state handles in the u-boot driver. This way, the kernel driver would skip both state handles (instead of just the first one as with the current u-boot driver) in the probe call. The dec0 registers would therefore never be used.
In addition, skipping RNG instantiation in the kernel driver means that the second state handle never gets instantiated.
On point #3 this ordinarily isn't a problem because unless TrustZone is activated the restrictions on the job-ring registers don't kick in, its only after enabling TrustZone that Linux will loose access to the job-ring registers.
Finally should OPTEE or another TEE want to do things with the job- ring registers it will have sufficient privilege to assign whichever job- ring registers it wants to OPTEE/TEE but will naturally then have to arbitrate with Linux to inform the Kernel CAAM driver which job-ring registers it can and cannot access.
That arbitration process is for a future putative OPTEE/TEE CAAM driver to solve and is out of scope of this patchset.
This is actually quite simple to solve, since each job ring has a separate device tree node. Simply disabling all job rings used by OPTEE / secure world software should be sufficient.
Yes I agree. Then again there is currently no CAAM/OPTEE driver so nothing to-do.
On 24/01/18 17:41, Auer, Lukas wrote:
Thanks for adding me to the CC list.
I have experienced the same thing regarding the dec0 registers. However, I don't understand why you want to detect secure mode in the kernel driver to skip RNG instantiation instead of instantiating all RNG state handles in the u-boot driver.
That's what we are doing though.
This set instantiates everything in u-boot and then detects and skips in the kernel if-and-only if
1. Trust zone is detected 2. It looks to the Linux CAAM driver as if u-boot has initialised the h/w
For #2 I actually have to variants
1. Which passes a DT parameter which indicates the kernel should skip RNG init
2. A module parameter which indicates the kernel should skip rng init
Could we discuss the kernel changes in the kernel thread ?
I believe we agree the u-boot side is right ?
On Wed, 2018-01-24 at 19:41 +0000, Bryan O'Donoghue wrote:
On 24/01/18 17:41, Auer, Lukas wrote:
Thanks for adding me to the CC list. I have experienced the same thing regarding the dec0 registers. However, I don't understand why you want to detect secure mode in the kernel driver to skip RNG instantiation instead of instantiating all RNG state handles in the u-boot driver.
That's what we are doing though.
This set instantiates everything in u-boot and then detects and skips in the kernel if-and-only if
- Trust zone is detected
- It looks to the Linux CAAM driver as if u-boot has initialised the
h/w
For #2 I actually have to variants
- Which passes a DT parameter which indicates the kernel should
skip RNG init
- A module parameter which indicates the kernel should skip rng init
Could we discuss the kernel changes in the kernel thread ?
I believe we agree the u-boot side is right ?
Sorry, I haven't explained what I mean very well.
You are right in that sec_init() must be called to instantiate the RNG, however the CAAM u-boot driver only partially does so. If you look at function instantiate_rng() in both u-boot (drivers/crypto/fsl/jr.c) and the kernel (drivers/crypto/caam/ctrl.c), you'll see that the kernel loops over all available state handles whereas u-boot does not.
Fixing this in u-boot should mean that you can drop patch 5 and 6 from your kernel series since the kernel should then skip over all state handles.
I can send out a patch later today to fix this on the u-boot side.
Thanks, Lukas
On 25/01/18 09:14, Auer, Lukas wrote:
On Wed, 2018-01-24 at 19:41 +0000, Bryan O'Donoghue wrote:
On 24/01/18 17:41, Auer, Lukas wrote:
Thanks for adding me to the CC list. I have experienced the same thing regarding the dec0 registers. However, I don't understand why you want to detect secure mode in the kernel driver to skip RNG instantiation instead of instantiating all RNG state handles in the u-boot driver.
That's what we are doing though.
This set instantiates everything in u-boot and then detects and skips in the kernel if-and-only if
- Trust zone is detected
- It looks to the Linux CAAM driver as if u-boot has initialised the
h/w
For #2 I actually have to variants
- Which passes a DT parameter which indicates the kernel should
skip RNG init
- A module parameter which indicates the kernel should skip rng init
Could we discuss the kernel changes in the kernel thread ?
I believe we agree the u-boot side is right ?
Sorry, I haven't explained what I mean very well.
You are right in that sec_init() must be called to instantiate the RNG, however the CAAM u-boot driver only partially does so. If you look at function instantiate_rng() in both u-boot (drivers/crypto/fsl/jr.c) and the kernel (drivers/crypto/caam/ctrl.c), you'll see that the kernel loops over all available state handles whereas u-boot does not.
Fixing this in u-boot should mean that you can drop patch 5 and 6 from your kernel series since the kernel should then skip over all state handles.
Are you sure about that ?
It looks to me as if we will hit this block of code fairly decisively without #5 and #6
clrsetbits_32(&ctrl->deco_rq, 0, DECORR_RQD0ENABLE);
while (!(rd_reg32(&ctrl->deco_rq) & DECORR_DEN0) && --timeout) cpu_relax();
if (!timeout) { dev_err(ctrldev, "failed to acquire DECO 0\n"); clrsetbits_32(&ctrl->deco_rq, DECORR_RQD0ENABLE, 0); return -ENODEV; }
... but again let's discuss that in the kernel thread.
I can send out a patch later today to fix this on the u-boot side.
I'll certainly try out your patch on top of these patches.
Thanks, Lukas
On 23 January 2018 at 21:10, Bryan O'Donoghue bryan.odonoghue@linaro.org wrote:
This series is the u-boot fix to a problem we encountered when enabling OPTEE/TrustZone on the WaRP7. The symptom is once TrustZone is activated the first page of CAAM registers becomes read-only, read-zero from the perspective of Linux and other non TrustZone contexts.
Offlining the problem with Peng Fan[1] we eventually came to realise the problem could be worked around by
Making Linux skip RNG initialisation - a set of patches should be hitting LKML to do just that.
Initialising the RNG either from u-boot or OPTEE. In this case u-boot is the right place to-do that because there's upstream code in u-boot that just works. Patch #2 does that for the WaRP7.
Ensuring the job-ring registers are assigned to the non TrustZone mode. On the i.MX7 after the BootROM runs the job-ring registers are assigned to TrustZone. Patch #1 does that for all CAAM hardware.
On point #3 this ordinarily isn't a problem because unless TrustZone is activated the restrictions on the job-ring registers don't kick in, its only after enabling TrustZone that Linux will loose access to the job-ring registers.
Finally should OPTEE or another TEE want to do things with the job-ring registers it will have sufficient privilege to assign whichever job-ring registers it wants to OPTEE/TEE but will naturally then have to arbitrate with Linux to inform the Kernel CAAM driver which job-ring registers it can and cannot access.
That arbitration process is for a future putative OPTEE/TEE CAAM driver to solve and is out of scope of this patchset.
[1] Thanks for all of your help BTW - Peng, there's no way this would be working without you giving direction on how.
Bryan O'Donoghue (2): drivers/crypto/fsl: assign job-rings to non-TrustZone warp7 : run sec_init for CAAM RNG
This series:
Tested-by: Ryan Harkin ryan.harkin@linaro.org
board/warp7/warp7.c | 6 +++++- drivers/crypto/fsl/jr.c | 9 +++++++++ drivers/crypto/fsl/jr.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)
-- 2.7.4
U-Boot mailing list U-Boot@lists.denx.de https://lists.denx.de/listinfo/u-boot
participants (3)
-
Auer, Lukas -
Bryan O'Donoghue -
Ryan Harkin