Hi Roger,
I was looking to test my system against U-Boot 2023.04. I'm running an OMAP 4460 SOM (I've been waiting to get kernel acceptance before U-Boot, but that has slipped) and it boots from NAND.
When I jumped from 2023.01 to 2023.04, I noticed I get spammed in the SPL by "omap-elm: uncorrectable ECC errors". I bisected, and this seems to be the result of commit 04fcd25873 ("mtd: rawnand: omap_gpmc: Fix BCH6/16 HW based correction").
Is the multi-sector ECC generation something that should work in a backward-compatible way? Do you know of anything that might be going wrong here?
Any advice would be greatly appreciated, as I find it is usually something simple I'm doing wrong that is causing most of my issues.
Thanks,
Colin Foster.
Hi Colin,
On 09/05/2023 18:31, Colin Foster wrote:
Hi Roger,
I was looking to test my system against U-Boot 2023.04. I'm running an OMAP 4460 SOM (I've been waiting to get kernel acceptance before U-Boot, but that has slipped) and it boots from NAND.
When I jumped from 2023.01 to 2023.04, I noticed I get spammed in the SPL by "omap-elm: uncorrectable ECC errors". I bisected, and this seems to be the result of commit 04fcd25873 ("mtd: rawnand: omap_gpmc: Fix BCH6/16 HW based correction").
oops. Sorry about that.
Is the multi-sector ECC generation something that should work in a backward-compatible way? Do you know of anything that might be going wrong here?
The patch wasn't supposed to break anything. I do not yet know what could be wrong. Most likely a wrong ECC configuration is being used.
Could you please share what ECC configuration you are using on your board? Can you please point me to the Linux device tree file if it exists?
Any advice would be greatly appreciated, as I find it is usually something simple I'm doing wrong that is causing most of my issues.
Thanks,
Colin Foster.
Hi Roger,
On Wed, May 10, 2023 at 12:42:43PM +0300, Roger Quadros wrote:
Hi Colin,
On 09/05/2023 18:31, Colin Foster wrote:
Hi Roger,
I was looking to test my system against U-Boot 2023.04. I'm running an OMAP 4460 SOM (I've been waiting to get kernel acceptance before U-Boot, but that has slipped) and it boots from NAND.
When I jumped from 2023.01 to 2023.04, I noticed I get spammed in the SPL by "omap-elm: uncorrectable ECC errors". I bisected, and this seems to be the result of commit 04fcd25873 ("mtd: rawnand: omap_gpmc: Fix BCH6/16 HW based correction").
oops. Sorry about that.
No worries!
Is the multi-sector ECC generation something that should work in a backward-compatible way? Do you know of anything that might be going wrong here?
The patch wasn't supposed to break anything. I do not yet know what could be wrong. Most likely a wrong ECC configuration is being used.
Could you please share what ECC configuration you are using on your board?
This is still out-of-U-Boot. I have an include/configs/our_product.h file with this:
""" #define CFG_SYS_NAND_ECCPOS {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, \ 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, \ 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, \ 33, 34, 35, 36, 37, 38, 39, 40, 41, \ 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, \ 52, 53, 54, 55, 56, 57}
#define CFG_SYS_NAND_ECCBYTES 14 #define CFG_SYS_NAND_MAX_ECCPOS 57 #define CFG_SYS_NAND_ECCSIZE 512 #define CFG_SYS_NAND_MAX_OOBFREE 2 """
Can you please point me to the Linux device tree file if it exists?
This is the latest submission. Still not accepted - I need to find time to button everything up and resubmit. My plan of attack was Kernel Acceptance, then U-Boot. Unfortunately my company lets the pesky "Shipping products" step get in the way :-)
https://lkml.org/lkml/2023/2/22/939
Or if you just want the ECC part:
+ nandflash: nand@0,0 { + compatible = "ti,omap2-nand"; + reg = <0 0 4>; + interrupt-parent = <&gpmc>; + + nand-bus-width = <16>; + ti,nand-ecc-opt = "bch8"; + ti,elm-id=<&elm>; + linux,mtd-name = "micron,nand";
I think that's all the info you're looking for. Let me know if I missed something.
Colin Foster
On 10/05/2023 18:38, Colin Foster wrote:
Hi Roger,
On Wed, May 10, 2023 at 12:42:43PM +0300, Roger Quadros wrote:
Hi Colin,
On 09/05/2023 18:31, Colin Foster wrote:
Hi Roger,
I was looking to test my system against U-Boot 2023.04. I'm running an OMAP 4460 SOM (I've been waiting to get kernel acceptance before U-Boot, but that has slipped) and it boots from NAND.
When I jumped from 2023.01 to 2023.04, I noticed I get spammed in the SPL by "omap-elm: uncorrectable ECC errors". I bisected, and this seems to be the result of commit 04fcd25873 ("mtd: rawnand: omap_gpmc: Fix BCH6/16 HW based correction").
oops. Sorry about that.
No worries!
Is the multi-sector ECC generation something that should work in a backward-compatible way? Do you know of anything that might be going wrong here?
The patch wasn't supposed to break anything. I do not yet know what could be wrong. Most likely a wrong ECC configuration is being used.
Could you please share what ECC configuration you are using on your board?
This is still out-of-U-Boot. I have an include/configs/our_product.h file with this:
""" #define CFG_SYS_NAND_ECCPOS {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, \ 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, \ 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, \ 33, 34, 35, 36, 37, 38, 39, 40, 41, \ 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, \ 52, 53, 54, 55, 56, 57}
#define CFG_SYS_NAND_ECCBYTES 14 #define CFG_SYS_NAND_MAX_ECCPOS 57
This should be 56 i.e. (57 - 2 + 1) But it won't fix the issue you are facing. :P
#define CFG_SYS_NAND_ECCSIZE 512 #define CFG_SYS_NAND_MAX_OOBFREE 2 """
Can you please point me to the Linux device tree file if it exists?
This is the latest submission. Still not accepted - I need to find time to button everything up and resubmit. My plan of attack was Kernel Acceptance, then U-Boot. Unfortunately my company lets the pesky "Shipping products" step get in the way :-)
https://lkml.org/lkml/2023/2/22/939
Or if you just want the ECC part:
- nandflash: nand@0,0 {
compatible = "ti,omap2-nand";reg = <0 0 4>;interrupt-parent = <&gpmc>;nand-bus-width = <16>;ti,nand-ecc-opt = "bch8";ti,elm-id=<&elm>;linux,mtd-name = "micron,nand";I think that's all the info you're looking for. Let me know if I missed something.
Yes this is all I was looking for.
Is CONFIG_NAND_OMAP_ECCSCHEME_BCH8_CODE_HW set in your u-boot config?
Hi Roger,
On Fri, May 12, 2023 at 02:53:07PM +0300, Roger Quadros wrote:
On 10/05/2023 18:38, Colin Foster wrote:
This is still out-of-U-Boot. I have an include/configs/our_product.h file with this:
""" #define CFG_SYS_NAND_ECCPOS {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, \ 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, \ 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, \ 33, 34, 35, 36, 37, 38, 39, 40, 41, \ 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, \ 52, 53, 54, 55, 56, 57}
#define CFG_SYS_NAND_ECCBYTES 14 #define CFG_SYS_NAND_MAX_ECCPOS 57
This should be 56 i.e. (57 - 2 + 1) But it won't fix the issue you are facing. :P
Oh, good catch. I know when I was trying to get this working I had to play with these values quite a bit. I must have missed changing this at one point.
#define CFG_SYS_NAND_ECCSIZE 512 #define CFG_SYS_NAND_MAX_OOBFREE 2 """
Can you please point me to the Linux device tree file if it exists?
This is the latest submission. Still not accepted - I need to find time to button everything up and resubmit. My plan of attack was Kernel Acceptance, then U-Boot. Unfortunately my company lets the pesky "Shipping products" step get in the way :-)
https://lkml.org/lkml/2023/2/22/939
Or if you just want the ECC part:
- nandflash: nand@0,0 {
compatible = "ti,omap2-nand";reg = <0 0 4>;interrupt-parent = <&gpmc>;nand-bus-width = <16>;ti,nand-ecc-opt = "bch8";ti,elm-id=<&elm>;linux,mtd-name = "micron,nand";I think that's all the info you're looking for. Let me know if I missed something.
Yes this is all I was looking for.
Is CONFIG_NAND_OMAP_ECCSCHEME_BCH8_CODE_HW set in your u-boot config?
Yes, it is set. I've attached our .config file. I just made a change to CONFIG_NAND_OMAP_GPMC_PREFETCH as a test, which didn't fix the issue.
And as another sanity check, I reverted the patch and have functionality again:
""" U-Boot SPL 2023.04-00029-g316faf4c46-dirty (May 12 2023 - 09:02:58 -0700) OMAP4460-GP ES1.1 Trying to boot from NAND
U-Boot 2023.04-00029-g316faf4c46-dirty (May 12 2023 - 09:02:58 -0700)
CPU : OMAP4460-GP ES1.1 """
-- cheers, -roger
On 12/05/2023 19:05, Colin Foster wrote:
Hi Roger,
On Fri, May 12, 2023 at 02:53:07PM +0300, Roger Quadros wrote:
On 10/05/2023 18:38, Colin Foster wrote:
This is still out-of-U-Boot. I have an include/configs/our_product.h file with this:
""" #define CFG_SYS_NAND_ECCPOS {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, \ 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, \ 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, \ 33, 34, 35, 36, 37, 38, 39, 40, 41, \ 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, \ 52, 53, 54, 55, 56, 57}
#define CFG_SYS_NAND_ECCBYTES 14 #define CFG_SYS_NAND_MAX_ECCPOS 57
This should be 56 i.e. (57 - 2 + 1) But it won't fix the issue you are facing. :P
Oh, good catch. I know when I was trying to get this working I had to play with these values quite a bit. I must have missed changing this at one point.
#define CFG_SYS_NAND_ECCSIZE 512 #define CFG_SYS_NAND_MAX_OOBFREE 2 """
Can you please point me to the Linux device tree file if it exists?
This is the latest submission. Still not accepted - I need to find time to button everything up and resubmit. My plan of attack was Kernel Acceptance, then U-Boot. Unfortunately my company lets the pesky "Shipping products" step get in the way :-)
https://lkml.org/lkml/2023/2/22/939
Or if you just want the ECC part:
- nandflash: nand@0,0 {
compatible = "ti,omap2-nand";reg = <0 0 4>;interrupt-parent = <&gpmc>;nand-bus-width = <16>;ti,nand-ecc-opt = "bch8";ti,elm-id=<&elm>;linux,mtd-name = "micron,nand";I think that's all the info you're looking for. Let me know if I missed something.
Yes this is all I was looking for.
Is CONFIG_NAND_OMAP_ECCSCHEME_BCH8_CODE_HW set in your u-boot config?
Yes, it is set. I've attached our .config file. I just made a change to CONFIG_NAND_OMAP_GPMC_PREFETCH as a test, which didn't fix the issue.
And as another sanity check, I reverted the patch and have functionality again:
""" U-Boot SPL 2023.04-00029-g316faf4c46-dirty (May 12 2023 - 09:02:58 -0700) OMAP4460-GP ES1.1 Trying to boot from NAND
U-Boot 2023.04-00029-g316faf4c46-dirty (May 12 2023 - 09:02:58 -0700)
CPU : OMAP4460-GP ES1.1 """
OK. next step is to dump calculated ECC values while reading a block and see if it is different in failing case and then debug why. Let me prepare a patch for that.
Hi Colin,
On 12/05/2023 19:05, Colin Foster wrote:
Hi Roger,
On Fri, May 12, 2023 at 02:53:07PM +0300, Roger Quadros wrote:
On 10/05/2023 18:38, Colin Foster wrote:
This is still out-of-U-Boot. I have an include/configs/our_product.h file with this:
""" #define CFG_SYS_NAND_ECCPOS {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, \ 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, \ 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, \ 33, 34, 35, 36, 37, 38, 39, 40, 41, \ 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, \ 52, 53, 54, 55, 56, 57}
#define CFG_SYS_NAND_ECCBYTES 14 #define CFG_SYS_NAND_MAX_ECCPOS 57
This should be 56 i.e. (57 - 2 + 1) But it won't fix the issue you are facing. :P
Oh, good catch. I know when I was trying to get this working I had to play with these values quite a bit. I must have missed changing this at one point.
#define CFG_SYS_NAND_ECCSIZE 512 #define CFG_SYS_NAND_MAX_OOBFREE 2 """
Can you please point me to the Linux device tree file if it exists?
This is the latest submission. Still not accepted - I need to find time to button everything up and resubmit. My plan of attack was Kernel Acceptance, then U-Boot. Unfortunately my company lets the pesky "Shipping products" step get in the way :-)
https://lkml.org/lkml/2023/2/22/939
Or if you just want the ECC part:
- nandflash: nand@0,0 {
compatible = "ti,omap2-nand";reg = <0 0 4>;interrupt-parent = <&gpmc>;nand-bus-width = <16>;ti,nand-ecc-opt = "bch8";ti,elm-id=<&elm>;linux,mtd-name = "micron,nand";I think that's all the info you're looking for. Let me know if I missed something.
Yes this is all I was looking for.
Is CONFIG_NAND_OMAP_ECCSCHEME_BCH8_CODE_HW set in your u-boot config?
Yes, it is set. I've attached our .config file. I just made a change to CONFIG_NAND_OMAP_GPMC_PREFETCH as a test, which didn't fix the issue.
And as another sanity check, I reverted the patch and have functionality again:
I just tested this on AM335x EVM which uses BCH8_CODE_HW but 8-bit NAND part. I see that you are using 16-bit NAND.
One more difference in u-boot configuration. For me: CONFIG_NAND_OMAP_GPMC_PREFETCH=y
Not sure if that matters but let's keep it set for now.
For debug can you please apply the patch (at end) to u-boot at commit a95410696d21 (before breakage) and run the test.
Test procedure:
nand dump 0
mmc dev 0
#replace 0 with whatever points to SD-card
nand read $loadaddr 0 800
#loadaddr is any address in RAM which is free for use.
fatwrite mmc 0:1 $loadaddr nand.org 800
- restore NAND page 0
fatload mmc 0:1 $loadaddr nand.org nand write $loadaddr 0 800
Now please run the below test on the offending commit 04fcd25873 after applying the patch (at end).
nand dump 0 mmc dev 0
#replace 0 with whatever points to SD-card
fatload mmc 0:1 $loadaddr nand.org nand write $loadaddr 0 800
Please send me the logs of both tests and the nand.org file so I can try it out here. Thanks!
--- patch starts--- diff --git a/drivers/mtd/nand/raw/omap_gpmc.c b/drivers/mtd/nand/raw/omap_gpmc.c index be3cb3c601..ac06d5f019 100644 --- a/drivers/mtd/nand/raw/omap_gpmc.c +++ b/drivers/mtd/nand/raw/omap_gpmc.c @@ -300,6 +300,7 @@ static int omap_calculate_ecc(struct mtd_info *mtd, const uint8_t *dat, ecc_code[i++] = (val >> 0) & 0xFF; ptr--; } + printf("ecc: %x\n", val); break; case OMAP_ECC_BCH16_CODE_HW: val = readl(&gpmc_cfg->bch_result_4_6[0].bch_result_x[2]);
Hi Roger,
Thanks for the tests. I attached the files and commented in line... but at the bottom of this email I have some findings...
On Wed, May 17, 2023 at 04:30:55PM +0300, Roger Quadros wrote:
Hi Colin,
I just tested this on AM335x EVM which uses BCH8_CODE_HW but 8-bit NAND part. I see that you are using 16-bit NAND.
One more difference in u-boot configuration. For me: CONFIG_NAND_OMAP_GPMC_PREFETCH=y
Not sure if that matters but let's keep it set for now.
For debug can you please apply the patch (at end) to u-boot at commit a95410696d21 (before breakage) and run the test.
Test procedure:
nand dump 0
=> nand dump 0 Page 00000000 dump: 40 00 00 00 0c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 43 48 53 45 54 54 49 4e 47 53 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff c1 c0 c0 c0 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 dc 99 00 00 00 00 30 40 ff ff ff ea 1a 06 00 ea 00 00 0f e1 1f 10 00 e2 1a 00 31 e3 1f 00 c0 13 13 00 80 13 c0 00 80 e3 00 f0 29 e1 06 00 00 eb 1d 00 00 eb 8a 06 00 eb 15 0f 07 ee 9a 0f 07 ee 95 0f 07 ee 1e ff 2f e1 f0 ff ff ea 00 00 a0 e3 17 0f 08 ee 15 0f 07 ee d5 0f 07 ee 9a 0f 07 ee 95 0f 07 ee 10 0f 11 ee 02 0a c0 e3 07 00 c0 e3 02 00 80 e3 02 0b 80 e3 01 0a 80 e3 10 0f 01 ee 0e 50 a0 e1 10 1f 10 ee 21 3a a0 e1 0f 30 03 e2 0f 40 01 e2 03 22 a0 e1 02 20 84 e1 0c 00 9f e5 07 00 c0 e3 00 d0 a0 e1 05 f0 a0 e1 64 06 00 ea 00 df 30 40 06 4b 1b 68 1b 68 1b 68 1b 68 bf f3 5f 8f 04 4a 03 f0 1f 03 52 f8 23 00 70 47 00 bf 90 92 30 40 68 6f 30 40 01 4b 03 60 70 47 00 bf 28 86 30 40 00 20 70 47 30 b5 47 4b 19 68 03 f1 80 63 a3 f5 3b 43 99 42 79 d0 44 4a 91 42 0c bf 4f f0 9e 32 4f f0 7c 32 bf f3 5f 8f 40 4b 18 68 00 68 d0 f8 e4 00 02 60 bf f3 5f 8f 18 68 22 f4 c0 24 00 68 d0 f8 e8 00 02 60 bf f3 5f 8f 18 68 00 68 d0 f8 ec 00 04 60 bf f3 5f 8f 18 68 00 68 d0 f8 f0 50 33 48 28 60 bf f3 5f 8f 1d 68 2d 68 d5 f8 f4 50 2a 60 bf f3 5f 8f 1d 68 2d 68 d5 f8 f8 50 2a 60 bf f3 5f 8f 1a 68 12 68 d2 f8 fc 20 14 60 bf f3 5f 8f 1a 68 12 68 d2 f8 00 21 10 60 1a 68 12 68 92 6c 12 68 bf f3 5f 8f 92 b2 92 b9 bf f3 5f 8f 1a 68 12 68 d0 6c 1f 4a 02 60 bf f3 5f 8f 18 68 00 68 00 6d 02 60 bf f3 5f 8f 18 68 00 68 40 6d 02 60 1a 68 12 68 d2 f8 84 21 12 68 bf f3 5f 8f 3a b9 bf f3 5f 8f 1a 68 13 48 12 68 d2 f8 84 21 10 60 12 4a 91 42 0b d8 bf f3 5f 8f 1b 68 10 4a 1b 68 d3 f8 88 31 1a 60 30 bd 4f f0 1c 32 89 e7 1a 68 12 68 d2 f8 88 21 12 68 bf f3 5f 8f 00 2a ea d0 f1 e7 00 bc 30 40 00 02 30 44 90 92 30 40 0f 8c 88 a0 0f 04 01 04 10 01 4d 1c ff 00 60 44 00 40 08 99 08 b5 00 f0 50 fa 1f 4b 98 42 a3 f5 6f 03 a3 f2 93 43 31 d0 0a d8 1c 4a 90 42 19 d0 02 f5 80 12 01 32 90 42 17 d0 4f f0 ff 32 12 e0 17 4a 90 42 f9 d1 17 4a 12 68 bf f3 5f 8f 16 49 8a 42 1d d0 01 f1 00 51 a1 f5 1c 31 8a 42 19 d0 12 4a 00 e0 12 4a 1a 60 08 bd 0e 4a 12 68 bf f3 5f 8f 10 49 8a 42 05 d0 01 f1 80 51 8a 42 03 d0 0d 4a f0 e7 0d 4a ee e7 0d 4a ec e7 0d 4a ea e7 0d 4a e8 e7 0d 4a e6 e7 93 c0 1f 41 91 c0 0f 41 9a c0 2f 41 04 22 00 4a 2f 50 97 0b 00 01 60 44 00 01 30 44 2f c0 95 3b 00 02 30 44 10 02 30 44 20 02 30 44 30 02 30 44 00 01 70 44 10 01 60 44 10 4b 1a 68 12 68 d2 f8 30 21 12 68 bf f3 5f 8f 22 f0 88 62 bf f3 5f 8f 19 68 09 68 d1 f8 30 11 0a 60 1a 68 12 68 d2 f8 30 21 12 68 bf f3 5f 8f 42 f0 8c 62 bf f3 5f 8f 1b 68 1b 68 d3 f8 30 31 1a 60 70 47 90 92 30 40 00 b5 4f 4a 9d b0 13 68 19 68 d1 f8 1c 13 01 91 19 68 d1 f8 dc 12 02 91 19 68 d1 f8 38 12 03 91 19 68 d1 f8 74 12 04 91 00 21 05 91 18 68 d0 f8 10 02 06 90 18 68 d0 f8 44 02 07 90 18 68 d0 f8 48 02 08 90 18 68 d0 f8 7c 02 09 90 18 68 d0 f8 14 04 0a 90 18 68 d0 f8 44 03 0b 90 18 68 d0 f8 48 03 0c 90 18 68 d0 f8 4c 03 0d 90 18 68 d0 f8 50 03 0e 90 18 68 10 91 d0 f8 54 03 0f 90 18 68 d0 f8 18 04 11 90 18 68 d0 f8 e0 02 12 90 18 68 d0 f8 e4 02 13 90 18 68 d0 f8 30 03 14 90 18 68 d0 f8 10 04 15 90 18 68 d0 f8 b8 03 16 90 18 68 d0 f8 64 03 17 90 18 68 d0 f8 68 03 18 90 18 68 d0 f8 6c 03 19 90 18 68 1b 91 d0 f8 70 03 1a 90 1b 68 01 a8 d3 f8 4c 13 0b 68 43 f4 80 73 0b 60 13 68 1b 68 d3 f8 e0 12 0b 68 43 f0 80 73 0b 60 13 68 1b 68 d3 f8 e4 12 0b 68 43 f0 80 73 0b 60 13 68 1b 68 d3 f8 18 14 0b 68 43 f0 80 73 0b 60 13 68 1b 68 d3 f8 bc 14 0b 68 43 f4 80 73 0b 60 13 68 06 a9 1b 68 d3 f8 e0 21 13 68 43 f4 80 73 13 60 01 23 11 aa 00 f0 1f fd 1d b0 5d f8 04 fb bc 97 30 40 08 b5 27 4a 27 4b 12 68 1b 68 27 49 1b 68 11 60 26 4a 93 42 40 d0 0d d8 20 3a 93 42 34 d0 10 32 93 42 39 d0 a2 f5 88 72 93 42 20 d0 20 48 02 f0 09 fb 24 e0 1f 4a 93 42 06 d0 0d d8 1e 4a 93 42 2a d0 1e 4a 93 42 f1 d1 1d 4b 1e 4a 1b 68 1a 60 1d 4b 1e 4a 1b 68 11 e0 1d 4a 93 42 e6 d1 18 4b 1c 4a 1b 68 1a 60 18 4b 1b 4a 1b 68 06 e0 14 4b 1a 4a 1b 68 1a 60 14 4b 19 4a 1b 68 1a 60 19 4b 19 4a 1b 68 1a 60 08 bd 0d 4b 18 4a 1b 68 1a 60 0d 4b 17 4a 1b 68 f1 e7 09 4b 16 4a 1b 68 f6 e7 bc 97 30 40 bc 70 30 40 88 72 30 40 20 02 30 44 2d 86 30 40 10 01 60 44 30 02 30 44 00 01 60 44 94 92 30 40 38 95 30 40 b8 97 30 40 58 95 30 40 00 01 70 44 78 96 30 40 98 96 30 40 b8 92 30 40 18 94 30 40 90 92 30 40 c0 70 30 40 d8 92 30 40 f8 92 30 40 98 92 30 40 04 4b 18 68 bf f3 5f 8f 20 f0 15 00 40 05 40 0d 70 47 00 bf 04 7b 30 4a 70 47 00 00 10 b5 0f 4b 98 6a bf f3 5f 8f 0e 4b d9 f8 c4 20 a0 fb 03 01 b2 eb 51 2f 4f ea 51 24 d9 f8 c4 10 d9 f8 c0 20 22 44 84 bf 02 f5 5a 22 02 f5 e8 62 52 1a c9 f8 c0 20 c9 f8 c4 40 d9 f8 c0 00 10 bd 00 20 03 48 b5 81 4e 1b bf f3 5f 8f 00 20 03 4b d8 62 bf f3 5f 8f 2b 22 5a 62 70 47 00 20 03 48 10 b5 04 46 ff f7 cc ff 00 1b 10 bd 4f f4 96 53 58 43 09 4b 09 4a a0 fb 03 01 8b 09 OOB: ff ff c8 b0 e8 12 b2 4a 63 70 83 0f c6 76 57 00 0d 20 e0 06 66 b6 1a b4 03 f7 8e d4 44 00 8d 04 79 b3 3c e6 30 0f bf da d5 af 87 00 ef a4 e7 b4 07 db 1e b0 1c 2d bc 31 3b 00 ff ff ff ff ff ff
mmc dev 0
#replace 0 with whatever points to SD-card
=> mmc dev 0 switch to partitions #0, OK mmc0 is current device
nand read $loadaddr 0 800
#loadaddr is any address in RAM which is free for use.
=> nand read $loadaddr 0 800
NAND read: device 0 offset 0x0, size 0x800 Scanning device for bad blocks ecc: 0 ecc: 0 ecc: 0 ecc: 0 2048 bytes read: OK
fatwrite mmc 0:1 $loadaddr nand.org 800
=> fatwrite mmc 0:1 $loadaddr nand.org 800 2048 bytes written in 3 ms (666 KiB/s)
fatload mmc 0:1 $loadaddr nand.org
=> fatload mmc 0:1 $loadaddr nand.org 2048 bytes read in 10 ms (199.2 KiB/s)
nand write $loadaddr 0 800
=> nand write $loadaddr 0 800
NAND write: device 0 offset 0x0, size 0x800 ecc: fc67657 ecc: f78ed444 ecc: dad5af87 ecc: 2dbc313b ecc: 0 ecc: 0 ecc: 0 ecc: 0 2048 bytes written: OK
Now please run the below test on the offending commit 04fcd25873 after applying the patch (at end).
nand dump 0
=> nand dump 0 Page 00000000 dump: 40 00 00 00 0c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 43 48 53 45 54 54 49 4e 47 53 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff c1 c0 c0 c0 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 dc 99 00 00 00 00 30 40 ff ff ff ea 1a 06 00 ea 00 00 0f e1 1f 10 00 e2 1a 00 31 e3 1f 00 c0 13 13 00 80 13 c0 00 80 e3 00 f0 29 e1 06 00 00 eb 1d 00 00 eb 8a 06 00 eb 15 0f 07 ee 9a 0f 07 ee 95 0f 07 ee 1e ff 2f e1 f0 ff ff ea 00 00 a0 e3 17 0f 08 ee 15 0f 07 ee d5 0f 07 ee 9a 0f 07 ee 95 0f 07 ee 10 0f 11 ee 02 0a c0 e3 07 00 c0 e3 02 00 80 e3 02 0b 80 e3 01 0a 80 e3 10 0f 01 ee 0e 50 a0 e1 10 1f 10 ee 21 3a a0 e1 0f 30 03 e2 0f 40 01 e2 03 22 a0 e1 02 20 84 e1 0c 00 9f e5 07 00 c0 e3 00 d0 a0 e1 05 f0 a0 e1 64 06 00 ea 00 df 30 40 06 4b 1b 68 1b 68 1b 68 1b 68 bf f3 5f 8f 04 4a 03 f0 1f 03 52 f8 23 00 70 47 00 bf 90 92 30 40 68 6f 30 40 01 4b 03 60 70 47 00 bf 28 86 30 40 00 20 70 47 30 b5 47 4b 19 68 03 f1 80 63 a3 f5 3b 43 99 42 79 d0 44 4a 91 42 0c bf 4f f0 9e 32 4f f0 7c 32 bf f3 5f 8f 40 4b 18 68 00 68 d0 f8 e4 00 02 60 bf f3 5f 8f 18 68 22 f4 c0 24 00 68 d0 f8 e8 00 02 60 bf f3 5f 8f 18 68 00 68 d0 f8 ec 00 04 60 bf f3 5f 8f 18 68 00 68 d0 f8 f0 50 33 48 28 60 bf f3 5f 8f 1d 68 2d 68 d5 f8 f4 50 2a 60 bf f3 5f 8f 1d 68 2d 68 d5 f8 f8 50 2a 60 bf f3 5f 8f 1a 68 12 68 d2 f8 fc 20 14 60 bf f3 5f 8f 1a 68 12 68 d2 f8 00 21 10 60 1a 68 12 68 92 6c 12 68 bf f3 5f 8f 92 b2 92 b9 bf f3 5f 8f 1a 68 12 68 d0 6c 1f 4a 02 60 bf f3 5f 8f 18 68 00 68 00 6d 02 60 bf f3 5f 8f 18 68 00 68 40 6d 02 60 1a 68 12 68 d2 f8 84 21 12 68 bf f3 5f 8f 3a b9 bf f3 5f 8f 1a 68 13 48 12 68 d2 f8 84 21 10 60 12 4a 91 42 0b d8 bf f3 5f 8f 1b 68 10 4a 1b 68 d3 f8 88 31 1a 60 30 bd 4f f0 1c 32 89 e7 1a 68 12 68 d2 f8 88 21 12 68 bf f3 5f 8f 00 2a ea d0 f1 e7 00 bc 30 40 00 02 30 44 90 92 30 40 0f 8c 88 a0 0f 04 01 04 10 01 4d 1c ff 00 60 44 00 40 08 99 08 b5 00 f0 50 fa 1f 4b 98 42 a3 f5 6f 03 a3 f2 93 43 31 d0 0a d8 1c 4a 90 42 19 d0 02 f5 80 12 01 32 90 42 17 d0 4f f0 ff 32 12 e0 17 4a 90 42 f9 d1 17 4a 12 68 bf f3 5f 8f 16 49 8a 42 1d d0 01 f1 00 51 a1 f5 1c 31 8a 42 19 d0 12 4a 00 e0 12 4a 1a 60 08 bd 0e 4a 12 68 bf f3 5f 8f 10 49 8a 42 05 d0 01 f1 80 51 8a 42 03 d0 0d 4a f0 e7 0d 4a ee e7 0d 4a ec e7 0d 4a ea e7 0d 4a e8 e7 0d 4a e6 e7 93 c0 1f 41 91 c0 0f 41 9a c0 2f 41 04 22 00 4a 2f 50 97 0b 00 01 60 44 00 01 30 44 2f c0 95 3b 00 02 30 44 10 02 30 44 20 02 30 44 30 02 30 44 00 01 70 44 10 01 60 44 10 4b 1a 68 12 68 d2 f8 30 21 12 68 bf f3 5f 8f 22 f0 88 62 bf f3 5f 8f 19 68 09 68 d1 f8 30 11 0a 60 1a 68 12 68 d2 f8 30 21 12 68 bf f3 5f 8f 42 f0 8c 62 bf f3 5f 8f 1b 68 1b 68 d3 f8 30 31 1a 60 70 47 90 92 30 40 00 b5 4f 4a 9d b0 13 68 19 68 d1 f8 1c 13 01 91 19 68 d1 f8 dc 12 02 91 19 68 d1 f8 38 12 03 91 19 68 d1 f8 74 12 04 91 00 21 05 91 18 68 d0 f8 10 02 06 90 18 68 d0 f8 44 02 07 90 18 68 d0 f8 48 02 08 90 18 68 d0 f8 7c 02 09 90 18 68 d0 f8 14 04 0a 90 18 68 d0 f8 44 03 0b 90 18 68 d0 f8 48 03 0c 90 18 68 d0 f8 4c 03 0d 90 18 68 d0 f8 50 03 0e 90 18 68 10 91 d0 f8 54 03 0f 90 18 68 d0 f8 18 04 11 90 18 68 d0 f8 e0 02 12 90 18 68 d0 f8 e4 02 13 90 18 68 d0 f8 30 03 14 90 18 68 d0 f8 10 04 15 90 18 68 d0 f8 b8 03 16 90 18 68 d0 f8 64 03 17 90 18 68 d0 f8 68 03 18 90 18 68 d0 f8 6c 03 19 90 18 68 1b 91 d0 f8 70 03 1a 90 1b 68 01 a8 d3 f8 4c 13 0b 68 43 f4 80 73 0b 60 13 68 1b 68 d3 f8 e0 12 0b 68 43 f0 80 73 0b 60 13 68 1b 68 d3 f8 e4 12 0b 68 43 f0 80 73 0b 60 13 68 1b 68 d3 f8 18 14 0b 68 43 f0 80 73 0b 60 13 68 1b 68 d3 f8 bc 14 0b 68 43 f4 80 73 0b 60 13 68 06 a9 1b 68 d3 f8 e0 21 13 68 43 f4 80 73 13 60 01 23 11 aa 00 f0 1f fd 1d b0 5d f8 04 fb bc 97 30 40 08 b5 27 4a 27 4b 12 68 1b 68 27 49 1b 68 11 60 26 4a 93 42 40 d0 0d d8 20 3a 93 42 34 d0 10 32 93 42 39 d0 a2 f5 88 72 93 42 20 d0 20 48 02 f0 09 fb 24 e0 1f 4a 93 42 06 d0 0d d8 1e 4a 93 42 2a d0 1e 4a 93 42 f1 d1 1d 4b 1e 4a 1b 68 1a 60 1d 4b 1e 4a 1b 68 11 e0 1d 4a 93 42 e6 d1 18 4b 1c 4a 1b 68 1a 60 18 4b 1b 4a 1b 68 06 e0 14 4b 1a 4a 1b 68 1a 60 14 4b 19 4a 1b 68 1a 60 19 4b 19 4a 1b 68 1a 60 08 bd 0d 4b 18 4a 1b 68 1a 60 0d 4b 17 4a 1b 68 f1 e7 09 4b 16 4a 1b 68 f6 e7 bc 97 30 40 bc 70 30 40 88 72 30 40 20 02 30 44 2d 86 30 40 10 01 60 44 30 02 30 44 00 01 60 44 94 92 30 40 38 95 30 40 b8 97 30 40 58 95 30 40 00 01 70 44 78 96 30 40 98 96 30 40 b8 92 30 40 18 94 30 40 90 92 30 40 c0 70 30 40 d8 92 30 40 f8 92 30 40 98 92 30 40 04 4b 18 68 bf f3 5f 8f 20 f0 15 00 40 05 40 0d 70 47 00 bf 04 7b 30 4a 70 47 00 00 10 b5 0f 4b 98 6a bf f3 5f 8f 0e 4b d9 f8 c4 20 a0 fb 03 01 b2 eb 51 2f 4f ea 51 24 d9 f8 c4 10 d9 f8 c0 20 22 44 84 bf 02 f5 5a 22 02 f5 e8 62 52 1a c9 f8 c0 20 c9 f8 c4 40 d9 f8 c0 00 10 bd 00 20 03 48 b5 81 4e 1b bf f3 5f 8f 00 20 03 4b d8 62 bf f3 5f 8f 2b 22 5a 62 70 47 00 20 03 48 10 b5 04 46 ff f7 cc ff 00 1b 10 bd 4f f4 96 53 58 43 09 4b 09 4a a0 fb 03 01 8b 09 OOB: ff ff c8 b0 e8 12 b2 4a 63 70 83 0f c6 76 57 00 0d 20 e0 06 66 b6 1a b4 03 f7 8e d4 44 00 8d 04 79 b3 3c e6 30 0f bf da d5 af 87 00 ef a4 e7 b4 07 db 1e b0 1c 2d bc 31 3b 00 ff ff ff ff ff ff
mmc dev 0
#replace 0 with whatever points to SD-card
=> mmc dev 0 switch to partitions #0, OK mmc0 is current device
""" NOTE """ # Here I did a nand read, and re-write to the "nand.post" file. It # shouldn't affect the results, but I kept the commands and output for # transparency.
=> nand read $loadaddr 0 800
NAND read: device 0 offset 0x0, size 0x800 Scanning device for bad blocks ecc: 0 ecc: 0 ecc: 0 ecc: 0 2048 bytes read: OK => fatwrite mmc 0:1 $loadaddr nand.post 800 2048 bytes written in 3 ms (666 KiB/s)
fatload mmc 0:1 $loadaddr nand.org
=> fatload mmc 0:1 $loadaddr nand.org 2048 bytes read in 10 ms (199.2 KiB/s)
nand write $loadaddr 0 800
=> nand write $loadaddr 0 800
NAND write: device 0 offset 0x0, size 0x800 ecc: fc67657 ecc: f78ed444 ecc: dad5af87 ecc: 2dbc313b ecc: 0 ecc: 0 ecc: 0 ecc: 0 2048 bytes written: OK
Please send me the logs of both tests and the nand.org file so I can try it out here. Thanks!
All are attached.
However I have some other initial findings:
I swapped in just U-Boot (not the SPL) with your patch, and everything seems to work!
The issue of Uncorrectable ECC errors spam came from the SPL. Here's a snippet of the boot log with the "ecc" print as well as your patch:
U-Boot SPL 2023.04-00029-g26a9ce5314-dirty (May 17 2023 - 12:06:49 -0700) OMAP4460-GP ES1.1 Trying to boot from NAND ecc: 2420106 ecc: ebd922f6 ecc: 333f844f ecc: ab812f72 omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors ecc: 2420106 ecc: ebd922f6 ecc: 333f844f ecc: ab812f72 omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors
Historically when I saw "uncorrectable ECC errors" that meant I should grab a cup of coffee, problems were going to happen. However, in this scenario, if I let the SPL spew the ECC error prints, it eventually loads U-Boot from NAND. And U-Boot successfully loads the kernel, DTB, and boots the filesystem (with the "ecc: %x" prints slowing things down right now)
Here's where U-Boot loads the DTB:
Loading file 'boot/omap4-ia-decoder-card.dtb' to addr 0x88000000... ecc: 0 ... Repeated over and over ... ecc: 0 Done Unmounting UBIFS volume rootfs-a!
So I'm thinking either:
1. I'm doing something horribly wrong and have been getting away with it. The patch simply highlights this. Or...
2. There's an issue with how the SPL is determining ECC errors, and it is incorrectly reporting errors.
I hope this information is useful, and let me know if there are any other tests I can run.
Many thanks,
Colin Foster
Hi Colin,
On 17/05/2023 22:39, Colin Foster wrote:
Hi Roger,
Thanks for the tests. I attached the files and commented in line... but at the bottom of this email I have some findings...
On Wed, May 17, 2023 at 04:30:55PM +0300, Roger Quadros wrote:
Hi Colin,
I just tested this on AM335x EVM which uses BCH8_CODE_HW but 8-bit NAND part. I see that you are using 16-bit NAND.
One more difference in u-boot configuration. For me: CONFIG_NAND_OMAP_GPMC_PREFETCH=y
Not sure if that matters but let's keep it set for now.
For debug can you please apply the patch (at end) to u-boot at commit a95410696d21 (before breakage) and run the test.
All are attached.
However I have some other initial findings:
I swapped in just U-Boot (not the SPL) with your patch, and everything seems to work!
The issue of Uncorrectable ECC errors spam came from the SPL. Here's a snippet of the boot log with the "ecc" print as well as your patch:
Thanks for the tests. Glad to hear issue is narrowed down to SPL.
U-Boot SPL 2023.04-00029-g26a9ce5314-dirty (May 17 2023 - 12:06:49 -0700) OMAP4460-GP ES1.1 Trying to boot from NAND ecc: 2420106 ecc: ebd922f6 ecc: 333f844f ecc: ab812f72
This is clearly the issue. They should all have been 0.
omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors ecc: 2420106 ecc: ebd922f6 ecc: 333f844f ecc: ab812f72 omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors omap-elm: uncorrectable ECC errors
Can you please share your spl/u-boot.cfg?
We have a stripped down driver "am335x_spl_bch.c" that deals with NAND at SPL. I haven't really looked much at that driver but it relies on omap_gpmc.c for
ecc.hwctl() read_buf() ecc.calculate()
We didn't do any functional change to these functions in commit 04fcd25873 unless something slipped through the cracks.
It seems to rely on following config options
CFG_SYS_NAND_ECCPOS CONFIG_SYS_NAND_PAGE_COUNT CONFIG_SYS_NAND_PAGE_SIZE CONFIG_SYS_NAND_5_ADDR_CYCLE CFG_SYS_NAND_ECCSIZE CFG_SYS_NAND_ECCBYTES CONFIG_SYS_NAND_OOBSIZE
Could you please share what they are set to for your SPL build?
Meanwhile, I'll try to reproduce this on AM335x-EVM.
Hi Roger,
I really appreciate the help!
On Thu, May 18, 2023 at 01:55:38PM +0300, Roger Quadros wrote:
Hi Colin,
On 17/05/2023 22:39, Colin Foster wrote:
I swapped in just U-Boot (not the SPL) with your patch, and everything seems to work!
The issue of Uncorrectable ECC errors spam came from the SPL. Here's a snippet of the boot log with the "ecc" print as well as your patch:
Thanks for the tests. Glad to hear issue is narrowed down to SPL.
I can "fix" the issue by just commenting out the "ECC uncorrectable errors" print :-)
U-Boot SPL 2023.04-00029-g26a9ce5314-dirty (May 17 2023 - 12:06:49 -0700) OMAP4460-GP ES1.1 Trying to boot from NAND ecc: 2420106 ecc: ebd922f6 ecc: 333f844f ecc: ab812f72
This is clearly the issue. They should all have been 0.
Interesting. With the "ecc" prints in U-Boot I also get some non-zero values:
ecc: 0 ecc: 6bff997b ecc: 6bff997b ecc: 6bff997b
Once I'm booted, I can use nanddump. It seems like everything is correct from the Linux side of things:
# nanddump -f mlo_dump /dev/mtd0 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 0 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00020000...
# nanddump -f uboot1_dump /dev/mtd1 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 0 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00180000...
# nanddump -f uboot2_dump /dev/mtd2 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 0 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00180000...
# nanddump -f /dev/null /dev/mtd3 ECC failed: 0 ECC corrected: 6 Number of bad blocks: 0 Number of bbt blocks: 0 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x1fce0000... ECC: 1 corrected bitflip(s) at offset 0x0ab30800 ECC: 1 corrected bitflip(s) at offset 0x0b008800 ECC: 1 corrected bitflip(s) at offset 0x0deaa000 ECC: 1 corrected bitflip(s) at offset 0x0ea5b000 ECC: 1 corrected bitflip(s) at offset 0x0ecbc000 ECC: 1 corrected bitflip(s) at offset 0x0ed61800
Can you please share your spl/u-boot.cfg?
Attached
We have a stripped down driver "am335x_spl_bch.c" that deals with NAND at SPL. I haven't really looked much at that driver but it relies on omap_gpmc.c for
ecc.hwctl() read_buf() ecc.calculate()
We didn't do any functional change to these functions in commit 04fcd25873 unless something slipped through the cracks.
I'll take a look at am335x_spl_bch.c and look at what I'm doing different. I was sad to see that `dump_stack()` didn't work off the bat for me.
It seems to rely on following config options
CFG_SYS_NAND_ECCPOS
{2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, \ 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, \ 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, \ 33, 34, 35, 36, 37, 38, 39, 40, 41, \ 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, \ 52, 53, 54, 55, 56, 57}
CONFIG_SYS_NAND_PAGE_COUNT
0x40
CONFIG_SYS_NAND_PAGE_SIZE
0x800
CONFIG_SYS_NAND_5_ADDR_CYCLE
1
CFG_SYS_NAND_ECCSIZE
512
CFG_SYS_NAND_ECCBYTES
14
CONFIG_SYS_NAND_OOBSIZE
0x40
Could you please share what they are set to for your SPL build?
All the CFG_* values should be identical for the SPL and U-Boot.
Meanwhile, I'll try to reproduce this on AM335x-EVM.
If you find anything let me know. I'll keep digging on my side as well.
Colin
Hi Colin,
On 19/05/2023 02:19, Colin Foster wrote:
Hi Roger,
I really appreciate the help!
On Thu, May 18, 2023 at 01:55:38PM +0300, Roger Quadros wrote:
Hi Colin,
On 17/05/2023 22:39, Colin Foster wrote:
I swapped in just U-Boot (not the SPL) with your patch, and everything seems to work!
The issue of Uncorrectable ECC errors spam came from the SPL. Here's a snippet of the boot log with the "ecc" print as well as your patch:
Thanks for the tests. Glad to hear issue is narrowed down to SPL.
I can "fix" the issue by just commenting out the "ECC uncorrectable errors" print :-)
lol
U-Boot SPL 2023.04-00029-g26a9ce5314-dirty (May 17 2023 - 12:06:49 -0700) OMAP4460-GP ES1.1 Trying to boot from NAND ecc: 2420106 ecc: ebd922f6 ecc: 333f844f ecc: ab812f72
This is clearly the issue. They should all have been 0.
Interesting. With the "ecc" prints in U-Boot I also get some non-zero values:
ecc: 0 ecc: 6bff997b ecc: 6bff997b ecc: 6bff997b
Once I'm booted, I can use nanddump. It seems like everything is correct from the Linux side of things:
# nanddump -f mlo_dump /dev/mtd0 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 0 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00020000...
# nanddump -f uboot1_dump /dev/mtd1 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 0 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00180000...
# nanddump -f uboot2_dump /dev/mtd2 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 0 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00180000...
# nanddump -f /dev/null /dev/mtd3 ECC failed: 0 ECC corrected: 6 Number of bad blocks: 0 Number of bbt blocks: 0 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x1fce0000... ECC: 1 corrected bitflip(s) at offset 0x0ab30800 ECC: 1 corrected bitflip(s) at offset 0x0b008800 ECC: 1 corrected bitflip(s) at offset 0x0deaa000 ECC: 1 corrected bitflip(s) at offset 0x0ea5b000 ECC: 1 corrected bitflip(s) at offset 0x0ecbc000 ECC: 1 corrected bitflip(s) at offset 0x0ed61800
Can you please share your spl/u-boot.cfg?
Attached
Couple of questions there
1) CONFIG_MTDPARTS_DEFAULT "mtdparts=nandflash:0x20000(xload_raw),0x180000(u-boot),0x180000(u-boot-2),0x1fce0000(main)" Is this correct and matches with what kernel sees? I couldn't see the NAND partition table in the Kernel Device tree patch.
2) #define CONFIG_SYS_NAND_U_BOOT_OFFS 0x20000 #define CONFIG_SYS_NAND_U_BOOT_OFFS_REDUND 0x1a0000
These don't seem to match what you have defined in MTDPARTS_DEFAULT. Which one is correct?
How do you flash the MLO and u-boot image to NAND?
I tried on AM335x-EVM and it works fine both before and after commit 04fcd25873.
Once change I had to do was to increase the u-boot partition size as u-boot image does not fit in original partition size.
-----boot log follows-----
U-Boot SPL 2023.01-rc4-00381-g04fcd25873-dirty (May 19 2023 - 15:10:15 +0300) Trying to boot from NAND
U-Boot 2023.01-rc4-00381-g04fcd25873-dirty (May 19 2023 - 15:10:15 +0300)
CPU : AM335X-GP rev 1.0 Model: TI AM335x EVM DRAM: 512 MiB Core: 156 devices, 17 uclasses, devicetree: separate WDT: Started wdt@44e35000 with servicing every 1000ms (60s timeout) NAND: 256 MiB MMC: OMAP SD/MMC: 0 Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1... <ethaddr> not set. Validating first E-fuse MAC Net: eth2: ethernet@4a100000, eth3: usb_ether Hit any key to stop autoboot: 0 =>
=> mtd
device nand0 <nand.0>, # parts = 10 #: name size offset mask_flags 0: NAND.SPL 0x00020000 0x00000000 0 1: NAND.SPL.backup1 0x00020000 0x00020000 0 2: NAND.SPL.backup2 0x00020000 0x00040000 0 3: NAND.SPL.backup3 0x00020000 0x00060000 0 4: NAND.u-boot-spl-os 0x00040000 0x00080000 0 5: NAND.u-boot 0x00200000 0x000c0000 0 6: NAND.u-boot-env 0x00020000 0x002c0000 0 7: NAND.u-boot-env.backup10x00020000 0x002e0000 0 8: NAND.kernel 0x00700000 0x00300000 0 9: NAND.file-system 0x0f600000 0x00a00000 0
On Fri, May 19, 2023 at 03:41:34PM +0300, Roger Quadros wrote:
Hi Colin,
On 19/05/2023 02:19, Colin Foster wrote:
Hi Roger,
Can you please share your spl/u-boot.cfg?
Attached
Couple of questions there
- CONFIG_MTDPARTS_DEFAULT "mtdparts=nandflash:0x20000(xload_raw),0x180000(u-boot),0x180000(u-boot-2),0x1fce0000(main)"
Is this correct and matches with what kernel sees? I couldn't see the NAND partition table in the Kernel Device tree patch.
Yes, this is correct. I intentionally left my MTD Partitions out of the kernel patch, since I don't want any changes I might make to the flash partitions to require further patches. I'm currently at this structure (SPL, 2x U-Boot, and main UBI with A/B partitions and 2x U-Boot Envs)
The SD Boot version of U-Boot doesn't use NAND, so it might have a stale partition layout that I'll need to remove / modify.
#define CONFIG_SYS_NAND_U_BOOT_OFFS 0x20000 #define CONFIG_SYS_NAND_U_BOOT_OFFS_REDUND 0x1a0000
These don't seem to match what you have defined in MTDPARTS_DEFAULT. Which one is correct?
This matches the above partition layout. 0x180000 + 0x20000 = 0x1a0000.
It wasn't until recently I realized I needed to remove CONFIG_SPL_RAW_IMAGE_SUPPORT in order for this fallback to succeed.
How do you flash the MLO and u-boot image to NAND?
I boot to from SD card, then run a commissioning script that contains:
``` echo "Erasing MLO partition $MLO_PART" flash_erase $MLO_PART 0 0
echo "Programming MLO partition" nandwrite -a -p $MLO_PART $MLO_FILE
echo "Erasing U-Boot partition $U_BOOT_PART" flash_erase $U_BOOT_PART 0 0
echo "Programming U-Boot partition" nandwrite -a -p $U_BOOT_PART $U_BOOT_FILE
echo "Erasing U-Boot redundant partition $U_BOOT_PART_REDUND" flash_erase $U_BOOT_PART_REDUND 0 0
echo "Programming U-Boot redund partition" nandwrite -a -p $U_BOOT_PART_REDUND $U_BOOT_FILE
echo "Clearing UBI partition" flash_erase $UBI_PART 0 0
echo "Formatting UBI partition" ubiformat $UBI_PART -y ubiattach -p $UBI_PART
echo "Making UBI volumes" ubimkvol /dev/ubi0 -N env1 -s 0x40000 ubimkvol /dev/ubi0 -N env2 -s 0x40000 ubimkvol /dev/ubi0 -N rootfs-a -s 0xc000000 ubimkvol /dev/ubi0 -N rootfs-b -s 0xc000000
echo "Writing rootfs partitions" ubiupdatevol /dev/ubi0_2 $ROOTFS_FILE ubiupdatevol /dev/ubi0_3 $ROOTFS_FILE ```
For all these tests I've been manually running the flash_erase / nandwrite process for the SPL / U-Boot partitions.
I tried on AM335x-EVM and it works fine both before and after commit 04fcd25873.
Once change I had to do was to increase the u-boot partition size as u-boot image does not fit in original partition size.
-----boot log follows-----
U-Boot SPL 2023.01-rc4-00381-g04fcd25873-dirty (May 19 2023 - 15:10:15 +0300) Trying to boot from NAND
U-Boot 2023.01-rc4-00381-g04fcd25873-dirty (May 19 2023 - 15:10:15 +0300)
CPU : AM335X-GP rev 1.0 Model: TI AM335x EVM DRAM: 512 MiB Core: 156 devices, 17 uclasses, devicetree: separate WDT: Started wdt@44e35000 with servicing every 1000ms (60s timeout) NAND: 256 MiB MMC: OMAP SD/MMC: 0 Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1... <ethaddr> not set. Validating first E-fuse MAC Net: eth2: ethernet@4a100000, eth3: usb_ether Hit any key to stop autoboot: 0 =>
=> mtd
device nand0 <nand.0>, # parts = 10 #: name size offset mask_flags 0: NAND.SPL 0x00020000 0x00000000 0 1: NAND.SPL.backup1 0x00020000 0x00020000 0 2: NAND.SPL.backup2 0x00020000 0x00040000 0 3: NAND.SPL.backup3 0x00020000 0x00060000 0
I need to go back to the 4460 datasheet. I looked and don't remember seeing anything about an SPL search. I'd sleep better at night knowing that when the day comes I need to update the SPL, I can do so with some redundancy. Sorry - I'm getting off topic.
I'll be back with hardware on Monday to keep looking at this.
4: NAND.u-boot-spl-os 0x00040000 0x00080000 0 5: NAND.u-boot 0x00200000 0x000c0000 0 6: NAND.u-boot-env 0x00020000 0x002c0000 0 7: NAND.u-boot-env.backup10x00020000 0x002e0000 0 8: NAND.kernel 0x00700000 0x00300000 0 9: NAND.file-system 0x0f600000 0x00a00000 0
-- cheers, -roger
Hi all
Il sab 20 mag 2023, 19:28 Colin Foster colin.foster@in-advantage.com ha scritto:
On Fri, May 19, 2023 at 03:41:34PM +0300, Roger Quadros wrote:
Hi Colin,
On 19/05/2023 02:19, Colin Foster wrote:
Hi Roger,
Can you please share your spl/u-boot.cfg?
Attached
Couple of questions there
- CONFIG_MTDPARTS_DEFAULT
"mtdparts=nandflash:0x20000(xload_raw),0x180000(u-boot),0x180000(u-boot-2),0x1fce0000(main)"
Is this correct and matches with what kernel sees? I couldn't see the NAND partition table in the Kernel Device tree patch.
Yes, this is correct. I intentionally left my MTD Partitions out of the kernel patch, since I don't want any changes I might make to the flash partitions to require further patches. I'm currently at this structure (SPL, 2x U-Boot, and main UBI with A/B partitions and 2x U-Boot Envs)
The SD Boot version of U-Boot doesn't use NAND, so it might have a stale partition layout that I'll need to remove / modify.
Was any end up here?
Michael
#define CONFIG_SYS_NAND_U_BOOT_OFFS 0x20000 #define CONFIG_SYS_NAND_U_BOOT_OFFS_REDUND 0x1a0000
These don't seem to match what you have defined in MTDPARTS_DEFAULT. Which one is correct?
This matches the above partition layout. 0x180000 + 0x20000 = 0x1a0000.
It wasn't until recently I realized I needed to remove CONFIG_SPL_RAW_IMAGE_SUPPORT in order for this fallback to succeed.
How do you flash the MLO and u-boot image to NAND?
I boot to from SD card, then run a commissioning script that contains:
echo "Erasing MLO partition $MLO_PART" flash_erase $MLO_PART 0 0 echo "Programming MLO partition" nandwrite -a -p $MLO_PART $MLO_FILE echo "Erasing U-Boot partition $U_BOOT_PART" flash_erase $U_BOOT_PART 0 0 echo "Programming U-Boot partition" nandwrite -a -p $U_BOOT_PART $U_BOOT_FILE echo "Erasing U-Boot redundant partition $U_BOOT_PART_REDUND" flash_erase $U_BOOT_PART_REDUND 0 0 echo "Programming U-Boot redund partition" nandwrite -a -p $U_BOOT_PART_REDUND $U_BOOT_FILE echo "Clearing UBI partition" flash_erase $UBI_PART 0 0 echo "Formatting UBI partition" ubiformat $UBI_PART -y ubiattach -p $UBI_PART echo "Making UBI volumes" ubimkvol /dev/ubi0 -N env1 -s 0x40000 ubimkvol /dev/ubi0 -N env2 -s 0x40000 ubimkvol /dev/ubi0 -N rootfs-a -s 0xc000000 ubimkvol /dev/ubi0 -N rootfs-b -s 0xc000000 echo "Writing rootfs partitions" ubiupdatevol /dev/ubi0_2 $ROOTFS_FILE ubiupdatevol /dev/ubi0_3 $ROOTFS_FILEFor all these tests I've been manually running the flash_erase / nandwrite process for the SPL / U-Boot partitions.
I tried on AM335x-EVM and it works fine both before and after commit
04fcd25873.
Once change I had to do was to increase the u-boot partition size as u-boot image does not fit in original partition size.
-----boot log follows-----
U-Boot SPL 2023.01-rc4-00381-g04fcd25873-dirty (May 19 2023 - 15:10:15
+0300)
Trying to boot from NAND
U-Boot 2023.01-rc4-00381-g04fcd25873-dirty (May 19 2023 - 15:10:15 +0300)
CPU : AM335X-GP rev 1.0 Model: TI AM335x EVM DRAM: 512 MiB Core: 156 devices, 17 uclasses, devicetree: separate WDT: Started wdt@44e35000 with servicing every 1000ms (60s timeout) NAND: 256 MiB MMC: OMAP SD/MMC: 0 Loading Environment from FAT... Unable to read "uboot.env" from
mmc0:1...
<ethaddr> not set. Validating first E-fuse MAC Net: eth2: ethernet@4a100000, eth3: usb_ether Hit any key to stop autoboot: 0 =>
=> mtd
device nand0 <nand.0>, # parts = 10 #: name size offset mask_flags 0: NAND.SPL 0x00020000 0x00000000 0 1: NAND.SPL.backup1 0x00020000 0x00020000 0 2: NAND.SPL.backup2 0x00020000 0x00040000 0 3: NAND.SPL.backup3 0x00020000 0x00060000 0
I need to go back to the 4460 datasheet. I looked and don't remember seeing anything about an SPL search. I'd sleep better at night knowing that when the day comes I need to update the SPL, I can do so with some redundancy. Sorry - I'm getting off topic.
I'll be back with hardware on Monday to keep looking at this.
4: NAND.u-boot-spl-os 0x00040000 0x00080000 0 5: NAND.u-boot 0x00200000 0x000c0000 0 6: NAND.u-boot-env 0x00020000 0x002c0000 0 7: NAND.u-boot-env.backup10x00020000 0x002e0000 0 8: NAND.kernel 0x00700000 0x00300000 0 9: NAND.file-system 0x0f600000 0x00a00000 0
-- cheers, -roger
Hi Michael,
On Mon, Jun 26, 2023 at 07:03:19AM +0200, Michael Nazzareno Trimarchi wrote:
Hi all
Il sab 20 mag 2023, 19:28 Colin Foster colin.foster@in-advantage.com ha scritto:
On Fri, May 19, 2023 at 03:41:34PM +0300, Roger Quadros wrote:
Hi Colin,
On 19/05/2023 02:19, Colin Foster wrote:
Hi Roger,
Can you please share your spl/u-boot.cfg?
Attached
Couple of questions there
- CONFIG_MTDPARTS_DEFAULT
"mtdparts=nandflash:0x20000(xload_raw),0x180000(u-boot),0x180000(u-boot-2),0x1fce0000(main)"
Is this correct and matches with what kernel sees? I couldn't see the NAND partition table in the Kernel Device tree patch.
Yes, this is correct. I intentionally left my MTD Partitions out of the kernel patch, since I don't want any changes I might make to the flash partitions to require further patches. I'm currently at this structure (SPL, 2x U-Boot, and main UBI with A/B partitions and 2x U-Boot Envs)
The SD Boot version of U-Boot doesn't use NAND, so it might have a stale partition layout that I'll need to remove / modify.
Was any end up here?
I got pulled away from this before I could figure it out, but I have an idea.
I suspect the ECC calculation is working as expected. It is just that now it is pre-calculating the ECC for regions that my flash writing was simply ignroing (just leaving erased). Previously the ECC would only be calculated for pages that were correctly written, so wouldn't cause any issues.
I do need to verify this hypothesis, but admittedly don't know how soon that'll be. (I won't be near hardware for at least a week)
Michael
participants (3)
-
Colin Foster -
Michael Nazzareno Trimarchi -
Roger Quadros