On 06/29/2018 04:44 PM, Jason Rush wrote:

On 6/29/2018 9:34 AM, Marek Vasut wrote:

...

On 06/29/2018 04:31 PM, Jason Rush wrote:

...

Dinh,

Hi,

...

A while ago, you posted the following patchset for SoCFPGA to add the PL330 DMA driver, and updated the SoCFPGA SDRAM init to write zeros to SDRAM to initialize the ECC bits if ECC was enabled:

https://lists.denx.de/pipermail/u-boot/2016-October/269643.html

I know it's been a long time, so I'll summarize some of the conversation...

At the time, you had a problem with the patchset causing the SPL to fail to find the MMC. You had tracked it down to an issue with the following commit "a78cd8613204 ARM: Rework and correct barrier definitions". You and Marek discussed it a bit, but I don't think there was a real conclusion. You submitted a second version of the patchset asking for advice on debugging the issue:

https://lists.denx.de/pipermail/u-boot/2016-December/275822.html

No real conversation came from the second patchset, and that was the end of the patch.

I was hoping we could revisit adding your patchset again. I am working on a custom SoCFPGA board with a Cyclone V and ECC SDRAM. I rebased your patchset against v2018.05 and it is working on my custom board (although I don't have an MMC). I also tested it on a SoCKit booting from an MMC (I forced it to scrub the SDRAM on the SoCKit, because it doesn't have ECC RAM), and the SoCKit finds the MMC and boots.

I don't have any suggestions on why it is working now on my board and not back when you first submitted the patchset. Maybe something else was fixed in the MMC? I was hoping you and Marek could test this patch again on some different SoCFPGA boards to see if you get the same results.

Look at this patch http://git.denx.de/?p=u-boot/u-boot-socfpga.git;a=commit;h=9bb8a249b292d26f1...

You likely want similar approach, it's faster then the DMA and much simpler.

Thanks Marek.  I'll give it a try.  Would you be interested in a similar patch for the Gen 5?

I don't have any Gen5 board which uses ECC, do you ? If so, yes, prepare a patch, it should be very similar.

Make sure to measure how long it takes to scrub the memory and how much memory you have, I'd be interested in the numbers.

On 06/29/2018 05:06 PM, Jason Rush wrote:

On 6/29/2018 9:52 AM, Marek Vasut wrote:

...

On 06/29/2018 04:44 PM, Jason Rush wrote:

...

On 6/29/2018 9:34 AM, Marek Vasut wrote:

...

On 06/29/2018 04:31 PM, Jason Rush wrote:

...

Dinh,

Hi,

...

A while ago, you posted the following patchset for SoCFPGA to add the PL330 DMA driver, and updated the SoCFPGA SDRAM init to write zeros to SDRAM to initialize the ECC bits if ECC was enabled:

https://lists.denx.de/pipermail/u-boot/2016-October/269643.html

I know it's been a long time, so I'll summarize some of the conversation...

At the time, you had a problem with the patchset causing the SPL to fail to find the MMC. You had tracked it down to an issue with the following commit "a78cd8613204 ARM: Rework and correct barrier definitions". You and Marek discussed it a bit, but I don't think there was a real conclusion. You submitted a second version of the patchset asking for advice on debugging the issue:

https://lists.denx.de/pipermail/u-boot/2016-December/275822.html

No real conversation came from the second patchset, and that was the end of the patch.

I was hoping we could revisit adding your patchset again. I am working on a custom SoCFPGA board with a Cyclone V and ECC SDRAM. I rebased your patchset against v2018.05 and it is working on my custom board (although I don't have an MMC). I also tested it on a SoCKit booting from an MMC (I forced it to scrub the SDRAM on the SoCKit, because it doesn't have ECC RAM), and the SoCKit finds the MMC and boots.

I don't have any suggestions on why it is working now on my board and not back when you first submitted the patchset. Maybe something else was fixed in the MMC? I was hoping you and Marek could test this patch again on some different SoCFPGA boards to see if you get the same results.

Look at this patch http://git.denx.de/?p=u-boot/u-boot-socfpga.git;a=commit;h=9bb8a249b292d26f1...

You likely want similar approach, it's faster then the DMA and much simpler.

Thanks Marek.  I'll give it a try.  Would you be interested in a similar patch for the Gen 5?

I don't have any Gen5 board which uses ECC, do you ? If so, yes, prepare a patch, it should be very similar.

Make sure to measure how long it takes to scrub the memory and how much memory you have, I'd be interested in the numbers.

Looking at the master branch, it doesn't look like that code is ever being called? The sdram_init_ecc_bits() function is called from the ddr_calibration_sequence function(), but I can't find where ddr_calibration_sequence is called().

git grep for it, it's called from somewhere in the arch/arm/mach-socfpga/

Either way, I can test it. I have a custom Cyclone V board with ECC, and the Intel Arria V SoC Dev Kit I can test it on too which I think has ECC.

Please do.

On 07/03/2018 03:58 PM, Jason Rush wrote:

On 6/29/2018 10:17 AM, Marek Vasut wrote:

...

On 06/29/2018 05:06 PM, Jason Rush wrote:

...

On 6/29/2018 9:52 AM, Marek Vasut wrote:

...

On 06/29/2018 04:44 PM, Jason Rush wrote:

...

On 6/29/2018 9:34 AM, Marek Vasut wrote:

...

On 06/29/2018 04:31 PM, Jason Rush wrote: > Dinh, Hi,

> A while ago, you posted the following patchset for SoCFPGA to add the PL330 > DMA driver, and updated the SoCFPGA SDRAM init to write zeros to SDRAM to > initialize the ECC bits if ECC was enabled: > > https://lists.denx.de/pipermail/u-boot/2016-October/269643.html > > I know it's been a long time, so I'll summarize some of the conversation... > > At the time, you had a problem with the patchset causing the SPL to fail to > find the MMC. You had tracked it down to an issue with the following commit > "a78cd8613204 ARM: Rework and correct barrier definitions". You and Marek > discussed it a bit, but I don't think there was a real conclusion. You > submitted a second version of the patchset asking for advice on debugging > the issue: > > https://lists.denx.de/pipermail/u-boot/2016-December/275822.html > > No real conversation came from the second patchset, and that was the end of > the patch. > > I was hoping we could revisit adding your patchset again. I am working on a > custom SoCFPGA board with a Cyclone V and ECC SDRAM. I rebased your patchset > against v2018.05 and it is working on my custom board (although I don't have > an MMC). I also tested it on a SoCKit booting from an MMC (I forced it to > scrub the SDRAM on the SoCKit, because it doesn't have ECC RAM), and the > SoCKit finds the MMC and boots. > > I don't have any suggestions on why it is working now on my board and not > back when you first submitted the patchset. Maybe something else was fixed > in the MMC? I was hoping you and Marek could test this patch again on some > different SoCFPGA boards to see if you get the same results. Look at this patch http://git.denx.de/?p=u-boot/u-boot-socfpga.git;a=commit;h=9bb8a249b292d26f1...

You likely want similar approach, it's faster then the DMA and much simpler.

Thanks Marek.  I'll give it a try.  Would you be interested in a similar patch for the Gen 5?

I don't have any Gen5 board which uses ECC, do you ? If so, yes, prepare a patch, it should be very similar.

Make sure to measure how long it takes to scrub the memory and how much memory you have, I'd be interested in the numbers.

Looking at the master branch, it doesn't look like that code is ever being called? The sdram_init_ecc_bits() function is called from the ddr_calibration_sequence function(), but I can't find where ddr_calibration_sequence is called().

git grep for it, it's called from somewhere in the arch/arm/mach-socfpga/

...

Either way, I can test it. I have a custom Cyclone V board with ECC, and the Intel Arria V SoC Dev Kit I can test it on too which I think has ECC.

Please do.

I implemented a similar memset approach for the gen 5 socfpga.  It's basically the same code as in that patch; however, when I performed a single memset the processor would reset for some reason.  I changed it to loop over calling memset with a size of 32MB over the entire address the address, and that worked as opposed to doing a single memset on the RAM.

Can you do grep MEMSET .config in your U-Boot build dir ? The arch memset is implemented in assembler and doesn't trigger WDT , so if it takes too long, it could be that the WDT resets the platform.

I started on a SoCKit because it was handy, I know it doesn't have ECC

It doesn't by default.

, but I forced it to initialize the RAM as a quick test.  It seems much slower than the DMA approach.  It should be noted, I didn't implement any code to time the scrubbing, but rather just roughly monitored the time to get a rough idea of how long it took.

On the SoCKit, which has 1GB of RAM, the memset takes around 8 seconds to complete, and the DMA takes under 2 seconds.

Did you enable i/d cache in the SPL ? It's mandatory, otherwise it's slow. Just be careful about the MMU tables placement, they are big and if you place them in RAM, make sure you don't overwrite them with the memset. The trick might be to memset the first 1 MiB of RAM, then put MMU tables at some offset therein (since 0x0 can be used for ARM vectors) and then turn on i/d cache and memset the rest.

Maybe I ported something wrong?  I used very similar code as the sdram_init_ecc_bits() function, with the exception I looped over the second memset with a size of 32MB.  I wasn't sure why the TLB address/size was being initialized, or if the values used on the Arria 10 are even correct for the Cyclone 5,

Because I needed to place MMU tables somewhere. The memory layout on CV/A10 are the same, RAM starts at 0, except the vectors can also be placed at 0, so the MMU tables should be at some offset.

but I used that part of the code as is from the Arria 10 patch as well as enabling the icache and dcache just like the Arria 10 code.

Any suggestions on what to look at?

Jason

On 07/04/2018 01:45 AM, Jason Rush wrote:

On 7/3/2018 9:08 AM, Marek Vasut wrote:

...

On 07/03/2018 03:58 PM, Jason Rush wrote:

...

On 6/29/2018 10:17 AM, Marek Vasut wrote:

...

On 06/29/2018 05:06 PM, Jason Rush wrote:

...

On 6/29/2018 9:52 AM, Marek Vasut wrote:

...

On 06/29/2018 04:44 PM, Jason Rush wrote: > On 6/29/2018 9:34 AM, Marek Vasut wrote: >> On 06/29/2018 04:31 PM, Jason Rush wrote: >>> Dinh, >> Hi, >> >>> A while ago, you posted the following patchset for SoCFPGA to add the PL330 >>> DMA driver, and updated the SoCFPGA SDRAM init to write zeros to SDRAM to >>> initialize the ECC bits if ECC was enabled: >>> >>> https://lists.denx.de/pipermail/u-boot/2016-October/269643.html >>> >>> I know it's been a long time, so I'll summarize some of the conversation... >>> >>> At the time, you had a problem with the patchset causing the SPL to fail to >>> find the MMC. You had tracked it down to an issue with the following commit >>> "a78cd8613204 ARM: Rework and correct barrier definitions". You and Marek >>> discussed it a bit, but I don't think there was a real conclusion. You >>> submitted a second version of the patchset asking for advice on debugging >>> the issue: >>> >>> https://lists.denx.de/pipermail/u-boot/2016-December/275822.html >>> >>> No real conversation came from the second patchset, and that was the end of >>> the patch. >>> >>> I was hoping we could revisit adding your patchset again. I am working on a >>> custom SoCFPGA board with a Cyclone V and ECC SDRAM. I rebased your patchset >>> against v2018.05 and it is working on my custom board (although I don't have >>> an MMC). I also tested it on a SoCKit booting from an MMC (I forced it to >>> scrub the SDRAM on the SoCKit, because it doesn't have ECC RAM), and the >>> SoCKit finds the MMC and boots. >>> >>> I don't have any suggestions on why it is working now on my board and not >>> back when you first submitted the patchset. Maybe something else was fixed >>> in the MMC? I was hoping you and Marek could test this patch again on some >>> different SoCFPGA boards to see if you get the same results. >> Look at this patch >> http://git.denx.de/?p=u-boot/u-boot-socfpga.git;a=commit;h=9bb8a249b292d26f1... >> >> You likely want similar approach, it's faster then the DMA and much simpler. >> > Thanks Marek.  I'll give it a try.  Would you be interested in a similar patch for the Gen 5? I don't have any Gen5 board which uses ECC, do you ? If so, yes, prepare a patch, it should be very similar.

Make sure to measure how long it takes to scrub the memory and how much memory you have, I'd be interested in the numbers.

Looking at the master branch, it doesn't look like that code is ever being called? The sdram_init_ecc_bits() function is called from the ddr_calibration_sequence function(), but I can't find where ddr_calibration_sequence is called().

git grep for it, it's called from somewhere in the arch/arm/mach-socfpga/

...

Either way, I can test it. I have a custom Cyclone V board with ECC, and the Intel Arria V SoC Dev Kit I can test it on too which I think has ECC.

Please do.

I implemented a similar memset approach for the gen 5 socfpga.  It's basically the same code as in that patch; however, when I performed a single memset the processor would reset for some reason.  I changed it to loop over calling memset with a size of 32MB over the entire address the address, and that worked as opposed to doing a single memset on the RAM.

Can you do grep MEMSET .config in your U-Boot build dir ? The arch memset is implemented in assembler and doesn't trigger WDT , so if it takes too long, it could be that the WDT resets the platform.

Both CONFIG_USE_ARCH_MEMSET and CONFIG_SPL_USE_ARCH_MEMSET are set in my .config, so it must be the WDT triggering as you suspect.

...

...

I started on a SoCKit because it was handy, I know it doesn't have ECC

It doesn't by default.

...

, but I forced it to initialize the RAM as a quick test.  It seems much slower than the DMA approach.  It should be noted, I didn't implement any code to time the scrubbing, but rather just roughly monitored the time to get a rough idea of how long it took.

On the SoCKit, which has 1GB of RAM, the memset takes around 8 seconds to complete, and the DMA takes under 2 seconds.

Did you enable i/d cache in the SPL ? It's mandatory, otherwise it's slow.

I have calls to icache_enable() and dcache_enable() just as you do in the Arria 10 sdram_init_ecc_bits() function.

I did double check that both these enable functions call the versions of the functions in the ./arch/arm/lib/cache-cp15.c file that are implemented in the SPL.  So I believe that both icache and dcache is enabled.

Are you sure it's not just the stubs that are called ? Or that the code doesn't skip the dcache enabling due to some funny stuff, like MMU being already enabled ?

I probably should have added a print of icache_status() and dcache_status() to verify the caches are enabled.  I'll add that tomorrow.

Yes, you really should verify that the dcache was enabled.

...

Just be careful about the MMU tables placement, they are big and if you place them in RAM, make sure you don't overwrite them with the memset. The trick might be to memset the first 1 MiB of RAM, then put MMU tables at some offset therein (since 0x0 can be used for ARM vectors) and then turn on i/d cache and memset the rest.

That is essentially what I am doing I believe, with the exception that I am only clearing the first 32KiB before initializing the MMU table (which is what you did in the Arria 10 version).

I modeled my code almost identically to yours with the exception that I loop over the memset calls 32MiB at a time. Here's the order of operations I perform:

1. icache_enable()
2. memset the first 0x8000 bytes to zero
3. setup gd->arch.tlb_arch and gd->arch.tlb_size
4. dcache_enable()
5. loop over remaining memory, memsetting 32MiB at a time to zero
6. flush_dcache_all()
7. dcache_disable()

It looks like the call to dcache_enable is what sets up the MMU tables. I suspect that's why you did a memset of the first 32KiB before enabling the dcache on the Arria 10.  I think the MMU is initialized okay since the SPL keeps executing, u-boot loads, and Linux boots after running the above (maybe that's not a fair assumption).

I had to write zeroes to the first 32kiB to init the ECC counters before putting MMU tables there.

You really should double check if the MMU and dcache are enabled, 8 seconds to scrub the memory is too long I think.

...

...

Maybe I ported something wrong?  I used very similar code as the sdram_init_ecc_bits() function, with the exception I looped over the second memset with a size of 32MB.  I wasn't sure why the TLB address/size was being initialized, or if the values used on the Arria 10 are even correct for the Cyclone 5,

Because I needed to place MMU tables somewhere. The memory layout on CV/A10 are the same, RAM starts at 0, except the vectors can also be placed at 0, so the MMU tables should be at some offset.

...

but I used that part of the code as is from the Arria 10 patch as well as enabling the icache and dcache just like the Arria 10 code.

Any suggestions on what to look at?

Jason

Thank you for your help.

Regards, Jason

On 07/06/2018 01:11 AM, Jason Rush wrote:

On 7/4/2018 2:23 AM, Marek Vasut wrote:

...

On 07/04/2018 01:45 AM, Jason Rush wrote:

...

On 7/3/2018 9:08 AM, Marek Vasut wrote:

...

On 07/03/2018 03:58 PM, Jason Rush wrote:

...

On 6/29/2018 10:17 AM, Marek Vasut wrote:

...

On 06/29/2018 05:06 PM, Jason Rush wrote: > On 6/29/2018 9:52 AM, Marek Vasut wrote: >> On 06/29/2018 04:44 PM, Jason Rush wrote: >>> On 6/29/2018 9:34 AM, Marek Vasut wrote: >>>> On 06/29/2018 04:31 PM, Jason Rush wrote: >>>>> Dinh, >>>> Hi, >>>> >>>>> A while ago, you posted the following patchset for SoCFPGA to add the PL330 >>>>> DMA driver, and updated the SoCFPGA SDRAM init to write zeros to SDRAM to >>>>> initialize the ECC bits if ECC was enabled: >>>>> >>>>> https://lists.denx.de/pipermail/u-boot/2016-October/269643.html >>>>> >>>>> I know it's been a long time, so I'll summarize some of the conversation... >>>>> >>>>> At the time, you had a problem with the patchset causing the SPL to fail to >>>>> find the MMC. You had tracked it down to an issue with the following commit >>>>> "a78cd8613204 ARM: Rework and correct barrier definitions". You and Marek >>>>> discussed it a bit, but I don't think there was a real conclusion. You >>>>> submitted a second version of the patchset asking for advice on debugging >>>>> the issue: >>>>> >>>>> https://lists.denx.de/pipermail/u-boot/2016-December/275822.html >>>>> >>>>> No real conversation came from the second patchset, and that was the end of >>>>> the patch. >>>>> >>>>> I was hoping we could revisit adding your patchset again. I am working on a >>>>> custom SoCFPGA board with a Cyclone V and ECC SDRAM. I rebased your patchset >>>>> against v2018.05 and it is working on my custom board (although I don't have >>>>> an MMC). I also tested it on a SoCKit booting from an MMC (I forced it to >>>>> scrub the SDRAM on the SoCKit, because it doesn't have ECC RAM), and the >>>>> SoCKit finds the MMC and boots. >>>>> >>>>> I don't have any suggestions on why it is working now on my board and not >>>>> back when you first submitted the patchset. Maybe something else was fixed >>>>> in the MMC? I was hoping you and Marek could test this patch again on some >>>>> different SoCFPGA boards to see if you get the same results. >>>> Look at this patch >>>> http://git.denx.de/?p=u-boot/u-boot-socfpga.git;a=commit;h=9bb8a249b292d26f1... >>>> >>>> You likely want similar approach, it's faster then the DMA and much simpler. >>>> >>> Thanks Marek.  I'll give it a try.  Would you be interested in a similar patch for the Gen 5? >> I don't have any Gen5 board which uses ECC, do you ? >> If so, yes, prepare a patch, it should be very similar. >> >> Make sure to measure how long it takes to scrub the memory and how much >> memory you have, I'd be interested in the numbers. >> > Looking at the master branch, it doesn't look like that code is ever being called? > The sdram_init_ecc_bits() function is called from the ddr_calibration_sequence function(), > but I can't find where ddr_calibration_sequence is called(). git grep for it, it's called from somewhere in the arch/arm/mach-socfpga/

> Either way, I can test it. I have a custom Cyclone V board with ECC, and the Intel Arria V SoC > Dev Kit I can test it on too which I think has ECC. Please do.

I implemented a similar memset approach for the gen 5 socfpga.  It's basically the same code as in that patch; however, when I performed a single memset the processor would reset for some reason.  I changed it to loop over calling memset with a size of 32MB over the entire address the address, and that worked as opposed to doing a single memset on the RAM.

Can you do grep MEMSET .config in your U-Boot build dir ? The arch memset is implemented in assembler and doesn't trigger WDT , so if it takes too long, it could be that the WDT resets the platform.

Both CONFIG_USE_ARCH_MEMSET and CONFIG_SPL_USE_ARCH_MEMSET are set in my .config, so it must be the WDT triggering as you suspect.

...

...

I started on a SoCKit because it was handy, I know it doesn't have ECC

It doesn't by default.

...

, but I forced it to initialize the RAM as a quick test.  It seems much slower than the DMA approach.  It should be noted, I didn't implement any code to time the scrubbing, but rather just roughly monitored the time to get a rough idea of how long it took.

On the SoCKit, which has 1GB of RAM, the memset takes around 8 seconds to complete, and the DMA takes under 2 seconds.

Did you enable i/d cache in the SPL ? It's mandatory, otherwise it's slow.

I have calls to icache_enable() and dcache_enable() just as you do in the Arria 10 sdram_init_ecc_bits() function.

I did double check that both these enable functions call the versions of the functions in the ./arch/arm/lib/cache-cp15.c file that are implemented in the SPL.  So I believe that both icache and dcache is enabled.

Are you sure it's not just the stubs that are called ? Or that the code doesn't skip the dcache enabling due to some funny stuff, like MMU being already enabled ?

I added prints to ensure it is calling the real icache_enable()/dcache_enable() functions, and not the stubs.

...

...

I probably should have added a print of icache_status() and dcache_status() to verify the caches are enabled.  I'll add that tomorrow.

Yes, you really should verify that the dcache was enabled.

...

...

Just be careful about the MMU tables placement, they are big and if you place them in RAM, make sure you don't overwrite them with the memset. The trick might be to memset the first 1 MiB of RAM, then put MMU tables at some offset therein (since 0x0 can be used for ARM vectors) and then turn on i/d cache and memset the rest.

That is essentially what I am doing I believe, with the exception that I am only clearing the first 32KiB before initializing the MMU table (which is what you did in the Arria 10 version).

I modeled my code almost identically to yours with the exception that I loop over the memset calls 32MiB at a time. Here's the order of operations I perform:

1. icache_enable()
2. memset the first 0x8000 bytes to zero
3. setup gd->arch.tlb_arch and gd->arch.tlb_size
4. dcache_enable()
5. loop over remaining memory, memsetting 32MiB at a time to zero
6. flush_dcache_all()
7. dcache_disable()

It looks like the call to dcache_enable is what sets up the MMU tables. I suspect that's why you did a memset of the first 32KiB before enabling the dcache on the Arria 10.  I think the MMU is initialized okay since the SPL keeps executing, u-boot loads, and Linux boots after running the above (maybe that's not a fair assumption).

I had to write zeroes to the first 32kiB to init the ECC counters before putting MMU tables there.

You really should double check if the MMU and dcache are enabled, 8 seconds to scrub the memory is too long I think.

I added checks to verify that the MMU, icache, and dcache are all setup and enabled.

Calling icache_enable() set the CR_I bit (Icache enable) in the CR (control register).  Then calling dcache_enable() called the mmu_setup() function, which setup the MMU and set the CR_M bit (MMU enable) in the CR, and finally dcache_enable() set the CR_C bit (Dcache enable) bit in the CR.

I also printed out the control register before the memset calls, and it indicated that the mmu, icache, and dcache were enabled.

Is the DRAM area set as cacheable in the MMU tables ?

On 7/5/2018 6:10 PM, Marek Vasut wrote:

On 07/06/2018 01:11 AM, Jason Rush wrote:

...

On 7/4/2018 2:23 AM, Marek Vasut wrote:

...

On 07/04/2018 01:45 AM, Jason Rush wrote:

...

On 7/3/2018 9:08 AM, Marek Vasut wrote:

...

On 07/03/2018 03:58 PM, Jason Rush wrote:

...

On 6/29/2018 10:17 AM, Marek Vasut wrote: > On 06/29/2018 05:06 PM, Jason Rush wrote: >> On 6/29/2018 9:52 AM, Marek Vasut wrote: >>> On 06/29/2018 04:44 PM, Jason Rush wrote: >>>> On 6/29/2018 9:34 AM, Marek Vasut wrote: >>>>> On 06/29/2018 04:31 PM, Jason Rush wrote: >>>>>> Dinh, >>>>> Hi, >>>>> >>>>>> A while ago, you posted the following patchset for SoCFPGA to add the PL330 >>>>>> DMA driver, and updated the SoCFPGA SDRAM init to write zeros to SDRAM to >>>>>> initialize the ECC bits if ECC was enabled: >>>>>> >>>>>> https://lists.denx.de/pipermail/u-boot/2016-October/269643.html >>>>>> >>>>>> I know it's been a long time, so I'll summarize some of the conversation... >>>>>> >>>>>> At the time, you had a problem with the patchset causing the SPL to fail to >>>>>> find the MMC. You had tracked it down to an issue with the following commit >>>>>> "a78cd8613204 ARM: Rework and correct barrier definitions". You and Marek >>>>>> discussed it a bit, but I don't think there was a real conclusion. You >>>>>> submitted a second version of the patchset asking for advice on debugging >>>>>> the issue: >>>>>> >>>>>> https://lists.denx.de/pipermail/u-boot/2016-December/275822.html >>>>>> >>>>>> No real conversation came from the second patchset, and that was the end of >>>>>> the patch. >>>>>> >>>>>> I was hoping we could revisit adding your patchset again. I am working on a >>>>>> custom SoCFPGA board with a Cyclone V and ECC SDRAM. I rebased your patchset >>>>>> against v2018.05 and it is working on my custom board (although I don't have >>>>>> an MMC). I also tested it on a SoCKit booting from an MMC (I forced it to >>>>>> scrub the SDRAM on the SoCKit, because it doesn't have ECC RAM), and the >>>>>> SoCKit finds the MMC and boots. >>>>>> >>>>>> I don't have any suggestions on why it is working now on my board and not >>>>>> back when you first submitted the patchset. Maybe something else was fixed >>>>>> in the MMC? I was hoping you and Marek could test this patch again on some >>>>>> different SoCFPGA boards to see if you get the same results. >>>>> Look at this patch >>>>> http://git.denx.de/?p=u-boot/u-boot-socfpga.git;a=commit;h=9bb8a249b292d26f1... >>>>> >>>>> You likely want similar approach, it's faster then the DMA and much simpler. >>>>> >>>> Thanks Marek.  I'll give it a try.  Would you be interested in a similar patch for the Gen 5? >>> I don't have any Gen5 board which uses ECC, do you ? >>> If so, yes, prepare a patch, it should be very similar. >>> >>> Make sure to measure how long it takes to scrub the memory and how much >>> memory you have, I'd be interested in the numbers. >>> >> Looking at the master branch, it doesn't look like that code is ever being called? >> The sdram_init_ecc_bits() function is called from the ddr_calibration_sequence function(), >> but I can't find where ddr_calibration_sequence is called(). > git grep for it, it's called from somewhere in the arch/arm/mach-socfpga/ > >> Either way, I can test it. I have a custom Cyclone V board with ECC, and the Intel Arria V SoC >> Dev Kit I can test it on too which I think has ECC. > Please do. > I implemented a similar memset approach for the gen 5 socfpga.  It's basically the same code as in that patch; however, when I performed a single memset the processor would reset for some reason.  I changed it to loop over calling memset with a size of 32MB over the entire address the address, and that worked as opposed to doing a single memset on the RAM.

Can you do grep MEMSET .config in your U-Boot build dir ? The arch memset is implemented in assembler and doesn't trigger WDT , so if it takes too long, it could be that the WDT resets the platform.

Both CONFIG_USE_ARCH_MEMSET and CONFIG_SPL_USE_ARCH_MEMSET are set in my .config, so it must be the WDT triggering as you suspect.

...

...

I started on a SoCKit because it was handy, I know it doesn't have ECC

It doesn't by default.

...

, but I forced it to initialize the RAM as a quick test.  It seems much slower than the DMA approach.  It should be noted, I didn't implement any code to time the scrubbing, but rather just roughly monitored the time to get a rough idea of how long it took.

On the SoCKit, which has 1GB of RAM, the memset takes around 8 seconds to complete, and the DMA takes under 2 seconds.

Did you enable i/d cache in the SPL ? It's mandatory, otherwise it's slow.

I have calls to icache_enable() and dcache_enable() just as you do in the Arria 10 sdram_init_ecc_bits() function.

I did double check that both these enable functions call the versions of the functions in the ./arch/arm/lib/cache-cp15.c file that are implemented in the SPL.  So I believe that both icache and dcache is enabled.

Are you sure it's not just the stubs that are called ? Or that the code doesn't skip the dcache enabling due to some funny stuff, like MMU being already enabled ?

I added prints to ensure it is calling the real icache_enable()/dcache_enable() functions, and not the stubs.

...

...

I probably should have added a print of icache_status() and dcache_status() to verify the caches are enabled.  I'll add that tomorrow.

Yes, you really should verify that the dcache was enabled.

...

...

Just be careful about the MMU tables placement, they are big and if you place them in RAM, make sure you don't overwrite them with the memset. The trick might be to memset the first 1 MiB of RAM, then put MMU tables at some offset therein (since 0x0 can be used for ARM vectors) and then turn on i/d cache and memset the rest.

That is essentially what I am doing I believe, with the exception that I am only clearing the first 32KiB before initializing the MMU table (which is what you did in the Arria 10 version).

I modeled my code almost identically to yours with the exception that I loop over the memset calls 32MiB at a time. Here's the order of operations I perform:

1. icache_enable()
2. memset the first 0x8000 bytes to zero
3. setup gd->arch.tlb_arch and gd->arch.tlb_size
4. dcache_enable()
5. loop over remaining memory, memsetting 32MiB at a time to zero
6. flush_dcache_all()
7. dcache_disable()

It looks like the call to dcache_enable is what sets up the MMU tables. I suspect that's why you did a memset of the first 32KiB before enabling the dcache on the Arria 10.  I think the MMU is initialized okay since the SPL keeps executing, u-boot loads, and Linux boots after running the above (maybe that's not a fair assumption).

I had to write zeroes to the first 32kiB to init the ECC counters before putting MMU tables there.

You really should double check if the MMU and dcache are enabled, 8 seconds to scrub the memory is too long I think.

I added checks to verify that the MMU, icache, and dcache are all setup and enabled.

Calling icache_enable() set the CR_I bit (Icache enable) in the CR (control register).  Then calling dcache_enable() called the mmu_setup() function, which setup the MMU and set the CR_M bit (MMU enable) in the CR, and finally dcache_enable() set the CR_C bit (Dcache enable) bit in the CR.

I also printed out the control register before the memset calls, and it indicated that the mmu, icache, and dcache were enabled.

Is the DRAM area set as cacheable in the MMU tables ?

Good news bad news...  The MMU tables weren't being set up because the bd->bi_dram[bank].start and bd->bi_dram[bank].size weren't set up.  As a quick test, I hardcoded start to 0 and size to 1GiB.  After that, the memset was really quick, U-Boot loads, Linux loads, and everything seems to work great.

However, if I press the HPS_RST push button on the SoCKit (which is connected to power on reset), occasionally U-Boot will lock up while booting.  It always boots and operates correctly from the initial power on, but it almost always fails to boot after pressing the HPS_RST button.

Usually after pressing the HPS_RST button, U-Boot makes it past the SPL, and hangs somewhere after the call to setup_reloc() in ./common/board_f.c.  Once it hangs there, pressing the HPS_RST button again usually causes the SPL to hang while setting up the MMU (before my call to memset).  Eventually the WDT kicks in, and it just keeps hanging up in the same place.  Once it gets in this mode, the only way to recover it is by toggling power on the board.

I spent a bunch of time today trying to track down where it was hanging, but I couldn't pin point anything.  The MMU tables looked correct.  The MMU registers looked good.  I'm not sure the best way to debug what's going on.

Any thoughts?

Regards, Jason

On 7/9/2018 3:08 AM, Marek Vasut wrote:

On 07/07/2018 12:56 AM, Jason Rush wrote:

...

On 7/5/2018 6:10 PM, Marek Vasut wrote:

...

On 07/06/2018 01:11 AM, Jason Rush wrote:

...

On 7/4/2018 2:23 AM, Marek Vasut wrote:

...

On 07/04/2018 01:45 AM, Jason Rush wrote:

...

On 7/3/2018 9:08 AM, Marek Vasut wrote: > On 07/03/2018 03:58 PM, Jason Rush wrote: >> On 6/29/2018 10:17 AM, Marek Vasut wrote: >>> On 06/29/2018 05:06 PM, Jason Rush wrote: >>>> On 6/29/2018 9:52 AM, Marek Vasut wrote: >>>>> On 06/29/2018 04:44 PM, Jason Rush wrote: >>>>>> On 6/29/2018 9:34 AM, Marek Vasut wrote: >>>>>>> On 06/29/2018 04:31 PM, Jason Rush wrote: >>>>>>>> Dinh, >>>>>>> Hi, >>>>>>> >>>>>>>> A while ago, you posted the following patchset for SoCFPGA to add the PL330 >>>>>>>> DMA driver, and updated the SoCFPGA SDRAM init to write zeros to SDRAM to >>>>>>>> initialize the ECC bits if ECC was enabled: >>>>>>>> >>>>>>>> https://lists.denx.de/pipermail/u-boot/2016-October/269643.html >>>>>>>> >>>>>>>> I know it's been a long time, so I'll summarize some of the conversation... >>>>>>>> >>>>>>>> At the time, you had a problem with the patchset causing the SPL to fail to >>>>>>>> find the MMC. You had tracked it down to an issue with the following commit >>>>>>>> "a78cd8613204 ARM: Rework and correct barrier definitions". You and Marek >>>>>>>> discussed it a bit, but I don't think there was a real conclusion. You >>>>>>>> submitted a second version of the patchset asking for advice on debugging >>>>>>>> the issue: >>>>>>>> >>>>>>>> https://lists.denx.de/pipermail/u-boot/2016-December/275822.html >>>>>>>> >>>>>>>> No real conversation came from the second patchset, and that was the end of >>>>>>>> the patch. >>>>>>>> >>>>>>>> I was hoping we could revisit adding your patchset again. I am working on a >>>>>>>> custom SoCFPGA board with a Cyclone V and ECC SDRAM. I rebased your patchset >>>>>>>> against v2018.05 and it is working on my custom board (although I don't have >>>>>>>> an MMC). I also tested it on a SoCKit booting from an MMC (I forced it to >>>>>>>> scrub the SDRAM on the SoCKit, because it doesn't have ECC RAM), and the >>>>>>>> SoCKit finds the MMC and boots. >>>>>>>> >>>>>>>> I don't have any suggestions on why it is working now on my board and not >>>>>>>> back when you first submitted the patchset. Maybe something else was fixed >>>>>>>> in the MMC? I was hoping you and Marek could test this patch again on some >>>>>>>> different SoCFPGA boards to see if you get the same results. >>>>>>> Look at this patch >>>>>>> http://git.denx.de/?p=u-boot/u-boot-socfpga.git;a=commit;h=9bb8a249b292d26f1... >>>>>>> >>>>>>> You likely want similar approach, it's faster then the DMA and much simpler. >>>>>>> >>>>>> Thanks Marek.  I'll give it a try.  Would you be interested in a similar patch for the Gen 5? >>>>> I don't have any Gen5 board which uses ECC, do you ? >>>>> If so, yes, prepare a patch, it should be very similar. >>>>> >>>>> Make sure to measure how long it takes to scrub the memory and how much >>>>> memory you have, I'd be interested in the numbers. >>>>> >>>> Looking at the master branch, it doesn't look like that code is ever being called? >>>> The sdram_init_ecc_bits() function is called from the ddr_calibration_sequence function(), >>>> but I can't find where ddr_calibration_sequence is called(). >>> git grep for it, it's called from somewhere in the arch/arm/mach-socfpga/ >>> >>>> Either way, I can test it. I have a custom Cyclone V board with ECC, and the Intel Arria V SoC >>>> Dev Kit I can test it on too which I think has ECC. >>> Please do. >>> >> I implemented a similar memset approach for the gen 5 socfpga.  It's basically the same >> code as in that patch; however, when I performed a single memset the processor would >> reset for some reason.  I changed it to loop over calling memset with a size of 32MB over >> the entire address the address, and that worked as opposed to doing a single memset on >> the RAM. > Can you do grep MEMSET .config in your U-Boot build dir ? The arch > memset is implemented in assembler and doesn't trigger WDT , so if it > takes too long, it could be that the WDT resets the platform. Both CONFIG_USE_ARCH_MEMSET and CONFIG_SPL_USE_ARCH_MEMSET are set in my .config, so it must be the WDT triggering as you suspect.

>> I started on a SoCKit because it was handy, I know it doesn't have ECC > It doesn't by default. > >> , but I forced it to >> initialize the RAM as a quick test.  It seems much slower than the DMA approach.  It >> should be noted, I didn't implement any code to time the scrubbing, but rather just >> roughly monitored the time to get a rough idea of how long it took. >> >> On the SoCKit, which has 1GB of RAM, the memset takes around 8 seconds to complete, >> and the DMA takes under 2 seconds. > Did you enable i/d cache in the SPL ? It's mandatory, otherwise it's > slow. I have calls to icache_enable() and dcache_enable() just as you do in the Arria 10 sdram_init_ecc_bits() function.

I did double check that both these enable functions call the versions of the functions in the ./arch/arm/lib/cache-cp15.c file that are implemented in the SPL.  So I believe that both icache and dcache is enabled.

Are you sure it's not just the stubs that are called ? Or that the code doesn't skip the dcache enabling due to some funny stuff, like MMU being already enabled ?

I added prints to ensure it is calling the real icache_enable()/dcache_enable() functions, and not the stubs.

...

...

I probably should have added a print of icache_status() and dcache_status() to verify the caches are enabled.  I'll add that tomorrow.

Yes, you really should verify that the dcache was enabled.

...

> Just be careful about the MMU tables placement, they are big and > if you place them in RAM, make sure you don't overwrite them with the > memset. The trick might be to memset the first 1 MiB of RAM, then put > MMU tables at some offset therein (since 0x0 can be used for ARM > vectors) and then turn on i/d cache and memset the rest. That is essentially what I am doing I believe, with the exception that I am only clearing the first 32KiB before initializing the MMU table (which is what you did in the Arria 10 version).

I modeled my code almost identically to yours with the exception that I loop over the memset calls 32MiB at a time. Here's the order of operations I perform:

1. icache_enable()
2. memset the first 0x8000 bytes to zero
3. setup gd->arch.tlb_arch and gd->arch.tlb_size
4. dcache_enable()
5. loop over remaining memory, memsetting 32MiB at a time to zero
6. flush_dcache_all()
7. dcache_disable()

It looks like the call to dcache_enable is what sets up the MMU tables. I suspect that's why you did a memset of the first 32KiB before enabling the dcache on the Arria 10.  I think the MMU is initialized okay since the SPL keeps executing, u-boot loads, and Linux boots after running the above (maybe that's not a fair assumption).

I had to write zeroes to the first 32kiB to init the ECC counters before putting MMU tables there.

You really should double check if the MMU and dcache are enabled, 8 seconds to scrub the memory is too long I think.

I added checks to verify that the MMU, icache, and dcache are all setup and enabled.

Calling icache_enable() set the CR_I bit (Icache enable) in the CR (control register).  Then calling dcache_enable() called the mmu_setup() function, which setup the MMU and set the CR_M bit (MMU enable) in the CR, and finally dcache_enable() set the CR_C bit (Dcache enable) bit in the CR.

I also printed out the control register before the memset calls, and it indicated that the mmu, icache, and dcache were enabled.

Is the DRAM area set as cacheable in the MMU tables ?

Good news bad news...  The MMU tables weren't being set up because the bd->bi_dram[bank].start and bd->bi_dram[bank].size weren't set up.  As a quick test, I hardcoded start to 0 and size to 1GiB.  After that, the memset was really quick, U-Boot loads, Linux loads, and everything seems to work great.

Good.

...

However, if I press the HPS_RST push button on the SoCKit (which is connected to power on reset), occasionally U-Boot will lock up while booting.  It always boots and operates correctly from the initial power on, but it almost always fails to boot after pressing the HPS_RST button.

Usually after pressing the HPS_RST button, U-Boot makes it past the SPL, and hangs somewhere after the call to setup_reloc() in ./common/board_f.c.  Once it hangs there, pressing the HPS_RST button again usually causes the SPL to hang while setting up the MMU (before my call to memset).  Eventually the WDT kicks in, and it just keeps hanging up in the same place.  Once it gets in this mode, the only way to recover it is by toggling power on the board.

I spent a bunch of time today trying to track down where it was hanging, but I couldn't pin point anything.  The MMU tables looked correct.  The MMU registers looked good.  I'm not sure the best way to debug what's going on.

Try triggering warm reset and cold reset via the reset register:

mw 0xffd05004 1 mw 0xffd05004 2

Does it hang in one case and not in the other ?

It hangs in both cases.

I did find that if I do not metset the last 1MiB of DRAM with the cache on, both warm and cold resets work.

I changed the ecc scrubbing to zero out the first 0x8000 bytes and the last 0x10000 bytes before the MMU is setup and I enable dcache.  Then with the dcache enabled, I zero out the rest of memory.  The resets work in this case as well.  So there seems to be some side effect of clearing out the relocate address space with the cache on.

On 7/10/2018 11:11 AM, Marek Vasut wrote:

On 07/10/2018 02:10 PM, Jason Rush wrote:

...

On 7/9/2018 3:08 AM, Marek Vasut wrote:

...

On 07/07/2018 12:56 AM, Jason Rush wrote:

...

On 7/5/2018 6:10 PM, Marek Vasut wrote:

...

On 07/06/2018 01:11 AM, Jason Rush wrote:

...

On 7/4/2018 2:23 AM, Marek Vasut wrote: > On 07/04/2018 01:45 AM, Jason Rush wrote: >> On 7/3/2018 9:08 AM, Marek Vasut wrote: >>> On 07/03/2018 03:58 PM, Jason Rush wrote: >>>> On 6/29/2018 10:17 AM, Marek Vasut wrote: >>>>> On 06/29/2018 05:06 PM, Jason Rush wrote: >>>>>> On 6/29/2018 9:52 AM, Marek Vasut wrote: >>>>>>> On 06/29/2018 04:44 PM, Jason Rush wrote: >>>>>>>> On 6/29/2018 9:34 AM, Marek Vasut wrote: >>>>>>>>> On 06/29/2018 04:31 PM, Jason Rush wrote: >>>>>>>>>> Dinh, >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>>> A while ago, you posted the following patchset for SoCFPGA to add the PL330 >>>>>>>>>> DMA driver, and updated the SoCFPGA SDRAM init to write zeros to SDRAM to >>>>>>>>>> initialize the ECC bits if ECC was enabled: >>>>>>>>>> >>>>>>>>>> https://lists.denx.de/pipermail/u-boot/2016-October/269643.html >>>>>>>>>> >>>>>>>>>> I know it's been a long time, so I'll summarize some of the conversation... >>>>>>>>>> >>>>>>>>>> At the time, you had a problem with the patchset causing the SPL to fail to >>>>>>>>>> find the MMC. You had tracked it down to an issue with the following commit >>>>>>>>>> "a78cd8613204 ARM: Rework and correct barrier definitions". You and Marek >>>>>>>>>> discussed it a bit, but I don't think there was a real conclusion. You >>>>>>>>>> submitted a second version of the patchset asking for advice on debugging >>>>>>>>>> the issue: >>>>>>>>>> >>>>>>>>>> https://lists.denx.de/pipermail/u-boot/2016-December/275822.html >>>>>>>>>> >>>>>>>>>> No real conversation came from the second patchset, and that was the end of >>>>>>>>>> the patch. >>>>>>>>>> >>>>>>>>>> I was hoping we could revisit adding your patchset again. I am working on a >>>>>>>>>> custom SoCFPGA board with a Cyclone V and ECC SDRAM. I rebased your patchset >>>>>>>>>> against v2018.05 and it is working on my custom board (although I don't have >>>>>>>>>> an MMC). I also tested it on a SoCKit booting from an MMC (I forced it to >>>>>>>>>> scrub the SDRAM on the SoCKit, because it doesn't have ECC RAM), and the >>>>>>>>>> SoCKit finds the MMC and boots. >>>>>>>>>> >>>>>>>>>> I don't have any suggestions on why it is working now on my board and not >>>>>>>>>> back when you first submitted the patchset. Maybe something else was fixed >>>>>>>>>> in the MMC? I was hoping you and Marek could test this patch again on some >>>>>>>>>> different SoCFPGA boards to see if you get the same results. >>>>>>>>> Look at this patch >>>>>>>>> http://git.denx.de/?p=u-boot/u-boot-socfpga.git;a=commit;h=9bb8a249b292d26f1... >>>>>>>>> >>>>>>>>> You likely want similar approach, it's faster then the DMA and much simpler. >>>>>>>>> >>>>>>>> Thanks Marek.  I'll give it a try.  Would you be interested in a similar patch for the Gen 5? >>>>>>> I don't have any Gen5 board which uses ECC, do you ? >>>>>>> If so, yes, prepare a patch, it should be very similar. >>>>>>> >>>>>>> Make sure to measure how long it takes to scrub the memory and how much >>>>>>> memory you have, I'd be interested in the numbers. >>>>>>> >>>>>> Looking at the master branch, it doesn't look like that code is ever being called? >>>>>> The sdram_init_ecc_bits() function is called from the ddr_calibration_sequence function(), >>>>>> but I can't find where ddr_calibration_sequence is called(). >>>>> git grep for it, it's called from somewhere in the arch/arm/mach-socfpga/ >>>>> >>>>>> Either way, I can test it. I have a custom Cyclone V board with ECC, and the Intel Arria V SoC >>>>>> Dev Kit I can test it on too which I think has ECC. >>>>> Please do. >>>>> >>>> I implemented a similar memset approach for the gen 5 socfpga.  It's basically the same >>>> code as in that patch; however, when I performed a single memset the processor would >>>> reset for some reason.  I changed it to loop over calling memset with a size of 32MB over >>>> the entire address the address, and that worked as opposed to doing a single memset on >>>> the RAM. >>> Can you do grep MEMSET .config in your U-Boot build dir ? The arch >>> memset is implemented in assembler and doesn't trigger WDT , so if it >>> takes too long, it could be that the WDT resets the platform. >> Both CONFIG_USE_ARCH_MEMSET and CONFIG_SPL_USE_ARCH_MEMSET >> are set in my .config, so it must be the WDT triggering as you suspect. >> >>>> I started on a SoCKit because it was handy, I know it doesn't have ECC >>> It doesn't by default. >>> >>>> , but I forced it to >>>> initialize the RAM as a quick test.  It seems much slower than the DMA approach.  It >>>> should be noted, I didn't implement any code to time the scrubbing, but rather just >>>> roughly monitored the time to get a rough idea of how long it took. >>>> >>>> On the SoCKit, which has 1GB of RAM, the memset takes around 8 seconds to complete, >>>> and the DMA takes under 2 seconds. >>> Did you enable i/d cache in the SPL ? It's mandatory, otherwise it's >>> slow. >> I have calls to icache_enable() and dcache_enable() just as you do in >> the Arria 10 sdram_init_ecc_bits() function. >> >> I did double check that both these enable functions call the versions >> of the functions in the ./arch/arm/lib/cache-cp15.c file that are >> implemented in the SPL.  So I believe that both icache and dcache is >> enabled. > Are you sure it's not just the stubs that are called ? Or that the code > doesn't skip the dcache enabling due to some funny stuff, like MMU being > already enabled ? I added prints to ensure it is calling the real icache_enable()/dcache_enable() functions, and not the stubs.

>> I probably should have added a print of icache_status() and >> dcache_status() to verify the caches are enabled.  I'll add that >> tomorrow. > Yes, you really should verify that the dcache was enabled. > >>> Just be careful about the MMU tables placement, they are big and >>> if you place them in RAM, make sure you don't overwrite them with the >>> memset. The trick might be to memset the first 1 MiB of RAM, then put >>> MMU tables at some offset therein (since 0x0 can be used for ARM >>> vectors) and then turn on i/d cache and memset the rest. >> That is essentially what I am doing I believe, with the exception that I >> am only clearing the first 32KiB before initializing the MMU table (which >> is what you did in the Arria 10 version). >> >> I modeled my code almost identically to yours with the exception that >> I loop over the memset calls 32MiB at a time. Here's the order of >> operations I perform: >> >> 1. icache_enable() >> 2. memset the first 0x8000 bytes to zero >> 3. setup gd->arch.tlb_arch and gd->arch.tlb_size >> 4. dcache_enable() >> 5. loop over remaining memory, memsetting 32MiB at a time to zero >> 6. flush_dcache_all() >> 7. dcache_disable() >> >> It looks like the call to dcache_enable is what sets up the MMU tables. >> I suspect that's why you did a memset of the first 32KiB before enabling >> the dcache on the Arria 10.  I think the MMU is initialized okay since the >> SPL keeps executing, u-boot loads, and Linux boots after running the >> above (maybe that's not a fair assumption). > I had to write zeroes to the first 32kiB to init the ECC counters before > putting MMU tables there. > > You really should double check if the MMU and dcache are enabled, 8 > seconds to scrub the memory is too long I think. I added checks to verify that the MMU, icache, and dcache are all setup and enabled.

Calling icache_enable() set the CR_I bit (Icache enable) in the CR (control register).  Then calling dcache_enable() called the mmu_setup() function, which setup the MMU and set the CR_M bit (MMU enable) in the CR, and finally dcache_enable() set the CR_C bit (Dcache enable) bit in the CR.

I also printed out the control register before the memset calls, and it indicated that the mmu, icache, and dcache were enabled.

Is the DRAM area set as cacheable in the MMU tables ?

Good news bad news...  The MMU tables weren't being set up because the bd->bi_dram[bank].start and bd->bi_dram[bank].size weren't set up.  As a quick test, I hardcoded start to 0 and size to 1GiB.  After that, the memset was really quick, U-Boot loads, Linux loads, and everything seems to work great.

Good.

...

However, if I press the HPS_RST push button on the SoCKit (which is connected to power on reset), occasionally U-Boot will lock up while booting.  It always boots and operates correctly from the initial power on, but it almost always fails to boot after pressing the HPS_RST button.

Usually after pressing the HPS_RST button, U-Boot makes it past the SPL, and hangs somewhere after the call to setup_reloc() in ./common/board_f.c.  Once it hangs there, pressing the HPS_RST button again usually causes the SPL to hang while setting up the MMU (before my call to memset).  Eventually the WDT kicks in, and it just keeps hanging up in the same place.  Once it gets in this mode, the only way to recover it is by toggling power on the board.

I spent a bunch of time today trying to track down where it was hanging, but I couldn't pin point anything.  The MMU tables looked correct.  The MMU registers looked good.  I'm not sure the best way to debug what's going on.

Try triggering warm reset and cold reset via the reset register:

mw 0xffd05004 1 mw 0xffd05004 2

Does it hang in one case and not in the other ?

It hangs in both cases.

I did find that if I do not metset the last 1MiB of DRAM with the cache on, both warm and cold resets work.

I changed the ecc scrubbing to zero out the first 0x8000 bytes and the last 0x10000 bytes before the MMU is setup and I enable dcache.  Then with the dcache enabled, I zero out the rest of memory.  The resets work in this case as well.  So there seems to be some side effect of clearing out the relocate address space with the cache on.

Can you investigate ?

I'd be happy to investigate more, but I'm not really sure what my next step should be.

Something appears to be happening differently when U-Boot relocates if the dcache is on.  But don't know how to track it down.

I was thinking I might dump the DRAM where U-Boot relocates to both with the dcache on and off, and see if there are any differences.  I'm not really sure what that tells me though if I find a difference.

Any suggestions?

Regards, Jason

On 7/11/2018 3:55 AM, Marek Vasut wrote:

On 07/11/2018 05:11 AM, Jason Rush wrote: [...]

...

...

...

...

...

However, if I press the HPS_RST push button on the SoCKit (which is connected to power on reset), occasionally U-Boot will lock up while booting.  It always boots and operates correctly from the initial power on, but it almost always fails to boot after pressing the HPS_RST button.

Usually after pressing the HPS_RST button, U-Boot makes it past the SPL, and hangs somewhere after the call to setup_reloc() in ./common/board_f.c.  Once it hangs there, pressing the HPS_RST button again usually causes the SPL to hang while setting up the MMU (before my call to memset).  Eventually the WDT kicks in, and it just keeps hanging up in the same place.  Once it gets in this mode, the only way to recover it is by toggling power on the board.

I spent a bunch of time today trying to track down where it was hanging, but I couldn't pin point anything.  The MMU tables looked correct.  The MMU registers looked good.  I'm not sure the best way to debug what's going on.

Try triggering warm reset and cold reset via the reset register:

mw 0xffd05004 1 mw 0xffd05004 2

Does it hang in one case and not in the other ?

It hangs in both cases.

I did find that if I do not metset the last 1MiB of DRAM with the cache on, both warm and cold resets work.

I changed the ecc scrubbing to zero out the first 0x8000 bytes and the last 0x10000 bytes before the MMU is setup and I enable dcache.  Then with the dcache enabled, I zero out the rest of memory.  The resets work in this case as well.  So there seems to be some side effect of clearing out the relocate address space with the cache on.

Can you investigate ?

I'd be happy to investigate more, but I'm not really sure what my next step should be.

Something appears to be happening differently when U-Boot relocates if the dcache is on.  But don't know how to track it down.

IIRC I disabled cache after scrubbing.

My mistake.  I did disable the dcache after scrubbing too.  The code is almost identical to the Arria10 code where after scrubbing it flushes the dcache, then turns it off.

The weird reset problems happens if I scrub the area where u-boot relocates to with the dcache on, then turn dcache off.

I tried to also tried turning the MMU off, but that didn't help.

...

I was thinking I might dump the DRAM where U-Boot relocates to both with the dcache on and off, and see if there are any differences.  I'm not really sure what that tells me though if I find a difference.

Any suggestions?

Regards, Jason

On 7/11/2018 8:48 AM, Marek Vasut wrote:

On 07/11/2018 03:49 PM, Jason Rush wrote:

...

On 7/11/2018 3:55 AM, Marek Vasut wrote:

...

On 07/11/2018 05:11 AM, Jason Rush wrote: [...]

...

...

...

>> However, if I press the HPS_RST push button on the SoCKit (which is connected >> to power on reset), occasionally U-Boot will lock up while booting.  It always >> boots and operates correctly from the initial power on, but it almost always >> fails to boot after pressing the HPS_RST button. >> >> Usually after pressing the HPS_RST button, U-Boot makes it past the SPL, and >> hangs somewhere after the call to setup_reloc() in ./common/board_f.c.  Once >> it hangs there, pressing the HPS_RST button again usually causes the SPL to >> hang while setting up the MMU (before my call to memset).  Eventually the >> WDT kicks in, and it just keeps hanging up in the same place.  Once it gets in >> this mode, the only way to recover it is by toggling power on the board. >> >> I spent a bunch of time today trying to track down where it was hanging, but >> I couldn't pin point anything.  The MMU tables looked correct.  The MMU >> registers looked good.  I'm not sure the best way to debug what's going on. > Try triggering warm reset and cold reset via the reset register: > > mw 0xffd05004 1 > mw 0xffd05004 2 > > Does it hang in one case and not in the other ? > It hangs in both cases.

I did find that if I do not metset the last 1MiB of DRAM with the cache on, both warm and cold resets work.

I changed the ecc scrubbing to zero out the first 0x8000 bytes and the last 0x10000 bytes before the MMU is setup and I enable dcache.  Then with the dcache enabled, I zero out the rest of memory.  The resets work in this case as well.  So there seems to be some side effect of clearing out the relocate address space with the cache on.

Can you investigate ?

I'd be happy to investigate more, but I'm not really sure what my next step should be.

Something appears to be happening differently when U-Boot relocates if the dcache is on.  But don't know how to track it down.

IIRC I disabled cache after scrubbing.

My mistake.  I did disable the dcache after scrubbing too.  The code is almost identical to the Arria10 code where after scrubbing it flushes the dcache, then turns it off.

The weird reset problems happens if I scrub the area where u-boot relocates to with the dcache on, then turn dcache off.

I tried to also tried turning the MMU off, but that didn't help.

Maybe there are some data used by the SPL there ? I think the SPL has malloc area in RAM at some point.

I thought something similar, so I narrowed it down to clearing just from where U-Boot relocates to the end of DRAM.  If I'm correct, that includes where U-Boot relocates and where the MMU tables are normally stored.

On Wed, 2018-07-11 at 08:56 -0500, Jason Rush wrote:

On 7/11/2018 8:48 AM, Marek Vasut wrote:

...

On 07/11/2018 03:49 PM, Jason Rush wrote:

...

My mistake. I did disable the dcache after scrubbing too. The code is almost identical to the Arria10 code where after scrubbing it flushes the dcache, then turns it off.

The weird reset problems happens if I scrub the area where u-boot relocates to with the dcache on, then turn dcache off.

I tried to also tried turning the MMU off, but that didn't help.

Maybe there are some data used by the SPL there ? I think the SPL has malloc area in RAM at some point.

I thought something similar, so I narrowed it down to clearing just from where U-Boot relocates to the end of DRAM. If I'm correct, that includes where U-Boot relocates and where the MMU tables are normally stored.

I wonder if the reset does not properly reset the CPU caches? The idea being that the CPU cache has stale data from before the reset, or maybe just stale tag bits, and the hang is due to using this bad data from the cache.

Or perhaps there is always something done incorrectly, but it is only the state of DRAM after a reset, vs a power cycle, that consistently triggers a hang?

If possible, try to add code before the hang point to invalidate both the i-cache and d-cache for the problem region above. Perhaps the SPL is doing something wrong w.r.t. cache invalidation, e.g. moving code around and not updating the i-cache, because it assumes nothing has yet used the caches, which is now no longer the case since you turn them on for scrubbing.

On 7/11/2018 1:54 PM, Marek Vasut wrote:

On 07/11/2018 07:30 PM, Trent Piepho wrote:

...

On Wed, 2018-07-11 at 08:56 -0500, Jason Rush wrote:

...

On 7/11/2018 8:48 AM, Marek Vasut wrote:

...

On 07/11/2018 03:49 PM, Jason Rush wrote:

...

My mistake. I did disable the dcache after scrubbing too. The code is almost identical to the Arria10 code where after scrubbing it flushes the dcache, then turns it off.

The weird reset problems happens if I scrub the area where u-boot relocates to with the dcache on, then turn dcache off.

I tried to also tried turning the MMU off, but that didn't help.

Maybe there are some data used by the SPL there ? I think the SPL has malloc area in RAM at some point.

I thought something similar, so I narrowed it down to clearing just from where U-Boot relocates to the end of DRAM. If I'm correct, that includes where U-Boot relocates and where the MMU tables are normally stored.

I wonder if the reset does not properly reset the CPU caches? The idea being that the CPU cache has stale data from before the reset, or maybe just stale tag bits, and the hang is due to using this bad data from the cache.

That's why we do dcache_off() icache_off() first, to make sure the caches are in consistent state. But a good point nonetheless, this should be checked.

I call dcache_off() and icache_off() after scrubbing, and I verified the MMU control register indicated they were off.

...

Or perhaps there is always something done incorrectly, but it is only the state of DRAM after a reset, vs a power cycle, that consistently triggers a hang?

The SoCFPGA has some weird warm/cold reset hooks even in the bootrom, could be. But the DRAM should be torn down in either case.

Maybe there is something wrong with the reset hooks? I could try and clear the on chip RAM after U-Boot relocates just to see what happens.  Or there is probably an enable for the hooks I could try and disable.

...

If possible, try to add code before the hang point to invalidate both the i-cache and d-cache for the problem region above. Perhaps the SPL is doing something wrong w.r.t. cache invalidation, e.g. moving code around and not updating the i-cache, because it assumes nothing has yet used the caches, which is now no longer the case since you turn them on for scrubbing.

After scrubbing I first call flush_dcache_all(), then I added calls to invalidate_icache_all() and invalidate_dcache_all(), and finally I call dcache_off() and icache_off().  I wasn't sure about the order I should call them, but there was no change.