[U-Boot] Very slow mtest on i.MX6UL over dual-die DDR3 (two chip selects)
Greetings,
I'm adding support to a 1GiB DDR3 chip that internally has two dies. I have configured the i.MX6UL memory controller for using two chip selects and assigned 512MiB to each. Although the RAM seems to work fine I'm experiencing extremely slow performance compared to a similar 1GiB chip with only one die (one chip select).
Testing one mtest loop over the first 496MiB takes: - 5s on the single die chip - 1m 16s on the dual die chip
I have run different memory benchmark tools in Linux like https://github.com/bingmann/pmbw.git and both memories perform similarly, but for some reason I see this weird behavior in U-Boot (v2015.04) with 'mtest'. The CONFIG_SYS_MEMTEST_START and CONFIG_SYS_MEMTEST_END are defined within secure limits and 'mtest' is not failing... it's just very slow.
Am I missing something obvious? Thanks -- Héctor Palacios
Hi Hector,
On Tue, May 16, 2017 at 1:15 PM, Palacios, Hector Hector.Palacios@digi.com wrote:
Greetings,
I'm adding support to a 1GiB DDR3 chip that internally has two dies. I have configured the i.MX6UL memory controller for using two chip selects and assigned 512MiB to each. Although the RAM seems to work fine I'm experiencing extremely slow performance compared to a similar 1GiB chip with only one die (one chip select).
Testing one mtest loop over the first 496MiB takes:
- 5s on the single die chip
- 1m 16s on the dual die chip
I have run different memory benchmark tools in Linux like https://github.com/bingmann/pmbw.git and both memories perform similarly, but for some reason I see this weird behavior in U-Boot (v2015.04) with 'mtest'. The CONFIG_SYS_MEMTEST_START and CONFIG_SYS_MEMTEST_END are defined within secure limits and 'mtest' is not failing... it's just very slow.
Am I missing something obvious?
It is hard to tell without seeing your code.
What about adding your mx6ul board support upstream?
Hi Fabio,
On Wed, 17 May 2017 00:34, Fabio Estevam wrote:
Hi Hector,
On Tue, May 16, 2017 at 1:15 PM, Palacios, Hector Hector.Palacios@digi.com wrote:
Greetings,
I'm adding support to a 1GiB DDR3 chip that internally has two dies. I
have configured the i.MX6UL memory controller for using two chip selects and assigned 512MiB to each. Although the RAM seems to work fine I'm experiencing extremely slow performance compared to a similar 1GiB chip with only one die (one chip select).
Testing one mtest loop over the first 496MiB takes:
- 5s on the single die chip
- 1m 16s on the dual die chip
I have run different memory benchmark tools in Linux like
https://github.com/bingmann/pmbw.git and both memories perform similarly, but for some reason I see this weird behavior in U-Boot (v2015.04) with 'mtest'.
The CONFIG_SYS_MEMTEST_START and CONFIG_SYS_MEMTEST_END are defined within secure limits and 'mtest' is not failing... it's just very slow.
Am I missing something obvious?
It is hard to tell without seeing your code.
What about adding your mx6ul board support upstream?
The code is on Github [1] (well, not the dual-die DDR3 yet) but there isn't much to see for this issue other than:
#define CONFIG_CMD_MEMTEST #define CONFIG_SYS_MEMTEST_START 0x80000000 #define CONFIG_SYS_MEMTEST_END 0x9fffffff /* 512 MiB */
For the dual-die DDR3, I apply the following change to the board .cfg file:
diff --git a/board/digi/ccimx6ul/imximage1GB.cfg b/board/digi/ccimx6ul/imximage1GB.cfg index f72888bc2bfe..952458d45970 100644 --- a/board/digi/ccimx6ul/imximage1GB.cfg +++ b/board/digi/ccimx6ul/imximage1GB.cfg @@ -115,8 +115,8 @@ DATA 4 0x021B0018 0x00201740 DATA 4 0x021B001C 0x00008000 DATA 4 0x021B002C 0x000026D2 DATA 4 0x021B0030 0x008F1023 -DATA 4 0x021B0040 0x0000005F -DATA 4 0x021B0000 0x85180000 +DATA 4 0x021B0040 0x0000004F +DATA 4 0x021B0000 0xC5180000
/* Mode registers writes for CS0 */ DATA 4 0x021B001C 0x02088032 @@ -125,6 +125,13 @@ DATA 4 0x021B001C 0x00048031 DATA 4 0x021B001C 0x15208030 DATA 4 0x021B001C 0x04008040
+/* Mode registers writes for CS1 */ +DATA 4 0x021B001C 0x0208803A +DATA 4 0x021B001C 0x0000803B +DATA 4 0x021B001C 0x00048039 +DATA 4 0x021B001C 0x15208038 +DATA 4 0x021B001C 0x04008048 + /* Final DDR setup */ DATA 4 0x021B0020 0x00000800 DATA 4 0x021B0818 0x00000227
So basically it's enabling the CS1 of the MMDC, and reducing the CS0_END to half the memory per chip select.
[1] https://github.com/digi-embedded/u-boot
Thanks, -- Héctor Palacios
Hi Hector,
On Wed, May 17, 2017 at 5:50 AM, Palacios, Hector Hector.Palacios@digi.com wrote:
The code is on Github [1] (well, not the dual-die DDR3 yet) but there isn't much to see for this issue other than:
I looked at your code and I see:
int dram_init(void) { gd->ram_size = ((ulong)CONFIG_DDR_MB * SZ_1M);
return 0; }
which may be worth investigating.
Take a look in this same function at board/freescale/mx53loco/mx53loco.c and also in include/configs/mx53loco.h (hint: we pass #define CONFIG_NR_DRAM_BANKS 2).
Hi Fabio,
On 05/17/2017 11:08 PM, Fabio Estevam wrote:
Hi Hector,
On Wed, May 17, 2017 at 5:50 AM, Palacios, Hector Hector.Palacios@digi.com wrote:
The code is on Github [1] (well, not the dual-die DDR3 yet) but there isn't much to see for this issue other than:
I looked at your code and I see:
int dram_init(void) { gd->ram_size = ((ulong)CONFIG_DDR_MB * SZ_1M);
return 0; }
which may be worth investigating.
Take a look in this same function at board/freescale/mx53loco/mx53loco.c and also in include/configs/mx53loco.h (hint: we pass #define CONFIG_NR_DRAM_BANKS 2).
In case I wasn't clear, I don't have any memory mapping problems. I have one single DDR chip, which internally uses two dies and two chip selects. Setting CONFIG_NR_DRAM_BANKS to 1 (and full size) or 2 (and half size) in U-Boot doesn't make any difference in terms of performance.
Additional investigation showed the following:
NXP DDR stress test takes the same time to complete (successfully) in both variants (single-die with one chip select and 1GB density per CS, and dual-die with two chip select and 512MB per CS)
The slow memory access is global in U-Boot, not limited to 'mtest' command. A memory copy command for 256M (time cp.l 90000000 80000000 4000000) takes: - 1.304s on the single-die DDR3 - 15.866 seconds on the dual-die DDR3
Note that both data/instuction cache are ON on both devices, and that I'm only exercising the lower memory (only CS0 on the dual-die chip) to avoid potential issues or delays between changing from CS0 to CS1.
I also verified that configuring the MMDC0 for using only one CS on the dual-die DDR3 chip (and only half the size), does not help. This DDR is still performing slowly in U-Boot, but I can't find the reason why.
Thanks for your help, anyway. -- Hector Palacios
Hi Hector,
On Thu, May 18, 2017 at 11:43 AM, Hector Palacios hector.palacios@digi.com wrote:
In case I wasn't clear, I don't have any memory mapping problems. I have one single DDR chip, which internally uses two dies and two chip selects. Setting CONFIG_NR_DRAM_BANKS to 1 (and full size) or 2 (and half size) in U-Boot doesn't make any difference in terms of performance.
Ok, thanks for the clarification.
Additional investigation showed the following:
NXP DDR stress test takes the same time to complete (successfully) in both variants (single-die with one chip select and 1GB density per CS, and dual-die with two chip select and 512MB per CS)
The slow memory access is global in U-Boot, not limited to 'mtest' command. A memory copy command for 256M (time cp.l 90000000 80000000 4000000) takes:
- 1.304s on the single-die DDR3
- 15.866 seconds on the dual-die DDR3
Note that both data/instuction cache are ON on both devices, and that I'm only exercising the lower memory (only CS0 on the dual-die chip) to avoid potential issues or delays between changing from CS0 to CS1.
I also verified that configuring the MMDC0 for using only one CS on the dual-die DDR3 chip (and only half the size), does not help. This DDR is still performing slowly in U-Boot, but I can't find the reason why.
I got very slow performance with MX6UL when U-Boot is loaded via serial download mode.
This gets fixed by setting the SMP bit: https://patchwork.ozlabs.org/patch/747074/
Looks like your issue is not related though.
Another experiment: could you try the same tests running on U-Boot 2017.05?
Hi Fabio,
On Thu, 18 May 2017 17:52, Fabio Estevam wrote:
I got very slow performance with MX6UL when U-Boot is loaded via serial download mode.
That's it, actually! I was booting the target via USB, booting from the NAND shows normal performance.
This gets fixed by setting the SMP bit: https://patchwork.ozlabs.org/patch/747074/
I don't have that patch. I guess a similar code exists somewhere that is not entered when booting from USB.
Many thanks for your help! -- Héctor Palacios
Hi Hector,
On Thu, May 18, 2017 at 2:06 PM, Palacios, Hector Hector.Palacios@digi.com wrote:
Hi Fabio,
On Thu, 18 May 2017 17:52, Fabio Estevam wrote:
I got very slow performance with MX6UL when U-Boot is loaded via serial download mode.
That's it, actually! I was booting the target via USB, booting from the NAND shows normal performance.
Excellent, now we are making progress :-)
This gets fixed by setting the SMP bit: https://patchwork.ozlabs.org/patch/747074/
I don't have that patch. I guess a similar code exists somewhere that is not entered when booting from USB.
This patch has not been applied in U-Boot mainline yet. You can apply it locally on your code for the time being.
When the board boots from SD/NAND the SMP bit is set by the ROM code and then we do not have this problem.
When booting from serial download mode the SMP bit is not set the system gets very slow because caches are disabled.
Regards,
Fabio Estevam
participants (3)
-
Fabio Estevam -
Hector Palacios -
Palacios, Hector