On Wed, 4 Jan 2023, 8:52 PM Pali Rohár, pali@kernel.org wrote:

On Wednesday 04 January 2023 08:50:28 Pali Rohár wrote:

...

Hello!

On Tuesday 03 January 2023 17:55:41 Tony Dinh wrote:

...

Hi Pali,

I'm building a new u-boot for the Thecus N2350 board (Armada 385 dual-core 1Ghz 1GB DDR4). The trouble with this board is DDR4, which is not currently supported in u-boot (also cc this to Chris for commenting about Marvell DDR4 training driver).

Yes, ddr4 training code is not in u-boot yet.

A38x u-boot ddr driver is in this directory:

https://source.denx.de/u-boot/u-boot/-/tree/master/drivers/ddr/marvell/a38x

...

And it is copied from the original marvell driver by stripping non-a38x code and removal of the ddr4 code from the master branch: https://github.com/MarvellEmbeddedProcessors/mv-ddr-marvell

So you can try to copy code again without stripping ddr4 parts (run git log drivers/ddr/marvell/a38x in u-boot)

https://source.denx.de/u-boot/u-boot/-/commit/107c3391b95bcc2ba09a876da4fa0c...

...

And adjust u-boot board code for ddr4.

I guess it could work but it would be needed to play with it.

Last time I looked the MarvellEmbeddedProcessors stuff was a long way behind what Marvell are releasing to customers. That was for the newer SoCs so maybe the A385 stuff is current.

...

...

So I'm building with binary.0 in the ./board/thecus/n2350/. This binary.0 is a copy of the Marvell bin_hdr for SPI in the GPL source for this board (provided by Thecus). My ./board/thecus/n2350/kwbimage.cfg.in file looks like this

# Armada 38x uses version 1 image format VERSION 1 # Boot Media configurations BOOT_FROM spi # Binary Header (bin_hdr) with DDR4 training code BINARY board/thecus/n2350/binary.0 0000005b 00000068

When I kwboot the u-boot.kwb image (using kwboot binary built with 2023.01-rc4), the header was transferred successfully, but then the BootROM jumped to SPI u-boot, instead of loading the u-boot payload from UART. Please see the log below.

<BEGIN LOG> # ./kwboot -t -p -B 115200 /dev/ttyUSB0 -b uboot.kwb

kwboot version 2023.01-tld-1-00048-g19e15f9081-dirty Patching image boot signature to UART Aligning image header to Xmodem block size Sending boot message. Please reboot the target.../ Sending boot image header (97792 bytes)... 0 %

[......................................................................]

...

...

9 %

[......................................................................]

...

...

<snip> 82 %

[......................................................................]

...

...

91 %

[................................................................ ]

...

...

Done

BootROM - 1.73 Booting from SPI flash

This indicates reset of the board. BootROM started execution of the image from the primary location.

...

<snip> U-Boot 2013.01 (Nov 12 2018 - 20:56:19) Marvell version:

2015_T1.0p18-tld-4

...

...

<END LOG>

It seems kwboot patched the image, but the BOOT_FROM indicator was still recognized as from SPI. So the BootROM loaded the stock u-boot image from SPI and executed it. Since I am booting with bin_hdr, I'm not sure if there is something inside this blob that has forced this indicator to SPI.

No, blob cannot force BootROM to switch boot location. This really indicates crash of the blob or crash of the payload which results in the board reset.

...

I'm attaching the u-boot.kwb image to this email, in case you are interested in taking a look at the structure.

Thanks, Tony

Hello,

On Wed, Jan 4, 2023 at 9:44 AM Pali Rohár pali@kernel.org wrote:

On Wednesday 04 January 2023 23:41:05 Chris Packham wrote:

...

On Wed, 4 Jan 2023, 8:52 PM Pali Rohár, pali@kernel.org wrote:

...

On Wednesday 04 January 2023 08:50:28 Pali Rohár wrote:

...

Hello!

On Tuesday 03 January 2023 17:55:41 Tony Dinh wrote:

...

Hi Pali,

I'm building a new u-boot for the Thecus N2350 board (Armada 385 dual-core 1Ghz 1GB DDR4). The trouble with this board is DDR4, which is not currently supported in u-boot (also cc this to Chris for commenting about Marvell DDR4 training driver).

Yes, ddr4 training code is not in u-boot yet.

A38x u-boot ddr driver is in this directory:

https://source.denx.de/u-boot/u-boot/-/tree/master/drivers/ddr/marvell/a38x

...

And it is copied from the original marvell driver by stripping non-a38x code and removal of the ddr4 code from the master branch: https://github.com/MarvellEmbeddedProcessors/mv-ddr-marvell

So you can try to copy code again without stripping ddr4 parts (run git log drivers/ddr/marvell/a38x in u-boot)

https://source.denx.de/u-boot/u-boot/-/commit/107c3391b95bcc2ba09a876da4fa0c...

...

And adjust u-boot board code for ddr4.

I guess it could work but it would be needed to play with it.

Last time I looked the MarvellEmbeddedProcessors stuff was a long way behind what Marvell are releasing to customers. That was for the newer SoCs so maybe the A385 stuff is current.

About half year ago, MarvellEmbeddedProcessors mv-ddr-marvell and A3700-utils-marvell were up-to-date and same as the version distributed to the Marvell customers. I was cooperating with Marvell to release all patches from Marvell Extranet portal to github as opensource and also to incorporate github patches from community to their Extranet version. Also I was synchronizing u-boot drivers/ddr/marvell/a38x directory with MarvellEmbeddedProcessors version. I do not think that Marvell released something super new in these repositories in last half of year, so I think that the code on MarvellEmbeddedProcessors (for those two repositories) is still up-to-date.

Thanks all for commenting. As Pali has suggested, I will try copying the latest Marvell Github DDR drivers.

All the best, Tony

...

...

...

...

So I'm building with binary.0 in the ./board/thecus/n2350/. This binary.0 is a copy of the Marvell bin_hdr for SPI in the GPL source for this board (provided by Thecus). My ./board/thecus/n2350/kwbimage.cfg.in file looks like this

# Armada 38x uses version 1 image format VERSION 1 # Boot Media configurations BOOT_FROM spi # Binary Header (bin_hdr) with DDR4 training code BINARY board/thecus/n2350/binary.0 0000005b 00000068

When I kwboot the u-boot.kwb image (using kwboot binary built with 2023.01-rc4), the header was transferred successfully, but then the BootROM jumped to SPI u-boot, instead of loading the u-boot payload from UART. Please see the log below.

<BEGIN LOG> # ./kwboot -t -p -B 115200 /dev/ttyUSB0 -b uboot.kwb

kwboot version 2023.01-tld-1-00048-g19e15f9081-dirty Patching image boot signature to UART Aligning image header to Xmodem block size Sending boot message. Please reboot the target.../ Sending boot image header (97792 bytes)... 0 %

[......................................................................]

...

...

9 %

[......................................................................]

...

...

<snip> 82 %

[......................................................................]

...

...

91 %

[................................................................ ]

...

...

Done

BootROM - 1.73 Booting from SPI flash

This indicates reset of the board. BootROM started execution of the image from the primary location.

...

<snip> U-Boot 2013.01 (Nov 12 2018 - 20:56:19) Marvell version:

2015_T1.0p18-tld-4

...

...

<END LOG>

It seems kwboot patched the image, but the BOOT_FROM indicator was still recognized as from SPI. So the BootROM loaded the stock u-boot image from SPI and executed it. Since I am booting with bin_hdr, I'm not sure if there is something inside this blob that has forced this indicator to SPI.

No, blob cannot force BootROM to switch boot location. This really indicates crash of the blob or crash of the payload which results in the board reset.

...

I'm attaching the u-boot.kwb image to this email, in case you are interested in taking a look at the structure.

Thanks, Tony

On Monday 09 January 2023 16:55:56 Tony Dinh wrote:

Hi Pali,

On Wed, Jan 4, 2023 at 1:04 PM Tony Dinh mibodhi@gmail.com wrote:

...

Hello,

On Wed, Jan 4, 2023 at 9:44 AM Pali Rohár pali@kernel.org wrote:

...

On Wednesday 04 January 2023 23:41:05 Chris Packham wrote:

...

On Wed, 4 Jan 2023, 8:52 PM Pali Rohár, pali@kernel.org wrote:

...

On Wednesday 04 January 2023 08:50:28 Pali Rohár wrote:

...

Hello!

On Tuesday 03 January 2023 17:55:41 Tony Dinh wrote: > Hi Pali, > > I'm building a new u-boot for the Thecus N2350 board (Armada 385 > dual-core 1Ghz 1GB DDR4). The trouble with this board is DDR4, which > is not currently supported in u-boot (also cc this to Chris for > commenting about Marvell DDR4 training driver).

Yes, ddr4 training code is not in u-boot yet.

A38x u-boot ddr driver is in this directory:

https://source.denx.de/u-boot/u-boot/-/tree/master/drivers/ddr/marvell/a38x

...

And it is copied from the original marvell driver by stripping non-a38x code and removal of the ddr4 code from the master branch: https://github.com/MarvellEmbeddedProcessors/mv-ddr-marvell

So you can try to copy code again without stripping ddr4 parts (run git log drivers/ddr/marvell/a38x in u-boot)

https://source.denx.de/u-boot/u-boot/-/commit/107c3391b95bcc2ba09a876da4fa0c...

...

And adjust u-boot board code for ddr4.

I guess it could work but it would be needed to play with it.

Last time I looked the MarvellEmbeddedProcessors stuff was a long way behind what Marvell are releasing to customers. That was for the newer SoCs so maybe the A385 stuff is current.

About half year ago, MarvellEmbeddedProcessors mv-ddr-marvell and A3700-utils-marvell were up-to-date and same as the version distributed to the Marvell customers. I was cooperating with Marvell to release all patches from Marvell Extranet portal to github as opensource and also to incorporate github patches from community to their Extranet version. Also I was synchronizing u-boot drivers/ddr/marvell/a38x directory with MarvellEmbeddedProcessors version. I do not think that Marvell released something super new in these repositories in last half of year, so I think that the code on MarvellEmbeddedProcessors (for those two repositories) is still up-to-date.

Thanks all for commenting. As Pali has suggested, I will try copying the latest Marvell Github DDR drivers.

I've tried to bring in the latest Marvell DDR drivers, but failed miserably after many hours playing :) I used your DDR3 commit info as a guide as how to strip out other parts, and only kept DDR3 and DDR4 for a38x:

https://source.denx.de/u-boot/u-boot/-/commit/107c3391b95bcc2ba09a876da4fa0c...

The current code has DDR4 interspersed in DDR3 with if-defs CONFIG_DDR4. And there are some minor dependencies on ATF types, quite difficult for me to get it to build cleanly. I guess I threw in the towel for now :) If you ever find some free time, please give it a shot.

Thanks, Tony

Hello! And what is the issue? DDR training is failing? Or SPL does not boot? Or it do not compile?

...

All the best, Tony

...

...

...

...

> So I'm building with binary.0 in the ./board/thecus/n2350/. This > binary.0 is a copy of the Marvell bin_hdr for SPI in the GPL source > for this board (provided by Thecus). My > ./board/thecus/n2350/kwbimage.cfg.in file looks like this > > # Armada 38x uses version 1 image format > VERSION 1 > # Boot Media configurations > BOOT_FROM spi > # Binary Header (bin_hdr) with DDR4 training code > BINARY board/thecus/n2350/binary.0 0000005b 00000068 > > When I kwboot the u-boot.kwb image (using kwboot binary built with > 2023.01-rc4), the header was transferred successfully, but then the > BootROM jumped to SPI u-boot, instead of loading the u-boot payload > from UART. Please see the log below. > <BEGIN LOG> > # ./kwboot -t -p -B 115200 /dev/ttyUSB0 -b uboot.kwb > > kwboot version 2023.01-tld-1-00048-g19e15f9081-dirty > Patching image boot signature to UART > Aligning image header to Xmodem block size > Sending boot message. Please reboot the target.../ > Sending boot image header (97792 bytes)... > 0 %

[......................................................................]

...

> 9 %

[......................................................................]

...

> <snip> > 82 %

[......................................................................]

...

> 91 %

[................................................................ ]

...

> Done > > BootROM - 1.73 > Booting from SPI flash

This indicates reset of the board. BootROM started execution of the image from the primary location.

> <snip> > U-Boot 2013.01 (Nov 12 2018 - 20:56:19) Marvell version:

2015_T1.0p18-tld-4

...

> <END LOG> > > It seems kwboot patched the image, but the BOOT_FROM indicator was > still recognized as from SPI. So the BootROM loaded the stock u-boot > image from SPI and executed it. Since I am booting with bin_hdr, I'm > not sure if there is something inside this blob that has forced this > indicator to SPI.

No, blob cannot force BootROM to switch boot location. This really indicates crash of the blob or crash of the payload which results in the board reset.

> I'm attaching the u-boot.kwb image to this email, in case you are > interested in taking a look at the structure. > > Thanks, > Tony

On Monday 09 January 2023 17:35:24 Tony Dinh wrote:

Hi Pali,

On Mon, Jan 9, 2023 at 5:02 PM Pali Rohár pali@kernel.org wrote:

...

On Monday 09 January 2023 16:55:56 Tony Dinh wrote:

...

Hi Pali,

On Wed, Jan 4, 2023 at 1:04 PM Tony Dinh mibodhi@gmail.com wrote:

...

Hello,

On Wed, Jan 4, 2023 at 9:44 AM Pali Rohár pali@kernel.org wrote:

...

On Wednesday 04 January 2023 23:41:05 Chris Packham wrote:

...

On Wed, 4 Jan 2023, 8:52 PM Pali Rohár, pali@kernel.org wrote:

> On Wednesday 04 January 2023 08:50:28 Pali Rohár wrote: > > Hello! > > > > On Tuesday 03 January 2023 17:55:41 Tony Dinh wrote: > > > Hi Pali, > > > > > > I'm building a new u-boot for the Thecus N2350 board (Armada 385 > > > dual-core 1Ghz 1GB DDR4). The trouble with this board is DDR4, which > > > is not currently supported in u-boot (also cc this to Chris for > > > commenting about Marvell DDR4 training driver). > > > > Yes, ddr4 training code is not in u-boot yet. > > > > A38x u-boot ddr driver is in this directory: > > > https://source.denx.de/u-boot/u-boot/-/tree/master/drivers/ddr/marvell/a38x > > > > And it is copied from the original marvell driver by stripping non-a38x > > code and removal of the ddr4 code from the master branch: > > https://github.com/MarvellEmbeddedProcessors/mv-ddr-marvell > > > > So you can try to copy code again without stripping ddr4 parts > > (run git log drivers/ddr/marvell/a38x in u-boot) > > > https://source.denx.de/u-boot/u-boot/-/commit/107c3391b95bcc2ba09a876da4fa0c... > > > And adjust u-boot board code for ddr4. > > > > I guess it could work but it would be needed to play with it. >

Last time I looked the MarvellEmbeddedProcessors stuff was a long way behind what Marvell are releasing to customers. That was for the newer SoCs so maybe the A385 stuff is current.

About half year ago, MarvellEmbeddedProcessors mv-ddr-marvell and A3700-utils-marvell were up-to-date and same as the version distributed to the Marvell customers. I was cooperating with Marvell to release all patches from Marvell Extranet portal to github as opensource and also to incorporate github patches from community to their Extranet version. Also I was synchronizing u-boot drivers/ddr/marvell/a38x directory with MarvellEmbeddedProcessors version. I do not think that Marvell released something super new in these repositories in last half of year, so I think that the code on MarvellEmbeddedProcessors (for those two repositories) is still up-to-date.

Thanks all for commenting. As Pali has suggested, I will try copying the latest Marvell Github DDR drivers.

I've tried to bring in the latest Marvell DDR drivers, but failed miserably after many hours playing :) I used your DDR3 commit info as a guide as how to strip out other parts, and only kept DDR3 and DDR4 for a38x:

https://source.denx.de/u-boot/u-boot/-/commit/107c3391b95bcc2ba09a876da4fa0c...

The current code has DDR4 interspersed in DDR3 with if-defs CONFIG_DDR4. And there are some minor dependencies on ATF types, quite difficult for me to get it to build cleanly. I guess I threw in the towel for now :) If you ever find some free time, please give it a shot.

Thanks, Tony

Hello! And what is the issue? DDR training is failing? Or SPL does not boot? Or it do not compile?

Each of the code files was compiled (after some includes adjustments). But the build failed to link due to some errors such as /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr_plat.c:558: undefined reference to `mv_ddr_freq_get'

It should have been seen by the linker (since that code was compiled), I could not see why. I feel I have spent too much time fighting the if-defs :)

Thanks, Tony

Send the whole error message and also ideally push your changes to some git repository. So I can look at the stage at which you are.

...

...

...

All the best, Tony

...

...

> > > > So I'm building with binary.0 in the ./board/thecus/n2350/. This > > > binary.0 is a copy of the Marvell bin_hdr for SPI in the GPL source > > > for this board (provided by Thecus). My > > > ./board/thecus/n2350/kwbimage.cfg.in file looks like this > > > > > > # Armada 38x uses version 1 image format > > > VERSION 1 > > > # Boot Media configurations > > > BOOT_FROM spi > > > # Binary Header (bin_hdr) with DDR4 training code > > > BINARY board/thecus/n2350/binary.0 0000005b 00000068 > > > > > > When I kwboot the u-boot.kwb image (using kwboot binary built with > > > 2023.01-rc4), the header was transferred successfully, but then the > > > BootROM jumped to SPI u-boot, instead of loading the u-boot payload > > > from UART. Please see the log below. > > > <BEGIN LOG> > > > # ./kwboot -t -p -B 115200 /dev/ttyUSB0 -b uboot.kwb > > > > > > kwboot version 2023.01-tld-1-00048-g19e15f9081-dirty > > > Patching image boot signature to UART > > > Aligning image header to Xmodem block size > > > Sending boot message. Please reboot the target.../ > > > Sending boot image header (97792 bytes)... > > > 0 % > [......................................................................] > > > 9 % > [......................................................................] > > > <snip> > > > 82 % > [......................................................................] > > > 91 % > [................................................................ ] > > > Done > > > > > > BootROM - 1.73 > > > Booting from SPI flash > > > > This indicates reset of the board. BootROM started execution of the > > image from the primary location. > > > > > <snip> > > > U-Boot 2013.01 (Nov 12 2018 - 20:56:19) Marvell version: > 2015_T1.0p18-tld-4 > > > <END LOG> > > > > > > It seems kwboot patched the image, but the BOOT_FROM indicator was > > > still recognized as from SPI. So the BootROM loaded the stock u-boot > > > image from SPI and executed it. Since I am booting with bin_hdr, I'm > > > not sure if there is something inside this blob that has forced this > > > indicator to SPI. > > > > No, blob cannot force BootROM to switch boot location. This really > > indicates crash of the blob or crash of the payload which results in the > > board reset. > > > > > I'm attaching the u-boot.kwb image to this email, in case you are > > > interested in taking a look at the structure. > > > > > > Thanks, > > > Tony > > > > >

Hi Pali,

On Mon, Jan 9, 2023 at 5:44 PM Tony Dinh mibodhi@gmail.com wrote:

On Mon, Jan 9, 2023 at 5:38 PM Pali Rohár pali@kernel.org wrote:

...

On Monday 09 January 2023 17:35:24 Tony Dinh wrote:

...

Hi Pali,

On Mon, Jan 9, 2023 at 5:02 PM Pali Rohár pali@kernel.org wrote:

...

On Monday 09 January 2023 16:55:56 Tony Dinh wrote:

...

Hi Pali,

On Wed, Jan 4, 2023 at 1:04 PM Tony Dinh mibodhi@gmail.com wrote:

...

Hello,

On Wed, Jan 4, 2023 at 9:44 AM Pali Rohár pali@kernel.org wrote: > > On Wednesday 04 January 2023 23:41:05 Chris Packham wrote: > > On Wed, 4 Jan 2023, 8:52 PM Pali Rohár, pali@kernel.org wrote: > > > > > On Wednesday 04 January 2023 08:50:28 Pali Rohár wrote: > > > > Hello! > > > > > > > > On Tuesday 03 January 2023 17:55:41 Tony Dinh wrote: > > > > > Hi Pali, > > > > > > > > > > I'm building a new u-boot for the Thecus N2350 board (Armada 385 > > > > > dual-core 1Ghz 1GB DDR4). The trouble with this board is DDR4, which > > > > > is not currently supported in u-boot (also cc this to Chris for > > > > > commenting about Marvell DDR4 training driver). > > > > > > > > Yes, ddr4 training code is not in u-boot yet. > > > > > > > > A38x u-boot ddr driver is in this directory: > > > > > > > https://source.denx.de/u-boot/u-boot/-/tree/master/drivers/ddr/marvell/a38x > > > > > > > > And it is copied from the original marvell driver by stripping non-a38x > > > > code and removal of the ddr4 code from the master branch: > > > > https://github.com/MarvellEmbeddedProcessors/mv-ddr-marvell > > > > > > > > So you can try to copy code again without stripping ddr4 parts > > > > (run git log drivers/ddr/marvell/a38x in u-boot) > > > > > > > > > https://source.denx.de/u-boot/u-boot/-/commit/107c3391b95bcc2ba09a876da4fa0c... > > > > > > > And adjust u-boot board code for ddr4. > > > > > > > > I guess it could work but it would be needed to play with it. > > > > > > > Last time I looked the MarvellEmbeddedProcessors stuff was a long way > > behind what Marvell are releasing to customers. That was for the newer SoCs > > so maybe the A385 stuff is current. > > About half year ago, MarvellEmbeddedProcessors mv-ddr-marvell and > A3700-utils-marvell were up-to-date and same as the version distributed > to the Marvell customers. I was cooperating with Marvell to release all > patches from Marvell Extranet portal to github as opensource and also to > incorporate github patches from community to their Extranet version. > Also I was synchronizing u-boot drivers/ddr/marvell/a38x directory with > MarvellEmbeddedProcessors version. I do not think that Marvell released > something super new in these repositories in last half of year, so I > think that the code on MarvellEmbeddedProcessors (for those two > repositories) is still up-to-date.

Thanks all for commenting. As Pali has suggested, I will try copying the latest Marvell Github DDR drivers.

I've tried to bring in the latest Marvell DDR drivers, but failed miserably after many hours playing :) I used your DDR3 commit info as a guide as how to strip out other parts, and only kept DDR3 and DDR4 for a38x:

https://source.denx.de/u-boot/u-boot/-/commit/107c3391b95bcc2ba09a876da4fa0c...

The current code has DDR4 interspersed in DDR3 with if-defs CONFIG_DDR4. And there are some minor dependencies on ATF types, quite difficult for me to get it to build cleanly. I guess I threw in the towel for now :) If you ever find some free time, please give it a shot.

Thanks, Tony

Hello! And what is the issue? DDR training is failing? Or SPL does not boot? Or it do not compile?

Each of the code files was compiled (after some includes adjustments). But the build failed to link due to some errors such as /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr_plat.c:558: undefined reference to `mv_ddr_freq_get'

It should have been seen by the linker (since that code was compiled), I could not see why. I feel I have spent too much time fighting the if-defs :)

Thanks, Tony

Send the whole error message and also ideally push your changes to some git repository. So I can look at the stage at which you are.

OK thanks!

I got burned for being lazy :) it turned out that the mix of #ifdef and #if defined was the problem. Just stepped away and came back for a few minutes and I can see that I just need to define the CONFIG_DDR4 properly in arch/arm/mach-mvebu/Kconfig and build it to pass the CONFIG_DDR4 stuff.

One more small hurdle after that was CONFIG_USE_PRIVATE_LIBGCC must be undefined for it to build (so I am using /usr/lib/gcc/arm-linux-gnueabi/10/libgcc.a, and not using arch/arm/lib/lib.a)

But Marvell DDR4 is working fine! Please see the log. <BEGIN LOG>

kwboot version 2023.01-tld-1-00048-g19e15f9081-dirty Patching image boot signature to UART Aligning image header to Xmodem block size Sending boot message. Please reboot the target.../ Sending boot image header (122112 bytes)... 0 % [......................................................................] 7 % [......................................................................] 14 % [......................................................................] <snip> 80 % [......................................................................] 88 % [......................................................................] 95 % [............................................ ] Done

U-Boot SPL 2023.01-tld-1-10204-g7b84c973b9-dirty (Jan 10 2023 - 20:39:38 -0800) High speed PHY - Version: 2.0 Detected Device ID 6820 board SerDes lanes topology details: | Lane # | Speed | Type | -------------------------------- | 0 | 0 | SGMII0 | | 1 | 3 | SATA0 | | 2 | 3 | SATA1 | | 4 | 5 | USB3 HOST0 | | 5 | 5 | USB3 HOST1 | -------------------------------- High speed PHY - Ended Successfully DDR4 Training Sequence - Switching XBAR Window to FastPath Window mv_ddr: completed successfully Trying to boot from BOOTROM Returning to BootROM (return address 0xffff05c4)...

Sending boot image data (471968 bytes)... 0 % [......................................................................] 1 % [......................................................................] <snip> 94 % [......................................................................] 96 % [......................................................................] 98 % [................................................ ] Done Finishing transfer [Type Ctrl-\ + c to quit]

U-Boot 2023.01-tld-1-10204-g7b84c973b9-dirty (Jan 10 2023 - 20:39:38 -0800) Thecus N2350

SoC: MV88F6820-A0 at 1066 MHz DRAM: 1 GiB (533 MHz, 32-bit, ECC not enabled) Core: 61 devices, 22 uclasses, devicetree: separate MMC: Loading Environment from SPIFlash... SF: Detected mx25l3205d with page size 256 Bytes, erase size 4 KiB, total 4 MiB *** Warning - bad CRC, using default environment

Model: Thecus N2350 Board: Thecus N2350 Net: Warning: ethernet@70000 (eth0) using random MAC address - d6:25:74:23:2c:21 eth0: ethernet@70000 Hit any key to stop autoboot: 0

<END LOG>

Thanks, Tony

Tony

...

...

...

...

...

All the best, Tony

> > > > > > > > So I'm building with binary.0 in the ./board/thecus/n2350/. This > > > > > binary.0 is a copy of the Marvell bin_hdr for SPI in the GPL source > > > > > for this board (provided by Thecus). My > > > > > ./board/thecus/n2350/kwbimage.cfg.in file looks like this > > > > > > > > > > # Armada 38x uses version 1 image format > > > > > VERSION 1 > > > > > # Boot Media configurations > > > > > BOOT_FROM spi > > > > > # Binary Header (bin_hdr) with DDR4 training code > > > > > BINARY board/thecus/n2350/binary.0 0000005b 00000068 > > > > > > > > > > When I kwboot the u-boot.kwb image (using kwboot binary built with > > > > > 2023.01-rc4), the header was transferred successfully, but then the > > > > > BootROM jumped to SPI u-boot, instead of loading the u-boot payload > > > > > from UART. Please see the log below. > > > > > <BEGIN LOG> > > > > > # ./kwboot -t -p -B 115200 /dev/ttyUSB0 -b uboot.kwb > > > > > > > > > > kwboot version 2023.01-tld-1-00048-g19e15f9081-dirty > > > > > Patching image boot signature to UART > > > > > Aligning image header to Xmodem block size > > > > > Sending boot message. Please reboot the target.../ > > > > > Sending boot image header (97792 bytes)... > > > > > 0 % > > > [......................................................................] > > > > > 9 % > > > [......................................................................] > > > > > <snip> > > > > > 82 % > > > [......................................................................] > > > > > 91 % > > > [................................................................ ] > > > > > Done > > > > > > > > > > BootROM - 1.73 > > > > > Booting from SPI flash > > > > > > > > This indicates reset of the board. BootROM started execution of the > > > > image from the primary location. > > > > > > > > > <snip> > > > > > U-Boot 2013.01 (Nov 12 2018 - 20:56:19) Marvell version: > > > 2015_T1.0p18-tld-4 > > > > > <END LOG> > > > > > > > > > > It seems kwboot patched the image, but the BOOT_FROM indicator was > > > > > still recognized as from SPI. So the BootROM loaded the stock u-boot > > > > > image from SPI and executed it. Since I am booting with bin_hdr, I'm > > > > > not sure if there is something inside this blob that has forced this > > > > > indicator to SPI. > > > > > > > > No, blob cannot force BootROM to switch boot location. This really > > > > indicates crash of the blob or crash of the payload which results in the > > > > board reset. > > > > > > > > > I'm attaching the u-boot.kwb image to this email, in case you are > > > > > interested in taking a look at the structure. > > > > > > > > > > Thanks, > > > > > Tony > > > > > > > > > > >

Hi Pali,

On Wed, Jan 11, 2023 at 12:04 AM Pali Rohár pali@kernel.org wrote:

On Tuesday 10 January 2023 21:07:45 Tony Dinh wrote:

...

Hi Pali,

On Mon, Jan 9, 2023 at 5:44 PM Tony Dinh mibodhi@gmail.com wrote:

...

On Mon, Jan 9, 2023 at 5:38 PM Pali Rohár pali@kernel.org wrote:

...

On Monday 09 January 2023 17:35:24 Tony Dinh wrote:

...

Hi Pali,

On Mon, Jan 9, 2023 at 5:02 PM Pali Rohár pali@kernel.org wrote:

...

On Monday 09 January 2023 16:55:56 Tony Dinh wrote: > Hi Pali, > > On Wed, Jan 4, 2023 at 1:04 PM Tony Dinh mibodhi@gmail.com wrote: > > > > Hello, > > > > On Wed, Jan 4, 2023 at 9:44 AM Pali Rohár pali@kernel.org wrote: > > > > > > On Wednesday 04 January 2023 23:41:05 Chris Packham wrote: > > > > On Wed, 4 Jan 2023, 8:52 PM Pali Rohár, pali@kernel.org wrote: > > > > > > > > > On Wednesday 04 January 2023 08:50:28 Pali Rohár wrote: > > > > > > Hello! > > > > > > > > > > > > On Tuesday 03 January 2023 17:55:41 Tony Dinh wrote: > > > > > > > Hi Pali, > > > > > > > > > > > > > > I'm building a new u-boot for the Thecus N2350 board (Armada 385 > > > > > > > dual-core 1Ghz 1GB DDR4). The trouble with this board is DDR4, which > > > > > > > is not currently supported in u-boot (also cc this to Chris for > > > > > > > commenting about Marvell DDR4 training driver). > > > > > > > > > > > > Yes, ddr4 training code is not in u-boot yet. > > > > > > > > > > > > A38x u-boot ddr driver is in this directory: > > > > > > > > > > > https://source.denx.de/u-boot/u-boot/-/tree/master/drivers/ddr/marvell/a38x > > > > > > > > > > > > And it is copied from the original marvell driver by stripping non-a38x > > > > > > code and removal of the ddr4 code from the master branch: > > > > > > https://github.com/MarvellEmbeddedProcessors/mv-ddr-marvell > > > > > > > > > > > > So you can try to copy code again without stripping ddr4 parts > > > > > > (run git log drivers/ddr/marvell/a38x in u-boot) > > > > > > > > > > > > > > > https://source.denx.de/u-boot/u-boot/-/commit/107c3391b95bcc2ba09a876da4fa0c... > > > > > > > > > > > And adjust u-boot board code for ddr4. > > > > > > > > > > > > I guess it could work but it would be needed to play with it. > > > > > > > > > > > > > Last time I looked the MarvellEmbeddedProcessors stuff was a long way > > > > behind what Marvell are releasing to customers. That was for the newer SoCs > > > > so maybe the A385 stuff is current. > > > > > > About half year ago, MarvellEmbeddedProcessors mv-ddr-marvell and > > > A3700-utils-marvell were up-to-date and same as the version distributed > > > to the Marvell customers. I was cooperating with Marvell to release all > > > patches from Marvell Extranet portal to github as opensource and also to > > > incorporate github patches from community to their Extranet version. > > > Also I was synchronizing u-boot drivers/ddr/marvell/a38x directory with > > > MarvellEmbeddedProcessors version. I do not think that Marvell released > > > something super new in these repositories in last half of year, so I > > > think that the code on MarvellEmbeddedProcessors (for those two > > > repositories) is still up-to-date. > > > > Thanks all for commenting. As Pali has suggested, I will try copying > > the latest Marvell Github DDR drivers. > > I've tried to bring in the latest Marvell DDR drivers, but failed > miserably after many hours playing :) I used your DDR3 commit info as > a guide as how to strip out other parts, and only kept DDR3 and DDR4 > for a38x: > > https://source.denx.de/u-boot/u-boot/-/commit/107c3391b95bcc2ba09a876da4fa0c... > > The current code has DDR4 interspersed in DDR3 with if-defs > CONFIG_DDR4. And there are some minor dependencies on ATF types, > quite difficult for me to get it to build cleanly. I guess I threw in > the towel for now :) If you ever find some free time, please give it a > shot. > > Thanks, > Tony

Hello! And what is the issue? DDR training is failing? Or SPL does not boot? Or it do not compile?

Each of the code files was compiled (after some includes adjustments). But the build failed to link due to some errors such as /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr_plat.c:558: undefined reference to `mv_ddr_freq_get'

It should have been seen by the linker (since that code was compiled), I could not see why. I feel I have spent too much time fighting the if-defs :)

Thanks, Tony

Send the whole error message and also ideally push your changes to some git repository. So I can look at the stage at which you are.

OK thanks!

I got burned for being lazy :) it turned out that the mix of #ifdef and #if defined was the problem. Just stepped away and came back for a few minutes and I can see that I just need to define the CONFIG_DDR4 properly in arch/arm/mach-mvebu/Kconfig and build it to pass the CONFIG_DDR4 stuff.

One more small hurdle after that was CONFIG_USE_PRIVATE_LIBGCC must be undefined for it to build (so I am using /usr/lib/gcc/arm-linux-gnueabi/10/libgcc.a, and not using arch/arm/lib/lib.a)

Hello! It is normally a good idea to unset CONFIG_USE_PRIVATE_LIBGCC unless you have compiled gcc for target CPU.

CONFIG_USE_PRIVATE_LIBGCC was set as a default for ARM target, since u-boot has arch/arm/lib/lib.a. But using arch/arm/lib/lib.a I got several build errors like these:

ld.bfd: drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.o: in function `mv_ddr4_copt_get': /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_d2uiz'

Perhaps this is something that needs to be looked at.

...

But Marvell DDR4 is working fine! Please see the log.

Nice! So you can send patches for that board to list.

Sure I will for the N2350 board. How will we be handling the Marvell DDR code resync?

Thanks, Tony

...

<BEGIN LOG>

kwboot version 2023.01-tld-1-00048-g19e15f9081-dirty Patching image boot signature to UART Aligning image header to Xmodem block size Sending boot message. Please reboot the target.../ Sending boot image header (122112 bytes)... 0 % [......................................................................] 7 % [......................................................................] 14 % [......................................................................]

<snip> 80 % [......................................................................] 88 % [......................................................................] 95 % [............................................ ] Done

U-Boot SPL 2023.01-tld-1-10204-g7b84c973b9-dirty (Jan 10 2023 - 20:39:38 -0800) High speed PHY - Version: 2.0 Detected Device ID 6820 board SerDes lanes topology details: | Lane # | Speed | Type |

---

| 0 | 0 | SGMII0 | | 1 | 3 | SATA0 | | 2 | 3 | SATA1 | | 4 | 5 | USB3 HOST0 | | 5 | 5 | USB3 HOST1 |

---

High speed PHY - Ended Successfully DDR4 Training Sequence - Switching XBAR Window to FastPath Window mv_ddr: completed successfully Trying to boot from BOOTROM Returning to BootROM (return address 0xffff05c4)...

Sending boot image data (471968 bytes)... 0 % [......................................................................] 1 % [......................................................................]

<snip> 94 % [......................................................................] 96 % [......................................................................] 98 % [................................................ ] Done Finishing transfer [Type Ctrl-\ + c to quit]

U-Boot 2023.01-tld-1-10204-g7b84c973b9-dirty (Jan 10 2023 - 20:39:38 -0800) Thecus N2350

SoC: MV88F6820-A0 at 1066 MHz DRAM: 1 GiB (533 MHz, 32-bit, ECC not enabled) Core: 61 devices, 22 uclasses, devicetree: separate MMC: Loading Environment from SPIFlash... SF: Detected mx25l3205d with page size 256 Bytes, erase size 4 KiB, total 4 MiB *** Warning - bad CRC, using default environment

Model: Thecus N2350 Board: Thecus N2350 Net: Warning: ethernet@70000 (eth0) using random MAC address - d6:25:74:23:2c:21 eth0: ethernet@70000 Hit any key to stop autoboot: 0

<END LOG>

Thanks, Tony

...

Tony

...

...

...

> > > > > All the best, > > Tony > > > > > > > > > > > > > > > > So I'm building with binary.0 in the ./board/thecus/n2350/. This > > > > > > > binary.0 is a copy of the Marvell bin_hdr for SPI in the GPL source > > > > > > > for this board (provided by Thecus). My > > > > > > > ./board/thecus/n2350/kwbimage.cfg.in file looks like this > > > > > > > > > > > > > > # Armada 38x uses version 1 image format > > > > > > > VERSION 1 > > > > > > > # Boot Media configurations > > > > > > > BOOT_FROM spi > > > > > > > # Binary Header (bin_hdr) with DDR4 training code > > > > > > > BINARY board/thecus/n2350/binary.0 0000005b 00000068 > > > > > > > > > > > > > > When I kwboot the u-boot.kwb image (using kwboot binary built with > > > > > > > 2023.01-rc4), the header was transferred successfully, but then the > > > > > > > BootROM jumped to SPI u-boot, instead of loading the u-boot payload > > > > > > > from UART. Please see the log below. > > > > > > > <BEGIN LOG> > > > > > > > # ./kwboot -t -p -B 115200 /dev/ttyUSB0 -b uboot.kwb > > > > > > > > > > > > > > kwboot version 2023.01-tld-1-00048-g19e15f9081-dirty > > > > > > > Patching image boot signature to UART > > > > > > > Aligning image header to Xmodem block size > > > > > > > Sending boot message. Please reboot the target.../ > > > > > > > Sending boot image header (97792 bytes)... > > > > > > > 0 % > > > > > [......................................................................] > > > > > > > 9 % > > > > > [......................................................................] > > > > > > > <snip> > > > > > > > 82 % > > > > > [......................................................................] > > > > > > > 91 % > > > > > [................................................................ ] > > > > > > > Done > > > > > > > > > > > > > > BootROM - 1.73 > > > > > > > Booting from SPI flash > > > > > > > > > > > > This indicates reset of the board. BootROM started execution of the > > > > > > image from the primary location. > > > > > > > > > > > > > <snip> > > > > > > > U-Boot 2013.01 (Nov 12 2018 - 20:56:19) Marvell version: > > > > > 2015_T1.0p18-tld-4 > > > > > > > <END LOG> > > > > > > > > > > > > > > It seems kwboot patched the image, but the BOOT_FROM indicator was > > > > > > > still recognized as from SPI. So the BootROM loaded the stock u-boot > > > > > > > image from SPI and executed it. Since I am booting with bin_hdr, I'm > > > > > > > not sure if there is something inside this blob that has forced this > > > > > > > indicator to SPI. > > > > > > > > > > > > No, blob cannot force BootROM to switch boot location. This really > > > > > > indicates crash of the blob or crash of the payload which results in the > > > > > > board reset. > > > > > > > > > > > > > I'm attaching the u-boot.kwb image to this email, in case you are > > > > > > > interested in taking a look at the structure. > > > > > > > > > > > > > > Thanks, > > > > > > > Tony > > > > > > > > > > > > > > > > >

Hi Stefan,

On Wed, Jan 11, 2023 at 12:56 PM Pali Rohár pali@kernel.org wrote:

On Wednesday 11 January 2023 12:44:45 Tony Dinh wrote:

...

Hi Pali,

On Wed, Jan 11, 2023 at 12:04 AM Pali Rohár pali@kernel.org wrote:

...

On Tuesday 10 January 2023 21:07:45 Tony Dinh wrote:

...

Hi Pali,

I got burned for being lazy :) it turned out that the mix of #ifdef and #if defined was the problem. Just stepped away and came back for a few minutes and I can see that I just need to define the CONFIG_DDR4 properly in arch/arm/mach-mvebu/Kconfig and build it to pass the CONFIG_DDR4 stuff.

One more small hurdle after that was CONFIG_USE_PRIVATE_LIBGCC must be undefined for it to build (so I am using /usr/lib/gcc/arm-linux-gnueabi/10/libgcc.a, and not using arch/arm/lib/lib.a)

Hello! It is normally a good idea to unset CONFIG_USE_PRIVATE_LIBGCC unless you have compiled gcc for target CPU.

CONFIG_USE_PRIVATE_LIBGCC was set as a default for ARM target, since u-boot has arch/arm/lib/lib.a. But using arch/arm/lib/lib.a I got several build errors like these:

ld.bfd: drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.o: in function `mv_ddr4_copt_get': /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_d2uiz'

This looks like a bug in that training code. I would need to see source code, so I can figure out how to fix it.

...

Perhaps this is something that needs to be looked at.

...

...

But Marvell DDR4 is working fine! Please see the log.

Nice! So you can send patches for that board to list.

Sure I will for the N2350 board. How will we be handling the Marvell DDR code resync?

Send that DDR code resync in one patch... and saying e.g. adding support for DDR4 from Marvell repo.

The DDR4 patch is quite large, about 272K. Will it get rejected from ML because it is over the 100K limit?

# git format-patch -1 HEAD # wc 0001-ddr-marvell-a38x-Add-support-for-DDR4-from-Marvell-m.patch 7962 35309 277976 0001-ddr-marvell-a38x-Add-support-for-DDR4-from-Marvell-m.patch

How would you like to receive the patch? Would you like 2 emails (1st go to the ML without the patch, 2nd with the patch but not to the ML)?

Thanks, Tony

Hi Stefan,

On Thu, Jan 12, 2023 at 10:13 PM Stefan Roese sr@denx.de wrote:

Hi Tony,

(added Tom to Cc)

On 1/13/23 02:40, Tony Dinh wrote:

...

Hi Stefan,

On Wed, Jan 11, 2023 at 12:56 PM Pali Rohár pali@kernel.org wrote:

...

On Wednesday 11 January 2023 12:44:45 Tony Dinh wrote:

...

Hi Pali,

On Wed, Jan 11, 2023 at 12:04 AM Pali Rohár pali@kernel.org wrote:

...

On Tuesday 10 January 2023 21:07:45 Tony Dinh wrote:

...

Hi Pali,

I got burned for being lazy :) it turned out that the mix of #ifdef and #if defined was the problem. Just stepped away and came back for a few minutes and I can see that I just need to define the CONFIG_DDR4 properly in arch/arm/mach-mvebu/Kconfig and build it to pass the CONFIG_DDR4 stuff.

One more small hurdle after that was CONFIG_USE_PRIVATE_LIBGCC must be undefined for it to build (so I am using /usr/lib/gcc/arm-linux-gnueabi/10/libgcc.a, and not using arch/arm/lib/lib.a)

Hello! It is normally a good idea to unset CONFIG_USE_PRIVATE_LIBGCC unless you have compiled gcc for target CPU.

CONFIG_USE_PRIVATE_LIBGCC was set as a default for ARM target, since u-boot has arch/arm/lib/lib.a. But using arch/arm/lib/lib.a I got several build errors like these:

ld.bfd: drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.o: in function `mv_ddr4_copt_get': /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_d2uiz'

This looks like a bug in that training code. I would need to see source code, so I can figure out how to fix it.

...

Perhaps this is something that needs to be looked at.

...

...

But Marvell DDR4 is working fine! Please see the log.

Nice! So you can send patches for that board to list.

Sure I will for the N2350 board. How will we be handling the Marvell DDR code resync?

Send that DDR code resync in one patch... and saying e.g. adding support for DDR4 from Marvell repo.

The DDR4 patch is quite large, about 272K. Will it get rejected from ML because it is over the 100K limit?

Usually Tom as the mailing list maintainer will get informed about such mails and if he decides how to handle such huge mails. If this can't be handled differently, which is probably the case with such a sync to a repository, then the mail will get sent to the list when he "releases" such mails.

...

# git format-patch -1 HEAD # wc 0001-ddr-marvell-a38x-Add-support-for-DDR4-from-Marvell-m.patch 7962 35309 277976 0001-ddr-marvell-a38x-Add-support-for-DDR4-from-Marvell-m.patch

How would you like to receive the patch? Would you like 2 emails (1st go to the ML without the patch, 2nd with the patch but not to the ML)?

As a normal mail / patch please.

Will do that!

Side note: Does your patch /sync only touch the currently not supported DDR4 handling? If changes also include DDR3 then this should be tested on the boards currently using this code very intensive. All A38x board maintainers should be pointed to such new changes in this case.

Yes, indeed. I plan to do at least a DDR3 regression test with the Synology DS116 board (Armada 385, DDR3). The existing A38x boards do not have CONFIG_DDR4 enabled, so the DDR3 code looks to be the same, AFAITC. The patch should be reviewed by Pali and Marek to confirm that on paper, at least. But preferably, all A38x board maintainers should be aware. I will grep for the list of the A38x maintainers, and send directly to them (or if you already have such a list please let me know).

Thanks, Tony

Thanks, Stefan

Hi Pali,

On 1/13/23 09:31, Pali Rohár wrote:

<snip>

...

...

Side note: Does your patch /sync only touch the currently not supported DDR4 handling? If changes also include DDR3 then this should be tested on the boards currently using this code very intensive. All A38x board maintainers should be pointed to such new changes in this case.

I guess that new code only adds ifdefs for DDR4 which should not be enabled nor compiled for existing boards. This should be safe.

Best would be, if we could make sure that this patch does not change anything for non-DDR4 platforms. And if, then we should extract this into a separate patch for the DDR3 stuff, that can be more easily reviewed.

Thanks, Stefan

Hi Pali,

On Fri, Jan 13, 2023 at 5:32 PM Pali Rohár pali@kernel.org wrote:

On Wednesday 11 January 2023 21:56:41 Pali Rohár wrote:

...

On Wednesday 11 January 2023 12:44:45 Tony Dinh wrote:

...

Hi Pali,

On Wed, Jan 11, 2023 at 12:04 AM Pali Rohár pali@kernel.org wrote:

...

On Tuesday 10 January 2023 21:07:45 Tony Dinh wrote:

...

Hi Pali,

I got burned for being lazy :) it turned out that the mix of #ifdef and #if defined was the problem. Just stepped away and came back for a few minutes and I can see that I just need to define the CONFIG_DDR4 properly in arch/arm/mach-mvebu/Kconfig and build it to pass the CONFIG_DDR4 stuff.

One more small hurdle after that was CONFIG_USE_PRIVATE_LIBGCC must be undefined for it to build (so I am using /usr/lib/gcc/arm-linux-gnueabi/10/libgcc.a, and not using arch/arm/lib/lib.a)

Hello! It is normally a good idea to unset CONFIG_USE_PRIVATE_LIBGCC unless you have compiled gcc for target CPU.

CONFIG_USE_PRIVATE_LIBGCC was set as a default for ARM target, since u-boot has arch/arm/lib/lib.a. But using arch/arm/lib/lib.a I got several build errors like these:

ld.bfd: drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.o: in function `mv_ddr4_copt_get': /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_d2uiz'

This looks like a bug in that training code. I would need to see source code, so I can figure out how to fix it.

Have you solved this issue? Or if not, can you show what you had on that problematic line 809?

No, I have not and actually have no idea what that code does! And I thought you recognized the error, so I submitted the DDR4 patch so you can compile and see the error. Here is the code snipet, the error is now at 717.

# grep -n10 PBS_VALUE_FACTOR /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c 704- u8 copt_val; 705- u8 dq_idx; 706- u8 center_zone_max_low = 0; 707- u8 center_zone_min_high = 128; 708- u8 vw_zone_max_low = 0; 709- u8 vw_zone_min_high = 128; 710- u8 min_vw = 63; /* minimum valid window between all bits */ 711- u8 center_low_el; 712- u8 center_high_el; 713- 714: /* lambda calculated as D * PBS_VALUE_FACTOR / d */ 715- //printf("Copt::Debug::\t"); 716- for (dq_idx = 0; dq_idx < 8; dq_idx++) { 717- center_per_dq[dq_idx] = 0.5 * (vw_h[dq_idx] + vw_l[dq_idx]); 718- vw_per_dq[dq_idx] = 1 + (vw_h[dq_idx] - vw_l[dq_idx]); 719- if (min_vw > vw_per_dq[dq_idx]) 720- min_vw = vw_per_dq[dq_idx]; 721- } 722- 723- /* calculate center zone */ 724- for (dq_idx = 0; dq_idx < 8; dq_idx++) {

Here are the build errors with the current code.

<BEGIN ERRORS BUILD LOG > rm -f spl/common/spl/built-in.o; ar cDPrsT spl/common/spl/built-in.o spl/common/spl/spl.o spl/common/spl/spl_bootrom.o spl/common/spl/spl_spi.o ( cd spl && ld.bfd -T u-boot-spl.lds --gc-sections -Bstatic --gc-sections --no-dynamic-linker --build-id=none -Ttext 0x40000030 arch/arm/cpu/armv7/start.o --whole-archive arch/arm/mach-mvebu/built-in.o arch/arm/cpu/armv7/built-in.o arch/arm/cpu/built-in.o arch/arm/lib/built-in.o board/thecus/n2350/built-in.o common/spl/built-in.o common/init/built-in.o boot/built-in.o common/built-in.o cmd/built-in.o env/built-in.o lib/built-in.o disk/built-in.o drivers/built-in.o dts/built-in.o fs/built-in.o --no-whole-archive arch/arm/lib/eabi_compat.o arch/arm/lib/lib.a -Map u-boot-spl.map -o u-boot-spl ) ld.bfd: drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.o: in function `mv_ddr4_copt_get': /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:717: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:717: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:717: undefined reference to `__aeabi_d2uiz' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_dadd' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_dcmpgt' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:758: undefined reference to `__aeabi_dsub' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:758: undefined reference to `__aeabi_d2uiz' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:770: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:770: undefined reference to `__aeabi_dadd' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:770: undefined reference to `__aeabi_d2uiz' ld.bfd: drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.o: in function `mv_ddr4_tap_tuning': /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:1124: undefined reference to `__aeabi_ui2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:1124: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:1124: undefined reference to `__aeabi_dsub' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:1124: undefined reference to `__aeabi_ddiv' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:1124: undefined reference to `__aeabi_d2uiz' make[1]: *** [scripts/Makefile.spl:527: spl/u-boot-spl] Error 1 <END ERRORS BUILD LOG >

And the config option that's relevant here is CONFIG_USE_PRIVATE_LIBGCC (default is Y) which enabled the usage of arch/arm/lib/lib.a, and caused the build errors.

# grep GCC .config CONFIG_CC_IS_GCC=y CONFIG_GCC_VERSION=100201 CONFIG_HAVE_PRIVATE_LIBGCC=y CONFIG_USE_PRIVATE_LIBGCC=y

When I unset CONFIG_USE_PRIVATE_LIBGCC, the GGC library used is /usr/lib/gcc/arm-linux-gnueabi/10 and everything built.

# grep CONFIG_USE_PRIVATE_LIBGCC .config # CONFIG_USE_PRIVATE_LIBGCC is not set

<BEGIN GOOD BUILD LOG> rm -f spl/common/spl/built-in.o; ar cDPrsT spl/common/spl/built-in.o spl/common/spl/spl.o spl/common/spl/spl_bootrom.o spl/common/spl/spl_spi.o ( cd spl && ld.bfd -T u-boot-spl.lds --gc-sections -Bstatic --gc-sections --no-dynamic-linker --build-id=none -Ttext 0x40000030 arch/arm/cpu/armv7/start.o --whole-archive arch/arm/mach-mvebu/built-in.o arch/arm/cpu/armv7/built-in.o arch/arm/cpu/built-in.o arch/arm/lib/built-in.o board/thecus/n2350/built-in.o common/spl/built-in.o common/init/built-in.o boot/built-in.o common/built-in.o cmd/built-in.o env/built-in.o lib/built-in.o disk/built-in.o drivers/built-in.o dts/built-in.o fs/built-in.o --no-whole-archive arch/arm/lib/eabi_compat.o -L /usr/lib/gcc/arm-linux-gnueabi/10 -lgcc -Map u-boot-spl.map -o u-boot-spl ) ld.bfd: warning: /usr/lib/gcc/arm-linux-gnueabi/10/libgcc.a(_udivmoddi4.o) uses 4-byte wchar_t yet the output is to use 2-byte wchar_t; use of wchar_t values across objects may fail objcopy -j .text -j .secure_text -j .secure_data -j .rodata -j .hash -j .data -j .got -j .got.plt -j __u_boot_list -j .rel.dyn -j .binman_sym_table -j .text_rest -j .dtb.init.rodata -j .efi_runtime -j .efi_runtime_rel -O binary spl/u-boot-spl spl/u-boot-spl-nodtb.bin objdump -t spl/u-boot-spl > spl/u-boot-spl.sym cat spl/u-boot-spl-nodtb.bin spl/u-boot-spl-pad.bin spl/u-boot-spl.dtb > spl/u-boot-spl-dtb.bin cp spl/u-boot-spl-dtb.bin spl/u-boot-spl.bin ./tools/mkimage -n ./arch/arm/mach-mvebu/kwbimage.cfg -T kwbimage -a 0x00800000 -e 0x00800000 -d u-boot.bin u-boot-with-spl.kwb >/dev/null && cat /dev/null ./scripts/check-of.sh .config ./scripts/of_allowlist.txt make: Leaving directory '/usr/src/u-boot-master' + /bin/ls -lh u-boot-with-spl.kwb -rw-r--r-- 1 root root 581K Jan 14 13:12 u-boot-with-spl.kwb <END LOG>

I can also send you the whole log if you want. I'm preparing the Thecus N2350 patch and will send it within a few days.

Thanks, Tony

Hi Pali,

On Sat, Jan 14, 2023 at 1:44 PM Pali Rohár pali@kernel.org wrote:

On Saturday 14 January 2023 13:35:08 Tony Dinh wrote:

...

Hi Pali,

On Fri, Jan 13, 2023 at 5:32 PM Pali Rohár pali@kernel.org wrote:

...

On Wednesday 11 January 2023 21:56:41 Pali Rohár wrote:

...

On Wednesday 11 January 2023 12:44:45 Tony Dinh wrote:

...

Hi Pali,

On Wed, Jan 11, 2023 at 12:04 AM Pali Rohár pali@kernel.org wrote:

...

On Tuesday 10 January 2023 21:07:45 Tony Dinh wrote: > Hi Pali, > > I got burned for being lazy :) it turned out that the mix of #ifdef > and #if defined was the problem. Just stepped away and came back for a > few minutes and I can see that I just need to define the CONFIG_DDR4 > properly in arch/arm/mach-mvebu/Kconfig and build it to pass the > CONFIG_DDR4 stuff. > > One more small hurdle after that was CONFIG_USE_PRIVATE_LIBGCC must be > undefined for it to build (so I am using > /usr/lib/gcc/arm-linux-gnueabi/10/libgcc.a, and not using > arch/arm/lib/lib.a)

Hello! It is normally a good idea to unset CONFIG_USE_PRIVATE_LIBGCC unless you have compiled gcc for target CPU.

CONFIG_USE_PRIVATE_LIBGCC was set as a default for ARM target, since u-boot has arch/arm/lib/lib.a. But using arch/arm/lib/lib.a I got several build errors like these:

ld.bfd: drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.o: in function `mv_ddr4_copt_get': /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:809: undefined reference to `__aeabi_d2uiz'

This looks like a bug in that training code. I would need to see source code, so I can figure out how to fix it.

Have you solved this issue? Or if not, can you show what you had on that problematic line 809?

No, I have not and actually have no idea what that code does! And I thought you recognized the error, so I submitted the DDR4 patch so you can compile and see the error. Here is the code snipet, the error is now at 717.

# grep -n10 PBS_VALUE_FACTOR /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c 704- u8 copt_val; 705- u8 dq_idx; 706- u8 center_zone_max_low = 0; 707- u8 center_zone_min_high = 128; 708- u8 vw_zone_max_low = 0; 709- u8 vw_zone_min_high = 128; 710- u8 min_vw = 63; /* minimum valid window between all bits */ 711- u8 center_low_el; 712- u8 center_high_el; 713- 714: /* lambda calculated as D * PBS_VALUE_FACTOR / d */ 715- //printf("Copt::Debug::\t"); 716- for (dq_idx = 0; dq_idx < 8; dq_idx++) { 717- center_per_dq[dq_idx] = 0.5 * (vw_h[dq_idx] + vw_l[dq_idx]); 718- vw_per_dq[dq_idx] = 1 + (vw_h[dq_idx] - vw_l[dq_idx]); 719- if (min_vw > vw_per_dq[dq_idx]) 720- min_vw = vw_per_dq[dq_idx]; 721- } 722- 723- /* calculate center zone */ 724- for (dq_idx = 0; dq_idx < 8; dq_idx++) {

Haha! That is pretty simple. On line 717 you have fractional number 0.5 which is represented by floating point and all numeric operation on that line are done as floating point.

U-Boot and kernel does not floating point HW and instruct compiler to replace all floating point operations by function calls (which implement software floating point support).

U-Boot (for very good reasons) do not implement any floating point operations.

Fix should be very simple, use only integer operations. So replace

0.5 * something

by:

something / 2

And check if it compiles and if it works.

I'm surprised that Marvell was trying to do floating point...

Yes!!! that was impressive how quick you saw the root cause. I've replaced those floating point operations with integer operations and it built fine. Let see it will run.

Thanks, Tony

...

Here are the build errors with the current code.

<BEGIN ERRORS BUILD LOG > rm -f spl/common/spl/built-in.o; ar cDPrsT spl/common/spl/built-in.o spl/common/spl/spl.o spl/common/spl/spl_bootrom.o spl/common/spl/spl_spi.o ( cd spl && ld.bfd -T u-boot-spl.lds --gc-sections -Bstatic --gc-sections --no-dynamic-linker --build-id=none -Ttext 0x40000030 arch/arm/cpu/armv7/start.o --whole-archive arch/arm/mach-mvebu/built-in.o arch/arm/cpu/armv7/built-in.o arch/arm/cpu/built-in.o arch/arm/lib/built-in.o board/thecus/n2350/built-in.o common/spl/built-in.o common/init/built-in.o boot/built-in.o common/built-in.o cmd/built-in.o env/built-in.o lib/built-in.o disk/built-in.o drivers/built-in.o dts/built-in.o fs/built-in.o --no-whole-archive arch/arm/lib/eabi_compat.o arch/arm/lib/lib.a -Map u-boot-spl.map -o u-boot-spl ) ld.bfd: drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.o: in function `mv_ddr4_copt_get': /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:717: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:717: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:717: undefined reference to `__aeabi_d2uiz' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_dadd' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:757: undefined reference to `__aeabi_dcmpgt' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:758: undefined reference to `__aeabi_dsub' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:758: undefined reference to `__aeabi_d2uiz' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:770: undefined reference to `__aeabi_i2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:770: undefined reference to `__aeabi_dadd' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:770: undefined reference to `__aeabi_d2uiz' ld.bfd: drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.o: in function `mv_ddr4_tap_tuning': /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:1124: undefined reference to `__aeabi_ui2d' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:1124: undefined reference to `__aeabi_dmul' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:1124: undefined reference to `__aeabi_dsub' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:1124: undefined reference to `__aeabi_ddiv' ld.bfd: /usr/src/u-boot-master/drivers/ddr/marvell/a38x/mv_ddr4_training_calibration.c:1124: undefined reference to `__aeabi_d2uiz' make[1]: *** [scripts/Makefile.spl:527: spl/u-boot-spl] Error 1 <END ERRORS BUILD LOG >

And the config option that's relevant here is CONFIG_USE_PRIVATE_LIBGCC (default is Y) which enabled the usage of arch/arm/lib/lib.a, and caused the build errors.

# grep GCC .config CONFIG_CC_IS_GCC=y CONFIG_GCC_VERSION=100201 CONFIG_HAVE_PRIVATE_LIBGCC=y CONFIG_USE_PRIVATE_LIBGCC=y

When I unset CONFIG_USE_PRIVATE_LIBGCC, the GGC library used is /usr/lib/gcc/arm-linux-gnueabi/10 and everything built.

# grep CONFIG_USE_PRIVATE_LIBGCC .config # CONFIG_USE_PRIVATE_LIBGCC is not set

<BEGIN GOOD BUILD LOG> rm -f spl/common/spl/built-in.o; ar cDPrsT spl/common/spl/built-in.o spl/common/spl/spl.o spl/common/spl/spl_bootrom.o spl/common/spl/spl_spi.o ( cd spl && ld.bfd -T u-boot-spl.lds --gc-sections -Bstatic --gc-sections --no-dynamic-linker --build-id=none -Ttext 0x40000030 arch/arm/cpu/armv7/start.o --whole-archive arch/arm/mach-mvebu/built-in.o arch/arm/cpu/armv7/built-in.o arch/arm/cpu/built-in.o arch/arm/lib/built-in.o board/thecus/n2350/built-in.o common/spl/built-in.o common/init/built-in.o boot/built-in.o common/built-in.o cmd/built-in.o env/built-in.o lib/built-in.o disk/built-in.o drivers/built-in.o dts/built-in.o fs/built-in.o --no-whole-archive arch/arm/lib/eabi_compat.o -L /usr/lib/gcc/arm-linux-gnueabi/10 -lgcc -Map u-boot-spl.map -o u-boot-spl ) ld.bfd: warning: /usr/lib/gcc/arm-linux-gnueabi/10/libgcc.a(_udivmoddi4.o) uses 4-byte wchar_t yet the output is to use 2-byte wchar_t; use of wchar_t values across objects may fail objcopy -j .text -j .secure_text -j .secure_data -j .rodata -j .hash -j .data -j .got -j .got.plt -j __u_boot_list -j .rel.dyn -j .binman_sym_table -j .text_rest -j .dtb.init.rodata -j .efi_runtime -j .efi_runtime_rel -O binary spl/u-boot-spl spl/u-boot-spl-nodtb.bin objdump -t spl/u-boot-spl > spl/u-boot-spl.sym cat spl/u-boot-spl-nodtb.bin spl/u-boot-spl-pad.bin spl/u-boot-spl.dtb > spl/u-boot-spl-dtb.bin cp spl/u-boot-spl-dtb.bin spl/u-boot-spl.bin ./tools/mkimage -n ./arch/arm/mach-mvebu/kwbimage.cfg -T kwbimage -a 0x00800000 -e 0x00800000 -d u-boot.bin u-boot-with-spl.kwb >/dev/null && cat /dev/null ./scripts/check-of.sh .config ./scripts/of_allowlist.txt make: Leaving directory '/usr/src/u-boot-master' + /bin/ls -lh u-boot-with-spl.kwb -rw-r--r-- 1 root root 581K Jan 14 13:12 u-boot-with-spl.kwb <END LOG>

I can also send you the whole log if you want. I'm preparing the Thecus N2350 patch and will send it within a few days.

Thanks, Tony