[U-Boot] PCI-E problems on Kilauea board
Hi,
I have several PCI-E related problems on Kilauea board Rev 1.2
1. With u-boot v2009.01 and latest git I'm getting machine check exception when PCI-E card is plugged into slot 0. There's no problem if the same card is plugged into slot 1.
This seems to be a regression introduced after 2008.10-rc2-02699-g725c8dd (version shipped on AMCC resource CD), as with this version both slots function properly.
Any hints what could be the problem with latest u-boots ?
U-Boot 2009.01-00336-g952a6bd (Feb 16 2009 - 16:29:56)
CPU: AMCC PowerPC 405EX Rev. C at 600 MHz (PLB=200, OPB=100, EBC=100 MHz) Security support Bootstrap Option H - Boot ROM Location I2C (Addr 0x52) 16 kB I-Cache 16 kB D-Cache Board: Kilauea - AMCC PPC405EX Evaluation Board I2C: ready DTT1: 31 C DRAM: 256 MB FLASH: 64 MB NAND: 64 MiB PCI: Bus Dev VenId DevId Class Int PCIE0: successfully set as root-complex Machine Check Exception. Caused by (from msr): regs 0fe9fc78 Data PLB Error NIP: 00000000 XER: 00000000 LR: 0FFD8794 REGS: 0fe9fc78 TRAP: 0200 DEAR: 00000000 MSR: 00000000 EE: 0 PR: 0 FP: 0 ME: 0 IR/DR: 00
GPR00: 00000000 0FE9FD68 0FE9FF44 B0000000 00000000 0FE9FD58 00000003 00100004 GPR08: 00000000 A0000000 0FE9FD58 0000003A 07FFFFFC 7FF67EFF 0FFF1500 10004000 GPR16: 74FEFFE7 0FFE000C 0FE9FE28 0FE9FE2A 0FE9FDB0 0FE9FDB4 0FE9FDBC 0FFF0CC0 GPR24: 00000000 00000018 00000004 00000003 00100004 0FFF0CA8 0FFF1C10 00000004 Call backtrace: 0FFF0CA8 0FFD86F8 0FFB4B28 0FFB6058 0FFB6118 0FFB5330 0FFB5404 0FFD2558 0FFD84A0 0FFB51C4 0FFA7F10 0FFA66A4 machine check
2. Another problem is that in bootstrap configuration B (CPU - 333 Mhz, PLB 166MHz) PCI-E cards are not recognized in any slot. This happens with all u-boot versions I've tested. Linux also does not recognize PCI-E cards. Kilauea board manual says that PCI clock is determined by CPLD. Could it be CPLD/FPGA bug ?
Thanks.
Felix.
Hi Felix,
On Monday 16 February 2009, Felix Radensky wrote:
I have several PCI-E related problems on Kilauea board Rev 1.2
With u-boot v2009.01 and latest git I'm getting machine check exception when PCI-E card is plugged into slot 0. There's no problem if the same card is plugged into slot 1.
This seems to be a regression introduced after
2008.10-rc2-02699-g725c8dd (version shipped on AMCC resource CD), as with this version both slots function properly.
Any hints what could be the problem with latest u-boots ?
I just checked on my Kilauea and reproduced the problem. No, I don't have any clue right now where this problem is generated. I can't remember any issues with PCIe slot 1 on Kilauea.
Perhaps you could try to find the git commit introducing this issue by using "git bisect"?
<snip>
- Another problem is that in bootstrap configuration B (CPU - 333 Mhz,
PLB 166MHz) PCI-E cards are not recognized in any slot. This happens with all u-boot versions I've tested. Linux also does not recognize PCI-E cards. Kilauea board manual says that PCI clock is determined by CPLD. Could it be CPLD/FPGA bug ?
Is bootstrap option C working?
Thanks.
Best regards, Stefan
===================================================================== DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: +49-8142-66989-0 Fax: +49-8142-66989-80 Email: office@denx.de =====================================================================
Stefan Roese wrote:
Hi Felix,
On Monday 16 February 2009, Felix Radensky wrote:
I have several PCI-E related problems on Kilauea board Rev 1.2
With u-boot v2009.01 and latest git I'm getting machine check exception when PCI-E card is plugged into slot 0. There's no problem if the same card is plugged into slot 1.
This seems to be a regression introduced after
2008.10-rc2-02699-g725c8dd (version shipped on AMCC resource CD), as with this version both slots function properly.
Any hints what could be the problem with latest u-boots ?I just checked on my Kilauea and reproduced the problem. No, I don't have any clue right now where this problem is generated. I can't remember any issues with PCIe slot 1 on Kilauea.
Perhaps you could try to find the git commit introducing this issue by using "git bisect"?
Ok, I'll try git bisect.
<snip>
- Another problem is that in bootstrap configuration B (CPU - 333 Mhz,
PLB 166MHz) PCI-E cards are not recognized in any slot. This happens with all u-boot versions I've tested. Linux also does not recognize PCI-E cards. Kilauea board manual says that PCI clock is determined by CPLD. Could it be CPLD/FPGA bug ?
Is bootstrap option C working?
No, C doesn't work either. The card I'm testing with is Intel Gigabit Ethernet adapter (PRO/1000 PT 1x)
Thanks.
Best regards, Stefan
===================================================================== DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: +49-8142-66989-0 Fax: +49-8142-66989-80 Email: office@denx.de =====================================================================
On Tuesday 17 February 2009, Felix Radensky wrote:
- Another problem is that in bootstrap configuration B (CPU - 333 Mhz,
PLB 166MHz) PCI-E cards are not recognized in any slot. This happens with all u-boot versions I've tested. Linux also does not recognize PCI-E cards. Kilauea board manual says that PCI clock is determined by CPLD. Could it be CPLD/FPGA bug ?
Is bootstrap option C working?
No, C doesn't work either. The card I'm testing with is Intel Gigabit Ethernet adapter (PRO/1000 PT 1x)
Perhaps its only working with the I2C bootstrap settings. Could be that one (or more) register values are configured via the I2C bootstrap EEPROM compared to the hardwired bootstrap option. Please check the users manual and compare the register settings of I2C bootstrap option H to option B or C.
Best regards, Stefan
===================================================================== DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: +49-8142-66989-0 Fax: +49-8142-66989-80 Email: office@denx.de =====================================================================
Stefan Roese wrote:
On Tuesday 17 February 2009, Felix Radensky wrote:
- Another problem is that in bootstrap configuration B (CPU - 333 Mhz,
PLB 166MHz) PCI-E cards are not recognized in any slot. This happens with all u-boot versions I've tested. Linux also does not recognize PCI-E cards. Kilauea board manual says that PCI clock is determined by CPLD. Could it be CPLD/FPGA bug ?
Is bootstrap option C working?
No, C doesn't work either. The card I'm testing with is Intel Gigabit Ethernet adapter (PRO/1000 PT 1x)
Perhaps its only working with the I2C bootstrap settings. Could be that one (or more) register values are configured via the I2C bootstrap EEPROM compared to the hardwired bootstrap option. Please check the users manual and compare the register settings of I2C bootstrap option H to option B or C.
I did some more tests, and have found the following:
1. When system powers up in bootstrap option B or C u-boot does not detect PCI-E device PCI: Bus Dev VenId DevId Class Int PCIE0: link is not up. PCIE0: initialization as root-complex failed PCIE1: link is not up. PCIE1: initialization as root-complex failed
2. Linux detects PCI-E device, but does not detect the link: PCIE1: Checking link... PCIE1: Device detected, waiting for link... PCIE1: Link up failed
3. If I execute reboot from linux, u-boot now detects the device PCI: Bus Dev VenId DevId Class Int PCIE0: link is not up. PCIE0: initialization as root-complex failed PCIE1: successfully set as root-complex 01 00 8086 10b9 0200 00
4. After that Linux detects both device and link PCIE1: Checking link... PCIE1: Device detected, waiting for link... PCIE1: link is up !
-bash-3.2# lspci 0000:00:00.0 PCI bridge: Unknown device aaa0:bed0 (rev 01) 0001:40:00.0 PCI bridge: Unknown device aaa1:bed1 (rev 01) 0001:41:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Also, I've compared the values of SDR0_SDSTP0-SDR0_SDSTP3 registers in bootstrap options C and H and didn't find anything suspicious. There are differences in various divisor values, but that is expected. Tested with u-boot 2009.03-rc1 and linux-2.6.29-rc5
Any other ideas ?
Thanks a lot.
Felix.
Felix Radensky wrote:
Stefan Roese wrote:
On Tuesday 17 February 2009, Felix Radensky wrote:
- Another problem is that in bootstrap configuration B (CPU - 333 Mhz,
PLB 166MHz) PCI-E cards are not recognized in any slot. This happens with all u-boot versions I've tested. Linux also does not recognize PCI-E cards. Kilauea board manual says that PCI clock is determined by CPLD. Could it be CPLD/FPGA bug ?
Is bootstrap option C working?
No, C doesn't work either. The card I'm testing with is Intel Gigabit Ethernet adapter (PRO/1000 PT 1x)
Perhaps its only working with the I2C bootstrap settings. Could be that one (or more) register values are configured via the I2C bootstrap EEPROM compared to the hardwired bootstrap option. Please check the users manual and compare the register settings of I2C bootstrap option H to option B or C.
I did some more tests, and have found the following:
1. When system powers up in bootstrap option B or C u-boot does notdetect PCI-E device PCI: Bus Dev VenId DevId Class Int PCIE0: link is not up. PCIE0: initialization as root-complex failed PCIE1: link is not up. PCIE1: initialization as root-complex failed
2. Linux detects PCI-E device, but does not detect the link: PCIE1: Checking link... PCIE1: Device detected, waiting for link... PCIE1: Link up failed 3. If I execute reboot from linux, u-boot now detects the device PCI: Bus Dev VenId DevId Class Int PCIE0: link is not up. PCIE0: initialization as root-complex failed PCIE1: successfully set as root-complex 01 00 8086 10b9 0200 00 4. After that Linux detects both device and link PCIE1: Checking link... PCIE1: Device detected, waiting for link... PCIE1: link is up ! -bash-3.2# lspci 0000:00:00.0 PCI bridge: Unknown device aaa0:bed0 (rev 01) 0001:40:00.0 PCI bridge: Unknown device aaa1:bed1 (rev 01) 0001:41:00.0 Ethernet controller: Intel Corporation 82572EIGigabit Ethernet Controller (Copper) (rev 06)
Also, I've compared the values of SDR0_SDSTP0-SDR0_SDSTP3 registers in bootstrap options C and H and didn't find anything suspicious. There are differences in various divisor values, but that is expected. Tested with u-boot 2009.03-rc1 and linux-2.6.29-rc5
It is even more simple. It's enough to power up the board and run "reset" in u-boot, to make u-boot recognize the card when it comes up.
U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
On Monday 02 March 2009, Felix Radensky wrote:
Also, I've compared the values of SDR0_SDSTP0-SDR0_SDSTP3 registers in bootstrap options C and H and didn't find anything suspicious. There are differences in various divisor values, but that is expected. Tested with u-boot 2009.03-rc1 and linux-2.6.29-rc5
It is even more simple. It's enough to power up the board and run "reset" in u-boot, to make u-boot recognize the card when it comes up.
Did you already try setting the "pciscandelay" env variable? Set it to 5 for 5 seconds delay before PnP scanning.
Does this help?
Best regards, Stefan
===================================================================== DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: +49-8142-66989-0 Fax: +49-8142-66989-80 Email: office@denx.de =====================================================================
Stefan Roese wrote:
On Monday 02 March 2009, Felix Radensky wrote:
Also, I've compared the values of SDR0_SDSTP0-SDR0_SDSTP3 registers in bootstrap options C and H and didn't find anything suspicious. There are differences in various divisor values, but that is expected. Tested with u-boot 2009.03-rc1 and linux-2.6.29-rc5
It is even more simple. It's enough to power up the board and run "reset" in u-boot, to make u-boot recognize the card when it comes up.
Did you already try setting the "pciscandelay" env variable? Set it to 5 for 5 seconds delay before PnP scanning.
Does this help?
pciscandelay is taken into account only if initialization of root complex succeeds, which is not the case. So it doesn't help.
Felix.
On Monday 02 March 2009, Felix Radensky wrote:
Did you already try setting the "pciscandelay" env variable? Set it to 5 for 5 seconds delay before PnP scanning.
Does this help?
pciscandelay is taken into account only if initialization of root complex succeeds, which is not the case. So it doesn't help.
OK.
I just tried this setup on my Kilauea board. Without any problems:
U-Boot 2009.03-rc1-00050-g3aaf315 (Mar 11 2009 - 13:18:35)
CPU: AMCC PowerPC 405EX Rev. C at 400 MHz (PLB=200, OPB=100, EBC=100 MHz) Security support Bootstrap Option C - Boot ROM Location EBC (16 bits) 16 kB I-Cache 16 kB D-Cache Board: Kilauea - AMCC PPC405EX Evaluation Board I2C: ready DRAM: 256 MB FLASH: 64 MB NAND: 64 MiB PCI: Bus Dev VenId DevId Class Int PCIE0: link is not up. PCIE0: initialization as root-complex failed PCIE1: successfully set as root-complex 01 00 8086 10b9 0200 00 DTT1: 30 C Net: ppc_4xx_eth0, ppc_4xx_eth1
Type run flash_nfs to mount root filesystem over NFS
Hit any key to stop autoboot: 4
U-Boot 2009.03-rc1-00050-g3aaf315 (Mar 11 2009 - 13:18:35)
CPU: AMCC PowerPC 405EX Rev. C at 400 MHz (PLB=200, OPB=100, EBC=100 MHz) Security support Bootstrap Option C - Boot ROM Location EBC (16 bits) 16 kB I-Cache 16 kB D-Cache Board: Kilauea - AMCC PPC405EX Evaluation Board I2C: ready DRAM: 256 MB FLASH: 64 MB NAND: 64 MiB PCI: Bus Dev VenId DevId Class Int PCIE0: successfully set as root-complex 01 00 8086 10b9 0200 00 PCIE1: link is not up. PCIE1: initialization as root-complex failed DTT1: 30 C Net: ppc_4xx_eth0, ppc_4xx_eth1
Type run flash_nfs to mount root filesystem over NFS
Hit any key to stop autoboot: 0 =>
The PCIe device used here is the Intel PRO/1000 PT Desktop card.
Which Kilauea board revision are you using? And do you see the same problems with other PCIe cards as well?
Best regards, Stefan
===================================================================== DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: +49-8142-66989-0 Fax: +49-8142-66989-80 Email: office@denx.de =====================================================================
Stefan Roese wrote:
OK.
I just tried this setup on my Kilauea board. Without any problems:
U-Boot 2009.03-rc1-00050-g3aaf315 (Mar 11 2009 - 13:18:35)
CPU: AMCC PowerPC 405EX Rev. C at 400 MHz (PLB=200, OPB=100, EBC=100 MHz) Security support Bootstrap Option C - Boot ROM Location EBC (16 bits) 16 kB I-Cache 16 kB D-Cache Board: Kilauea - AMCC PPC405EX Evaluation Board I2C: ready DRAM: 256 MB FLASH: 64 MB NAND: 64 MiB PCI: Bus Dev VenId DevId Class Int PCIE0: link is not up. PCIE0: initialization as root-complex failed PCIE1: successfully set as root-complex 01 00 8086 10b9 0200 00 DTT1: 30 C Net: ppc_4xx_eth0, ppc_4xx_eth1
Type run flash_nfs to mount root filesystem over NFS
Hit any key to stop autoboot: 4
U-Boot 2009.03-rc1-00050-g3aaf315 (Mar 11 2009 - 13:18:35)
CPU: AMCC PowerPC 405EX Rev. C at 400 MHz (PLB=200, OPB=100, EBC=100 MHz) Security support Bootstrap Option C - Boot ROM Location EBC (16 bits) 16 kB I-Cache 16 kB D-Cache Board: Kilauea - AMCC PPC405EX Evaluation Board I2C: ready DRAM: 256 MB FLASH: 64 MB NAND: 64 MiB PCI: Bus Dev VenId DevId Class Int PCIE0: successfully set as root-complex 01 00 8086 10b9 0200 00 PCIE1: link is not up. PCIE1: initialization as root-complex failed DTT1: 30 C Net: ppc_4xx_eth0, ppc_4xx_eth1
Type run flash_nfs to mount root filesystem over NFS
Hit any key to stop autoboot: 0 =>
The PCIe device used here is the Intel PRO/1000 PT Desktop card.
Which Kilauea board revision are you using? And do you see the same problems with other PCIe cards as well?
I use Kilauea revision 1.2 and exactly the same network card as you. The problem is still here. It doesn't happen with other PCI-E card I have (10G Ethernet adapter).
I have a feeling that 2 Kilauea boards of the same revision may sometimes exhibit different behavior with the same version of u-boot. For example, with u-boot 2008.10-rc2 that comes with AMCC CD, memory was properly detected on one board in bootstrap option B but not detected on the another. I had to recompile u-boot and disable autocalibration to make things work on this board.
Felix.
On Wednesday 11 March 2009, Felix Radensky wrote:
The PCIe device used here is the Intel PRO/1000 PT Desktop card.
Which Kilauea board revision are you using? And do you see the same problems with other PCIe cards as well?
I use Kilauea revision 1.2 and exactly the same network card as you.
I'm using the same board revision too.
The problem is still here.
Strange.
It doesn't happen with other PCI-E card I have (10G Ethernet adapter).
I have a feeling that 2 Kilauea boards of the same revision may sometimes exhibit different behavior with the same version of u-boot.
This could be the case. Do you have another Kilauea board?
For example, with u-boot 2008.10-rc2 that comes with AMCC CD, memory was properly detected on one board in bootstrap option B but not detected on the another. I had to recompile u-boot and disable autocalibration to make things work on this board.
This is a different issue. The DDR2 autocalibration had a bug. Because of this some boards could boot just fine with autocalibration enabled and some not. This bug should be fixed now. In the latest U-Boot version autocalibration is enabled again for Kilauea and seems to work now. At least I didn't notice any problems anymore. Please give it a try and let me know if this doesn't work for you.
But again, this is a different issue. Not sure how to help you with the PCIe detection problem.
Best regards, Stefan
===================================================================== DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: +49-8142-66989-0 Fax: +49-8142-66989-80 Email: office@denx.de =====================================================================
Stefan Roese wrote:
On Wednesday 11 March 2009, Felix Radensky wrote:
The PCIe device used here is the Intel PRO/1000 PT Desktop card.
Which Kilauea board revision are you using? And do you see the same problems with other PCIe cards as well?
I use Kilauea revision 1.2 and exactly the same network card as you.
I'm using the same board revision too.
The problem is still here.
Strange.
It doesn't happen with other PCI-E card I have (10G Ethernet adapter).
I have a feeling that 2 Kilauea boards of the same revision may sometimes exhibit different behavior with the same version of u-boot.
This could be the case. Do you have another Kilauea board?
My customer has another board, which had issues with DDR2 autocalibration.
For example, with u-boot 2008.10-rc2 that comes with AMCC CD, memory was properly detected on one board in bootstrap option B but not detected on the another. I had to recompile u-boot and disable autocalibration to make things work on this board.
This is a different issue. The DDR2 autocalibration had a bug. Because of this some boards could boot just fine with autocalibration enabled and some not. This bug should be fixed now. In the latest U-Boot version autocalibration is enabled again for Kilauea and seems to work now. At least I didn't notice any problems anymore. Please give it a try and let me know if this doesn't work for you.
I have tried latest u-boot on Kilauea and on custom board based on 405EXr and autocalibration seems to work fine.
But again, this is a different issue. Not sure how to help you with the PCIe detection problem.
I've used Kilauea as a testbed before the custom board arrived. On this board PCI-E device is properly detected, so PCI-E detection on Kilauea is not critical to me now. I appreciate your efforts and advice very much. In case you have some ideas I can test them on my Kilauea.
Thank you very much for your help.
Felix.
Stefan Roese wrote:
Hi Felix,
On Monday 16 February 2009, Felix Radensky wrote:
I have several PCI-E related problems on Kilauea board Rev 1.2
With u-boot v2009.01 and latest git I'm getting machine check exception when PCI-E card is plugged into slot 0. There's no problem if the same card is plugged into slot 1.
This seems to be a regression introduced after
2008.10-rc2-02699-g725c8dd (version shipped on AMCC resource CD), as with this version both slots function properly.
Any hints what could be the problem with latest u-boots ?I just checked on my Kilauea and reproduced the problem. No, I don't have any clue right now where this problem is generated. I can't remember any issues with PCIe slot 1 on Kilauea.
Perhaps you could try to find the git commit introducing this issue by using "git bisect"?
<snip>
According to "git bisect" the problem was introduced by this commit:
30e76d5e3bc4c5208ee63585fe12b409d9308cd8 is first bad commit commit 30e76d5e3bc4c5208ee63585fe12b409d9308cd8 Author: Kumar Gala galak@kernel.crashing.org Date: Tue Oct 21 08:36:08 2008 -0500
pci: Allow for PCI addresses to be 64-bit
PCI bus is inherently 64-bit. While not all system require access to the full 64-bit PCI address range some do. This allows those systems to enable the full PCI address width via CONFIG_SYS_PCI_64BIT.
Felix.
participants (2)
-
Felix Radensky -
Stefan Roese