[U-Boot] [PATCH 0/2] vexpress64: Juno: Add support for PCIe on Juno

Hello,
This patchset enables PCIe for ARM's Juno boards and configures the host bridge's address translation block. This enables the Linux kernel to boot on Juno r1 using just a device tree and the generic host bridge driver.
No support has been added at this phase for the SATA or Ethernet devices present on the PCIe bus.
Liviu Dudau (2): vexpress64: Juno: Declare all 8GB of RAM and make them visible to the kernel. vexpress64: Juno: Add initialisation code for Juno R1 PCIe host bridge.
board/armltd/vexpress64/vexpress64.c | 128 +++++++++++++++++++++++++++++++++++ board/armltd/vexpress64/vexpress64.h | 60 ++++++++++++++++ include/configs/vexpress_aemv8a.h | 4 +- 3 files changed, 191 insertions(+), 1 deletion(-) create mode 100644 board/armltd/vexpress64/vexpress64.h

Juno comes with 8GB RAM, but U-Boot only passes 2GB to the kernel. Declare a secondary memory bank and set the sizes correctly.
Signed-off-by: Liviu Dudau Liviu.Dudau@foss.arm.com --- board/armltd/vexpress64/vexpress64.c | 8 ++++++++ include/configs/vexpress_aemv8a.h | 4 +++- 2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/board/armltd/vexpress64/vexpress64.c b/board/armltd/vexpress64/vexpress64.c index 7cb4e00..6df9d60 100644 --- a/board/armltd/vexpress64/vexpress64.c +++ b/board/armltd/vexpress64/vexpress64.c @@ -38,6 +38,14 @@ int dram_init(void) return 0; }
+void dram_init_banksize(void) +{ + gd->bd->bi_dram[0].start = PHYS_SDRAM_1; + gd->bd->bi_dram[0].size = PHYS_SDRAM_1_SIZE; + gd->bd->bi_dram[1].start = PHYS_SDRAM_2; + gd->bd->bi_dram[1].size = PHYS_SDRAM_2_SIZE; +} + /* * Board specific reset that is system reset. */ diff --git a/include/configs/vexpress_aemv8a.h b/include/configs/vexpress_aemv8a.h index 6107c64..f06344a 100644 --- a/include/configs/vexpress_aemv8a.h +++ b/include/configs/vexpress_aemv8a.h @@ -164,11 +164,13 @@ #define CONFIG_SYS_LOAD_ADDR (V2M_BASE + 0x10000000)
/* Physical Memory Map */ -#define CONFIG_NR_DRAM_BANKS 1 +#define CONFIG_NR_DRAM_BANKS 2 #define PHYS_SDRAM_1 (V2M_BASE) /* SDRAM Bank #1 */ +#define PHYS_SDRAM_2 (0x880000000) /* Top 16MB reserved for secure world use */ #define DRAM_SEC_SIZE 0x01000000 #define PHYS_SDRAM_1_SIZE 0x80000000 - DRAM_SEC_SIZE +#define PHYS_SDRAM_2_SIZE 0x180000000 #define CONFIG_SYS_SDRAM_BASE PHYS_SDRAM_1
/* Enable memtest */

On Fri, Oct 9, 2015 at 3:09 PM, Liviu Dudau Liviu.Dudau@foss.arm.com wrote:
Juno comes with 8GB RAM, but U-Boot only passes 2GB to the kernel. Declare a secondary memory bank and set the sizes correctly.
Signed-off-by: Liviu Dudau Liviu.Dudau@foss.arm.com
Reviewed-by: Linus Walleij linus.walleij@linaro.org
Yours, Linus Walleij

Juno R1 has an XpressRICH3 PCIe host bridge that needs to be initialised in order for the Linux kernel to be able to enumerate the bus. Add support code here that enables the host bridge, trains the links and sets up the Address Translation Tables.
Signed-off-by: Liviu Dudau Liviu.Dudau@foss.arm.com --- board/armltd/vexpress64/vexpress64.c | 120 +++++++++++++++++++++++++++++++++++ board/armltd/vexpress64/vexpress64.h | 60 ++++++++++++++++++ 2 files changed, 180 insertions(+) create mode 100644 board/armltd/vexpress64/vexpress64.h
diff --git a/board/armltd/vexpress64/vexpress64.c b/board/armltd/vexpress64/vexpress64.c index 6df9d60..a90fa65 100644 --- a/board/armltd/vexpress64/vexpress64.c +++ b/board/armltd/vexpress64/vexpress64.c @@ -13,6 +13,7 @@ #include <linux/compiler.h> #include <dm/platdata.h> #include <dm/platform_data/serial_pl01x.h> +#include "vexpress64.h"
DECLARE_GLOBAL_DATA_PTR;
@@ -27,8 +28,127 @@ U_BOOT_DEVICE(vexpress_serials) = { .platdata = &serial_platdata, };
+#ifdef CONFIG_TARGET_VEXPRESS64_JUNO +void xr3pci_set_atr_entry(unsigned long base, unsigned long src_addr, + unsigned long trsl_addr, int window_size, + int trsl_param) +{ + /* X3PCI_ATR_SRC_ADDR_LOW: + - bit 0: enable entry, + - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) + - bits 7-11: reserved + - bits 12-31: start of source address + */ + writel((u32)(src_addr & 0xfffff000) | (window_size - 1) << 1 | 1, + base + XR3PCI_ATR_SRC_ADDR_LOW); + writel((u32)(trsl_addr & 0xfffff000), base + XR3PCI_ATR_TRSL_ADDR_LOW); + writel((u32)(src_addr >> 32), base + XR3PCI_ATR_SRC_ADDR_HIGH); + writel((u32)(trsl_addr >> 32), base + XR3PCI_ATR_TRSL_ADDR_HIGH); + writel(trsl_param, base + XR3PCI_ATR_TRSL_PARAM); + + printf("ATR entry: 0x%010lx %s 0x%010lx [0x%010llx] (param: 0x%06x)\n", + src_addr, (trsl_param & 0x400000) ? "<-" : "->", trsl_addr, + ((u64)1) << window_size, trsl_param); +} + +void xr3pci_setup_atr(void) +{ + /* setup PCIe to CPU address translation tables */ + unsigned long base = XR3_CONFIG_BASE + XR3PCI_ATR_PCIE_WIN0; + + /* forward all writes from PCIe to GIC V2M (used for MSI) */ + xr3pci_set_atr_entry(base, JUNO_V2M_MSI_START, JUNO_V2M_MSI_START, + JUNO_V2M_MSI_SIZE, XR3PCI_ATR_TRSLID_AXIDEVICE); + + base += XR3PCI_ATR_TABLE_SIZE; + + /* PCIe devices can write anywhere in memory */ + xr3pci_set_atr_entry(base, PHYS_SDRAM_1, PHYS_SDRAM_1, + 31 /* grant access to all RAM under 4GB */, + XR3PCI_ATR_TRSLID_AXIMEMORY); + base += XR3PCI_ATR_TABLE_SIZE; + xr3pci_set_atr_entry(base, PHYS_SDRAM_2, PHYS_SDRAM_2, + XR3_PCI_MEMSPACE64_SIZE, + XR3PCI_ATR_TRSLID_AXIMEMORY); + + + /* setup CPU to PCIe address translation table */ + base = XR3_CONFIG_BASE + XR3PCI_ATR_AXI4_SLV0; + + /* setup ECAM space to bus configuration interface */ + xr3pci_set_atr_entry(base, XR3_PCI_ECAM_START, 0, XR3_PCI_ECAM_SIZE, + XR3PCI_ATR_TRSLID_PCIE_CONF); + + base += XR3PCI_ATR_TABLE_SIZE; + + /* setup IO space translation */ + xr3pci_set_atr_entry(base, XR3_PCI_IOSPACE_START, XR3_PCI_IOSPACE_START, + XR3_PCI_IOSPACE_SIZE, XR3PCI_ATR_TRSLID_PCIE_IO); + + base += XR3PCI_ATR_TABLE_SIZE; + + /* setup 32bit MEM space translation */ + xr3pci_set_atr_entry(base, XR3_PCI_MEMSPACE_START, XR3_PCI_MEMSPACE_START, + XR3_PCI_MEMSPACE_SIZE, XR3PCI_ATR_TRSLID_PCIE_MEMORY); + + base += XR3PCI_ATR_TABLE_SIZE; + + /* setup 64bit MEM space translation */ + xr3pci_set_atr_entry(base, XR3_PCI_MEMSPACE64_START, XR3_PCI_MEMSPACE64_START, + XR3_PCI_MEMSPACE64_SIZE, XR3PCI_ATR_TRSLID_PCIE_MEMORY); +} +#endif + int board_init(void) { +#ifdef CONFIG_TARGET_VEXPRESS64_JUNO + u32 val; + int timeout = 200; + + /* Initialise the XpressRICH3 PCIe host bridge */ + + /* add credits */ + writel(0x00f0b818, XR3_CONFIG_BASE + XR3PCI_VIRTCHAN_CREDITS); + writel(0x1, XR3_CONFIG_BASE + XR3PCI_VIRTCHAN_CREDITS); + /* allow ECRC */ + writel(0x6006, XR3_CONFIG_BASE + XR3PCI_PEX_SPC2); + + /* reset phy and root complex */ + writel(JUNO_RESET_CTRL_PHY | JUNO_RESET_CTRL_RC, + XR3_RESET_BASE + JUNO_RESET_CTRL); + + do { + udelay(1000); + val = readl(XR3_RESET_BASE + JUNO_RESET_STATUS); + } while (--timeout && + (val & JUNO_RESET_STATUS_MASK) != JUNO_RESET_STATUS_MASK); + + if (!timeout) { + printf("PCI XR3 Root complex reset timed out\n"); + goto init_done; + } + + /* Wait for the link to train */ + udelay(20000); + timeout = 20; + + do { + udelay(1000); + val = readl(XR3_CONFIG_BASE + XR3PCI_BASIC_STATUS); + } while (--timeout && !(val & XR3PCI_BS_LINK_MASK)); + + if (!(val & XR3PCI_BS_LINK_MASK)) { + printf("Failed to negociate a link!\n"); + goto init_done; + } + + printf("PCIe XR3 Host Bridge enabled: x%d link (Gen %d)\n", + val & XR3PCI_BS_LINK_MASK, (val & XR3PCI_BS_GEN_MASK) >> 8); + + xr3pci_setup_atr(); + +init_done: +#endif return 0; }
diff --git a/board/armltd/vexpress64/vexpress64.h b/board/armltd/vexpress64/vexpress64.h new file mode 100644 index 0000000..433591b --- /dev/null +++ b/board/armltd/vexpress64/vexpress64.h @@ -0,0 +1,60 @@ +#ifndef __VEXPRESS64_H__ +#define __VEXPRESS64_H__ + +/* XpressRICH3 support */ +#define XR3_CONFIG_BASE 0x7ff30000 +#define XR3_RESET_BASE 0x7ff20000 + +#define XR3_PCI_ECAM_START 0x40000000 +#define XR3_PCI_ECAM_SIZE 28 /* as power of 2 = 0x10000000 */ +#define XR3_PCI_IOSPACE_START 0x5f800000 +#define XR3_PCI_IOSPACE_SIZE 23 /* as power of 2 = 0x800000 */ +#define XR3_PCI_MEMSPACE_START 0x50000000 +#define XR3_PCI_MEMSPACE_SIZE 27 /* as power of 2 = 0x8000000 */ +#define XR3_PCI_MEMSPACE64_START 0x4000000000 +#define XR3_PCI_MEMSPACE64_SIZE 33 /* as power of 2 = 0x200000000 */ + +#define JUNO_V2M_MSI_START 0x2c1c0000 +#define JUNO_V2M_MSI_SIZE 12 /* as power of 2 = 4096 */ + +#define XR3PCI_BASIC_STATUS 0x18 +#define XR3PCI_BS_GEN_MASK (0xf << 8) +#define XR3PCI_BS_LINK_MASK 0xff + +#define XR3PCI_VIRTCHAN_CREDITS 0x90 +#define XR3PCI_PEX_SPC2 0xd8 + +#define XR3PCI_ATR_PCIE_WIN0 0x600 +#define XR3PCI_ATR_PCIE_WIN1 0x700 +#define XR3PCI_ATR_AXI4_SLV0 0x800 + +#define XR3PCI_ATR_TABLE_SIZE 0x20 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_AXIDEVICE (0x420004) +#define XR3PCI_ATR_TRSLID_AXIMEMORY (0x4e0004) /* Write-through, read/write allocate */ +#define XR3PCI_ATR_TRSLID_PCIE_CONF (0x000001) +#define XR3PCI_ATR_TRSLID_PCIE_IO (0x020000) +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY (0x000000) + +#define XR3PCI_ECAM_OFFSET(b, d, o) (((b) << 20) | \ + (PCI_SLOT(d) << 15) | \ + (PCI_FUNC(d) << 12) | o) + +#define JUNO_RESET_CTRL 0x1004 +#define JUNO_RESET_CTRL_PHY (1 << 0) +#define JUNO_RESET_CTRL_RC (1 << 1) + +#define JUNO_RESET_STATUS 0x1008 +#define JUNO_RESET_STATUS_PLL (1 << 0) +#define JUNO_RESET_STATUS_PHY (1 << 1) +#define JUNO_RESET_STATUS_RC (1 << 2) +#define JUNO_RESET_STATUS_MASK (JUNO_RESET_STATUS_PLL | \ + JUNO_RESET_STATUS_PHY | \ + JUNO_RESET_STATUS_RC) + +#endif /* __VEXPRESS64_H__ */

On Fri, Oct 9, 2015 at 3:09 PM, Liviu Dudau Liviu.Dudau@foss.arm.com wrote:
Juno R1 has an XpressRICH3 PCIe host bridge that needs to be initialised in order for the Linux kernel to be able to enumerate the bus. Add support code here that enables the host bridge, trains the links and sets up the Address Translation Tables.
Signed-off-by: Liviu Dudau Liviu.Dudau@foss.arm.com
Very nice! Now we (soon) have PCIe on the Juno.
Did you:
- Test with compiling in e.g. network cards and booting off of ethernet on PCIe?
- Test what happens with a simple VGA card on PCIe? Sometimes the VGA card BIOS need to be initialized using an emulator running the x86 ROM and I never got that working on anything ARM :( (Maybe PCIe doesn't suffer from this? Just vanilla PCI has this problem? What do I know.)
+#ifdef CONFIG_TARGET_VEXPRESS64_JUNO +void xr3pci_set_atr_entry(unsigned long base, unsigned long src_addr,
unsigned long trsl_addr, int window_size,
int trsl_param)
+{
(...)
int board_init(void) { +#ifdef CONFIG_TARGET_VEXPRESS64_JUNO
(...)
+#endif return 0; }
+++ b/board/armltd/vexpress64/vexpress64.h @@ -0,0 +1,60 @@ +#ifndef __VEXPRESS64_H__ +#define __VEXPRESS64_H__
Instead of peppering with #ifdefs I suggest you do like this:
- Create two new files named board/armltd/vexpress64/pcie.c board/armltd/vexpress64/pcie.h
- Move all the #defines from the vexpress64.h file into the top of the pcie.c file.
- Use the pcie.h file for prototypes + stubs like this:
#ifdef CONFIG_TARGET_VEXPRESS64_JUNO void vexpress64_pcie_init(void); #else static inline void vexpress64_pcie_init(void) { } #endif
Then board_init() can unconditionallt call these functions and they will be stubbed out if compiled for anything else than Juno.
Yours, Linus Walleij

On Thu, 15 Oct 2015 16:42:19 +0200 Linus Walleij linus.walleij@linaro.org wrote:
On Fri, Oct 9, 2015 at 3:09 PM, Liviu Dudau Liviu.Dudau@foss.arm.com wrote:
Juno R1 has an XpressRICH3 PCIe host bridge that needs to be initialised in order for the Linux kernel to be able to enumerate the bus. Add support code here that enables the host bridge, trains the links and sets up the Address Translation Tables.
Signed-off-by: Liviu Dudau Liviu.Dudau@foss.arm.com
Very nice! Now we (soon) have PCIe on the Juno.
Did you:
Test with compiling in e.g. network cards and booting off of ethernet on PCIe?
Test what happens with a simple VGA card on PCIe? Sometimes the VGA card BIOS need to be initialized using an emulator running the x86 ROM and I never got that working on anything ARM :( (Maybe PCIe doesn't suffer from this? Just vanilla PCI has this problem? What do I know.)
Hi Linus,
Please note that this patchset does not enable enough functionality in U-Boot to allow for PCIe testing.
However, together with my Linux patchset series[1] I have tested U-Boot plus mainline Linux v4.3-rc5 on Juno r1 with 3x USB-to-PCIe cards and a Radeon HD6400 card. I have even got framebuffer out of the Radeon card :)
+#ifdef CONFIG_TARGET_VEXPRESS64_JUNO +void xr3pci_set_atr_entry(unsigned long base, unsigned long src_addr,
unsigned long trsl_addr, int window_size,
int trsl_param)
+{
(...)
int board_init(void) { +#ifdef CONFIG_TARGET_VEXPRESS64_JUNO
(...)
+#endif return 0; }
+++ b/board/armltd/vexpress64/vexpress64.h @@ -0,0 +1,60 @@ +#ifndef __VEXPRESS64_H__ +#define __VEXPRESS64_H__
Instead of peppering with #ifdefs I suggest you do like this:
Create two new files named board/armltd/vexpress64/pcie.c board/armltd/vexpress64/pcie.h
Move all the #defines from the vexpress64.h file into the top of the pcie.c file.
Use the pcie.h file for prototypes + stubs like this:
#ifdef CONFIG_TARGET_VEXPRESS64_JUNO void vexpress64_pcie_init(void); #else static inline void vexpress64_pcie_init(void) { } #endif
Then board_init() can unconditionallt call these functions and they will be stubbed out if compiled for anything else than Juno.
Yours, Linus Walleij
OK, I will do that. I'm waiting on some feedback from Ryan Harkin that wanted to test the patchset with some Linaro kernels and I will post a v2.
Best regards, Liviu
participants (3)
-
Linus Walleij
-
Liviu Dudau
-
Liviu Dudau