[U-Boot] PCIE_MEM_BUS for Freescale SoC
Roy,
Do you remember the reason why we use different virtual memory address from pci bus address with 36-bit? For example
include/configs/P1022DS.h-496-#define CONFIG_SYS_PCIE1_MEM_VIRT 0xc0000000 include/configs/P1022DS.h-497-#ifdef CONFIG_PHYS_64BIT include/configs/P1022DS.h:498:#define CONFIG_SYS_PCIE1_MEM_BUS 0xe0000000 include/configs/P1022DS.h-499-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc40000000ull include/configs/P1022DS.h-500-#else include/configs/P1022DS.h:501:#define CONFIG_SYS_PCIE1_MEM_BUS 0xc0000000 include/configs/P1022DS.h-502-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc0000000 include/configs/P1022DS.h-503-#endif
As far as I can tell, the following is the mapping
TLB: MEM_VIRT=>MEM_PHYS PCI: MEM_BUS=>MEM_PHYS LAW: MEM_PHYS=>pcie interface
Being different for MEM_VIRT and MEM_BUS cause confusion. When I run "pci header" command to show the BARs, I expect I can use "md" to access the BAR address. That's not the case if MEM_BUS is different from MEM_VIRT.
I forget why we did this for 36-bit addressing. The MEM_VIRT is the same as MEM_BUS for 32-bit addressing. And why do we use the same MEM_BUS address for all PCIe hose? I know they are not conflicting, but is it necessary?
Let me know if you have any concerns. Otherwise I will change all platforms to use consistent addresses.
York
On 11/21/2015 02:55 PM, York Sun wrote:
Roy,
Do you remember the reason why we use different virtual memory address from pci bus address with 36-bit? For example
include/configs/P1022DS.h-496-#define CONFIG_SYS_PCIE1_MEM_VIRT 0xc0000000 include/configs/P1022DS.h-497-#ifdef CONFIG_PHYS_64BIT include/configs/P1022DS.h:498:#define CONFIG_SYS_PCIE1_MEM_BUS 0xe0000000 include/configs/P1022DS.h-499-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc40000000ull include/configs/P1022DS.h-500-#else include/configs/P1022DS.h:501:#define CONFIG_SYS_PCIE1_MEM_BUS 0xc0000000 include/configs/P1022DS.h-502-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc0000000 include/configs/P1022DS.h-503-#endif
As far as I can tell, the following is the mapping
TLB: MEM_VIRT=>MEM_PHYS PCI: MEM_BUS=>MEM_PHYS LAW: MEM_PHYS=>pcie interface
Being different for MEM_VIRT and MEM_BUS cause confusion. When I run "pci header" command to show the BARs, I expect I can use "md" to access the BAR address. That's not the case if MEM_BUS is different from MEM_VIRT.
I forget why we did this for 36-bit addressing. The MEM_VIRT is the same as MEM_BUS for 32-bit addressing. And why do we use the same MEM_BUS address for all PCIe hose? I know they are not conflicting, but is it necessary?
(I hope Becky and Kumar still follow this mailing list)
I dug out an old commit 4c78d4a6c01621721b732418e1c6da684a56bbb1 by Becky Bruce. She believed overlapping the bus address for PCI controllers leaves more space. That's true. But we haven't use more than 512MB in u-boot. If we do need more space, we can easily move things around if we have PHYS_64BIT. If no objection, I'd like to change this back.
York
On Sat, 2015-11-21 at 15:29 -0800, York Sun wrote:
On 11/21/2015 02:55 PM, York Sun wrote:
Roy,
Do you remember the reason why we use different virtual memory address from pci bus address with 36-bit? For example
include/configs/P1022DS.h-496-#define CONFIG_SYS_PCIE1_MEM_VIRT 0xc0000000 include/configs/P1022DS.h-497-#ifdef CONFIG_PHYS_64BIT include/configs/P1022DS.h:498:#define CONFIG_SYS_PCIE1_MEM_BUS 0xe0000000 include/configs/P1022DS.h-499-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc40000000ull include/configs/P1022DS.h-500-#else include/configs/P1022DS.h:501:#define CONFIG_SYS_PCIE1_MEM_BUS 0xc0000000 include/configs/P1022DS.h-502-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc0000000 include/configs/P1022DS.h-503-#endif
As far as I can tell, the following is the mapping
TLB: MEM_VIRT=>MEM_PHYS PCI: MEM_BUS=>MEM_PHYS LAW: MEM_PHYS=>pcie interface
Being different for MEM_VIRT and MEM_BUS cause confusion. When I run "pci header" command to show the BARs, I expect I can use "md" to access the BAR address. That's not the case if MEM_BUS is different from MEM_VIRT.
I forget why we did this for 36-bit addressing. The MEM_VIRT is the same as MEM_BUS for 32-bit addressing. And why do we use the same MEM_BUS address for all PCIe hose? I know they are not conflicting, but is it necessary?
(I hope Becky and Kumar still follow this mailing list)
I dug out an old commit 4c78d4a6c01621721b732418e1c6da684a56bbb1 by Becky Bruce. She believed overlapping the bus address for PCI controllers leaves more space. That's true. But we haven't use more than 512MB in u-boot. If we do need more space, we can easily move things around if we have PHYS_64BIT. If no objection, I'd like to change this back.
I object. It's not about how much RAM is used in U-Boot; it's about how much memory the OS needs to bounce-buffer for DMA. The addresses set up by U-Boot should match what's in the device tree. Yes, on 85xx Linux reprograms the ATMU based on the device tree rather than trusting U-Boot, but that doesn't mean every OS does.
-Scott
On 11/23/2015 08:24 AM, Scott Wood wrote:
On Sat, 2015-11-21 at 15:29 -0800, York Sun wrote:
On 11/21/2015 02:55 PM, York Sun wrote:
Roy,
Do you remember the reason why we use different virtual memory address from pci bus address with 36-bit? For example
include/configs/P1022DS.h-496-#define CONFIG_SYS_PCIE1_MEM_VIRT 0xc0000000 include/configs/P1022DS.h-497-#ifdef CONFIG_PHYS_64BIT include/configs/P1022DS.h:498:#define CONFIG_SYS_PCIE1_MEM_BUS 0xe0000000 include/configs/P1022DS.h-499-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc40000000ull include/configs/P1022DS.h-500-#else include/configs/P1022DS.h:501:#define CONFIG_SYS_PCIE1_MEM_BUS 0xc0000000 include/configs/P1022DS.h-502-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc0000000 include/configs/P1022DS.h-503-#endif
As far as I can tell, the following is the mapping
TLB: MEM_VIRT=>MEM_PHYS PCI: MEM_BUS=>MEM_PHYS LAW: MEM_PHYS=>pcie interface
Being different for MEM_VIRT and MEM_BUS cause confusion. When I run "pci header" command to show the BARs, I expect I can use "md" to access the BAR address. That's not the case if MEM_BUS is different from MEM_VIRT.
I forget why we did this for 36-bit addressing. The MEM_VIRT is the same as MEM_BUS for 32-bit addressing. And why do we use the same MEM_BUS address for all PCIe hose? I know they are not conflicting, but is it necessary?
(I hope Becky and Kumar still follow this mailing list)
I dug out an old commit 4c78d4a6c01621721b732418e1c6da684a56bbb1 by Becky Bruce. She believed overlapping the bus address for PCI controllers leaves more space. That's true. But we haven't use more than 512MB in u-boot. If we do need more space, we can easily move things around if we have PHYS_64BIT. If no objection, I'd like to change this back.
I object. It's not about how much RAM is used in U-Boot; it's about how much memory the OS needs to bounce-buffer for DMA. The addresses set up by U-Boot should match what's in the device tree. Yes, on 85xx Linux reprograms the ATMU based on the device tree rather than trusting U-Boot, but that doesn't mean every OS does.
Isn't this backward? I mean the device tree should match u-boot, or u-boot should fix up the device tree, right? I checked several device tree files in kernel. We are not using more than 512MB for each PCI. What's the benefit of using the same address 0xe0000000?
York
On Mon, 2015-11-23 at 09:00 -0800, York Sun wrote:
On 11/23/2015 08:24 AM, Scott Wood wrote:
On Sat, 2015-11-21 at 15:29 -0800, York Sun wrote:
On 11/21/2015 02:55 PM, York Sun wrote:
Roy,
Do you remember the reason why we use different virtual memory address from pci bus address with 36-bit? For example
include/configs/P1022DS.h-496-#define CONFIG_SYS_PCIE1_MEM_VIRT 0xc0000000 include/configs/P1022DS.h-497-#ifdef CONFIG_PHYS_64BIT include/configs/P1022DS.h:498:#define CONFIG_SYS_PCIE1_MEM_BUS 0xe0000000 include/configs/P1022DS.h-499-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc40000000ull include/configs/P1022DS.h-500-#else include/configs/P1022DS.h:501:#define CONFIG_SYS_PCIE1_MEM_BUS 0xc0000000 include/configs/P1022DS.h-502-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc0000000 include/configs/P1022DS.h-503-#endif
As far as I can tell, the following is the mapping
TLB: MEM_VIRT=>MEM_PHYS PCI: MEM_BUS=>MEM_PHYS LAW: MEM_PHYS=>pcie interface
Being different for MEM_VIRT and MEM_BUS cause confusion. When I run "pci header" command to show the BARs, I expect I can use "md" to access the BAR address. That's not the case if MEM_BUS is different from MEM_VIRT.
I forget why we did this for 36-bit addressing. The MEM_VIRT is the same as MEM_BUS for 32-bit addressing. And why do we use the same MEM_BUS address for all PCIe hose? I know they are not conflicting, but is it necessary?
(I hope Becky and Kumar still follow this mailing list)
I dug out an old commit 4c78d4a6c01621721b732418e1c6da684a56bbb1 by Becky Bruce. She believed overlapping the bus address for PCI controllers leaves more space. That's true. But we haven't use more than 512MB in u-boot. If we do need more space, we can easily move things around if we have PHYS_64BIT. If no objection, I'd like to change this back.
I object. It's not about how much RAM is used in U-Boot; it's about how much memory the OS needs to bounce-buffer for DMA. The addresses set up by U -Boot should match what's in the device tree. Yes, on 85xx Linux reprograms the ATMU based on the device tree rather than trusting U-Boot, but that doesn't mean every OS does.
Isn't this backward? I mean the device tree should match u-boot,
Yes, and I don't want to deal with the churn of these device trees changing.
or u-boot should fix up the device tree, right?
In theory yes, but then you really are affecting Linux's usage.
I checked several device tree files in kernel. We are not using more than 512MB for each PCI. What's the benefit of using the same address 0xe0000000?
Reducing the amount of RAM whose address matches a PCI bus address and thus can't be used for PCI DMA.
-Scott
On 11/23/2015 09:07 AM, Scott Wood wrote:
On Mon, 2015-11-23 at 09:00 -0800, York Sun wrote:
On 11/23/2015 08:24 AM, Scott Wood wrote:
On Sat, 2015-11-21 at 15:29 -0800, York Sun wrote:
On 11/21/2015 02:55 PM, York Sun wrote:
Roy,
Do you remember the reason why we use different virtual memory address from pci bus address with 36-bit? For example
include/configs/P1022DS.h-496-#define CONFIG_SYS_PCIE1_MEM_VIRT 0xc0000000 include/configs/P1022DS.h-497-#ifdef CONFIG_PHYS_64BIT include/configs/P1022DS.h:498:#define CONFIG_SYS_PCIE1_MEM_BUS 0xe0000000 include/configs/P1022DS.h-499-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc40000000ull include/configs/P1022DS.h-500-#else include/configs/P1022DS.h:501:#define CONFIG_SYS_PCIE1_MEM_BUS 0xc0000000 include/configs/P1022DS.h-502-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc0000000 include/configs/P1022DS.h-503-#endif
As far as I can tell, the following is the mapping
TLB: MEM_VIRT=>MEM_PHYS PCI: MEM_BUS=>MEM_PHYS LAW: MEM_PHYS=>pcie interface
Being different for MEM_VIRT and MEM_BUS cause confusion. When I run "pci header" command to show the BARs, I expect I can use "md" to access the BAR address. That's not the case if MEM_BUS is different from MEM_VIRT.
I forget why we did this for 36-bit addressing. The MEM_VIRT is the same as MEM_BUS for 32-bit addressing. And why do we use the same MEM_BUS address for all PCIe hose? I know they are not conflicting, but is it necessary?
(I hope Becky and Kumar still follow this mailing list)
I dug out an old commit 4c78d4a6c01621721b732418e1c6da684a56bbb1 by Becky Bruce. She believed overlapping the bus address for PCI controllers leaves more space. That's true. But we haven't use more than 512MB in u-boot. If we do need more space, we can easily move things around if we have PHYS_64BIT. If no objection, I'd like to change this back.
I object. It's not about how much RAM is used in U-Boot; it's about how much memory the OS needs to bounce-buffer for DMA. The addresses set up by U -Boot should match what's in the device tree. Yes, on 85xx Linux reprograms the ATMU based on the device tree rather than trusting U-Boot, but that doesn't mean every OS does.
Isn't this backward? I mean the device tree should match u-boot,
Yes, and I don't want to deal with the churn of these device trees changing.
or u-boot should fix up the device tree, right?
In theory yes, but then you really are affecting Linux's usage.
I checked several device tree files in kernel. We are not using more than 512MB for each PCI. What's the benefit of using the same address 0xe0000000?
Reducing the amount of RAM whose address matches a PCI bus address and thus can't be used for PCI DMA.
That sounds like a good reason. OK, I will defer my patch until we have a stronger need to access the BARs under u-boot, or something else changes.
York
On 11/23/2015 09:07 AM, Scott Wood wrote:
On Mon, 2015-11-23 at 09:00 -0800, York Sun wrote:
On 11/23/2015 08:24 AM, Scott Wood wrote:
On Sat, 2015-11-21 at 15:29 -0800, York Sun wrote:
On 11/21/2015 02:55 PM, York Sun wrote:
Roy,
Do you remember the reason why we use different virtual memory address from pci bus address with 36-bit? For example
include/configs/P1022DS.h-496-#define CONFIG_SYS_PCIE1_MEM_VIRT 0xc0000000 include/configs/P1022DS.h-497-#ifdef CONFIG_PHYS_64BIT include/configs/P1022DS.h:498:#define CONFIG_SYS_PCIE1_MEM_BUS 0xe0000000 include/configs/P1022DS.h-499-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc40000000ull include/configs/P1022DS.h-500-#else include/configs/P1022DS.h:501:#define CONFIG_SYS_PCIE1_MEM_BUS 0xc0000000 include/configs/P1022DS.h-502-#define CONFIG_SYS_PCIE1_MEM_PHYS 0xc0000000 include/configs/P1022DS.h-503-#endif
As far as I can tell, the following is the mapping
TLB: MEM_VIRT=>MEM_PHYS PCI: MEM_BUS=>MEM_PHYS LAW: MEM_PHYS=>pcie interface
Being different for MEM_VIRT and MEM_BUS cause confusion. When I run "pci header" command to show the BARs, I expect I can use "md" to access the BAR address. That's not the case if MEM_BUS is different from MEM_VIRT.
I forget why we did this for 36-bit addressing. The MEM_VIRT is the same as MEM_BUS for 32-bit addressing. And why do we use the same MEM_BUS address for all PCIe hose? I know they are not conflicting, but is it necessary?
(I hope Becky and Kumar still follow this mailing list)
I dug out an old commit 4c78d4a6c01621721b732418e1c6da684a56bbb1 by Becky Bruce. She believed overlapping the bus address for PCI controllers leaves more space. That's true. But we haven't use more than 512MB in u-boot. If we do need more space, we can easily move things around if we have PHYS_64BIT. If no objection, I'd like to change this back.
I object. It's not about how much RAM is used in U-Boot; it's about how much memory the OS needs to bounce-buffer for DMA. The addresses set up by U -Boot should match what's in the device tree. Yes, on 85xx Linux reprograms the ATMU based on the device tree rather than trusting U-Boot, but that doesn't mean every OS does.
Isn't this backward? I mean the device tree should match u-boot,
Yes, and I don't want to deal with the churn of these device trees changing.
or u-boot should fix up the device tree, right?
In theory yes, but then you really are affecting Linux's usage.
I checked several device tree files in kernel. We are not using more than 512MB for each PCI. What's the benefit of using the same address 0xe0000000?
Reducing the amount of RAM whose address matches a PCI bus address and thus can't be used for PCI DMA.
Another thought, Linux scans and enumerates PCIe regardless U-boot. The BARs are setup according to device tree. So setting done u-boot is overwritten.
And, reusing 0xe0000000 for all PCI(e) controllers only applies to PHYS_64BIT in current code. Isn't address space more crowed for 32-bit?
Regarding the ATMU, if OS relies on u-boot to setup correct ATMU, it should have correct device tree as well.
Those being said, I can leave the code as is. It is not broken badly, but confusing if one tries to access BAR space manually.
York
participants (2)
-
Scott Wood -
York Sun