Hi Mark,
Hi Nicolas,
On Sat, 2021-03-13 at 10:24 +0100, Mark Kettenis wrote: [...]
Fortunately Nicolas Saenz Julienne recently introduced dev_phys_to_bus() and dev_bus_to_phys() interfaces to do this. Those interfaces make use of a dma-ranges property in the device tree which doesn't work so well for PCI devices though.
Why doesn't it work with PCI devices? Raspberry Pi 4 has a PCIe bus that needs DMA translations. This is handled by parsing its 'dma-ranges' property. Here's how rpi4's devicetree looks like[1]:
pcie0: pcie@7d500000 { compatible = "brcm,bcm2711-pcie"; ranges = <0x02000000 0x0 0xf8000000 0x6 0x00000000 0x0 0x04000000>; /* * The wrapper around the PCIe block has a bug preventing it * from accessing beyond the first 3GB of memory. */ dma-ranges = <0x02000000 0x0 0x00000000 0x0 0x00000000 0x0 0xc0000000>; };
Does that work even though there are no device tree nodes for the PCI devices themselves?
'dma-ranges' only makes sense on bus nodes. So whenever you perform a DMA address tranlation you start parsing from the endpoint device's parent. In this case the PCIe root bridge, which has a proper DT node.
However, the PCI code in U-Boot already has a way to describe DMA address translation through its regions support.
regions contain a representation of devicetree's 'ranges' property, which doesn't describe the DMA address translations, but CPU's view of PCI memory.
It describes both. The DMA address translations are described by regions that have the PCI_REGION_SYS_MEMORY flag set. Some of the PCI host bridge drivers set up these regions, and the generic DM_PCI code by default adds such regions for the system RAM banks. And there is a fdt_pci_dma_ranges() function to populate a "dma-ranges" property based on these regions.
Sorry, I wasn't aware PCI_REGION_SYS_MEMORY existed.
The benefit of the PCI regions code is that it can describe more complex setups where a single translation offset isn't enough. And the code will print a warbing if you try to do DMA to memory that isn't "visable" for the PCI device.
In the particular example of the M1 code I don't really want to add a dma-ranges property to the device tree since the IOMMU setup is only used by u-boot and not really usable by Linux or another OS as it only covers a (relatively) small part of system memory.
So I understand that this region will be hard-coded in your PCI driver. And there will be no trace of it in DT. If so, I think I understand your strategy.
diff --git a/include/phys2bus.h b/include/phys2bus.h index 866b8b51a8..13d23ef4bb 100645 --- a/include/phys2bus.h +++ b/include/phys2bus.h @@ -23,14 +23,21 @@ static inline unsigned long bus_to_phys(unsigned long bus)
#if CONFIG_IS_ENABLED(DM) #include <dm/device.h> +#include <pci.h>
static inline dma_addr_t dev_phys_to_bus(struct udevice *dev, phys_addr_t phys) {
- if (device_is_on_pci_bus(dev))
return dm_pci_phys_to_bus(dev, phys, PCI_REGION_SYS_MEMORY);Note that this would break USB support on RPi4.
It should work if the PCI host bridge driver sets up the regions properly based on the dma-ranges property. Looks like the current driver doesn't do this, so I'll have to fix that.
That would work. But it would be better to integrate your setup with struct udevice's DMA tranlation attributes. It's as simple as performing the PCI_REGION_SYS_MEMORY parsing in device_get_dma_constraints(). Then you'd be able to use plain, generic, dev_phys_to_bus(). With the added benefit of not having the parse all resources everytime you perform a translation, and not having to add churn in pcie-brcmstb.
Regards, Nicolas