On Sat, 1 May 2021 14:23:32 +0200 (CEST) Mark Kettenis mark.kettenis@xs4all.nl wrote:
Hi,
From: Bin Meng bmeng.cn@gmail.com Date: Sat, 1 May 2021 19:45:02 +0800
On Fri, Apr 30, 2021 at 7:22 PM Andre Przywara andre.przywara@arm.com wrote:
Hi,
We now see the first Allwinner devices [1] having DRAM located above 4GB in address space (4GB DRAM starting at 1GB). After one fix[2] this works somewhat fine, but the sun8i-emac network device is still limited to 32-bit DMA addresses. With U-Boot relocating itself (plus stack and heap) to the end of DRAM, it now runs completely beyond 4GB on those machines, so not giving pure 32-bit addresses for buffers anymore. In Linux we handle this easily by just keeping the default DMA mask at 32 bits, and letting the DMA framework deal with the nasty details.
I was wondering how this should be handled in U-Boot? The straight forward solution would be:
- Let the driver allocate the RX and TX buffers separately, placing them below 4GB in the address space (using lmb_reserve(), I guess?)
- Use those RX buffers and hand the addresses back to the upper layers.
- We already copy TX packets, so this would also be covered, in this situation. Other drivers might need to introduce copying.
This sounds like a common problem, so I was wondering if there is a more generic solution to this? Maybe there are already platforms or devices affected? Or should the whole heap and stack be moved below 4GB (if this is easily possible)?
My understanding is that the relocated address of U-Boot should be below 4GB then there is no problem for the 32-bit DMA. I thought this is a rule to be followed by every board, but this is not the case on your board?
Bin, interesting, where is this coming from? Was this originally for 32-bit CPUs with some address extension (PAE/LPAE)? I think on *sane* 64-bit systems there would be no need for this restriction, except maybe for this 32-bit DMA limitation (which is more of a device problem).
Yes, that was my impression as well. And I think that would work fine on this board as there is plenty of DRAM below 4GB. And this can be achieved by implementing the board_get_usable_ram_top() function.
Ah, I think this is the thing I missed and was looking for: So we *can* restrict everything *U-Boot* to 32 bits and save us a lot of hassle.
Thanks for that hint!
As I indicated in my reply, some care is needed in the EFI subsystem, but there already is a solution for that. There is CONFIG_EFI_LOADER_BOUNCE_BUFFER, but that might not actually be needed in this case. By default the EFI subsystem will mark all conventional memory above "ram_top" as EFI_BOOT_SERVICES_DATA. So EFI applications uch as OS loaders will not allocate that memory until they've called ExitBootServices() at which point U-Boot will be completely out of the picture.
Oh nice, this looks like what I need. So EFI apps would never use this memory for I/O buffers.
So I gave this a try and this solves my problem quite neatly: Linux sees the full DRAM, but U-Boot never touches anything beyond 4GB. Briefly tested Linux with both EFI and booti. Will include the board_get_usable_ram_top() implementation in the v2 of my 4GB enablement patch.
Thanks again!
Cheers, Andre
In our case we make the buffers part of our priv struct, so should there be an option to let the priv_auto allocation come from below 4GB?
Grateful for any input on this!
Regards, Bin