Hello Miquel,
Initially investigating a Linux network issue causing a lot of drop and poor network performances on a custom system based on a TQMA6A module (based on an iMX6Q), [1st link below].
I eventually correlated my observations with a contention at the NIC level when in concurrency with the graphics pipeline. Troubleshooting this in the kernel lead to disabling DMA bursts accesses made by the IPU in order to avoid triggering the QoS at the interconnect level, reducing from 50 to 10% the drop rate on eth0, [2nd link below]. The solution worked on my setup but not on others, which still suffered from abnormally high drop rates even with this "fix".
After looking a while into TQ Systems BSP I figured out a number of differences in recent U-Boot out-of-tree patches they had in their repository [3rd link]. Parsing the differences one after the other lead me to this final solution.
The reset pad of the DDR controller was apparently misconfigured, Bit 18-19 picturing the "DDR select field". The current value b11 is reserved. The only defined value as of version 6 of the iMX6Q manual was b00 "DDR3 and LPDDR2 mode". In practice no register difference has been spotted after changing this configuration but all issues tracked thus far just vanished. All previous fixes have been proven irrelevant. Just clearing this field solved all our network issues and the drop rate as measured by iperf3 felt back to 0%.
Link: https://lore.kernel.org/netdev/20231012193410.3d1812cf@xps-13/
Link: https://lists.freedesktop.org/archives/dri-devel/2023-October/428251.html
Link: https://github.com/tq-systems/u-boot-tqmaxx/commit/15eb6abbefbf6916c28467b85...
Signed-off-by: Miquel Raynal < miquel.raynal@bootlin.com
board/tq/tqma6/tqma6q.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/board/tq/tqma6/tqma6q.cfg b/board/tq/tqma6/tqma6q.cfg index a49489aed3f..a345c4de93d 100644 --- a/board/tq/tqma6/tqma6q.cfg +++ b/board/tq/tqma6/tqma6q.cfg @@ -36,7 +36,7 @@ DATA 4, MX6_IOM_DRAM_SDCLK_1, 0x00008030 DATA 4, MX6_IOM_DRAM_CAS, 0x00008030 DATA 4, MX6_IOM_DRAM_RAS, 0x00008030 DATA 4, MX6_IOM_GRP_ADDDS, 0x00000030 -DATA 4, MX6_IOM_DRAM_RESET, 0x000C3030 +DATA 4, MX6_IOM_DRAM_RESET, 0x00003030
Thank you for pointing this out. Originally this error came from an older/ancient reference manual. Sorry that we missed to bring this upstream. We will send the changes for DCD data in the next days.
DATA 4, MX6_IOM_DRAM_SDCKE0, 0x00003000 DATA 4, MX6_IOM_DRAM_SDCKE1, 0x00000000 DATA 4, MX6_IOM_DRAM_SDBA2, 0x00000000 -- 2.34.1
Markus
Hi Markus,
Markus.Niebel@ew.tq-group.com wrote on Thu, 23 Nov 2023 14:04:43 +0100:
Hello Miquel,
Initially investigating a Linux network issue causing a lot of drop and poor network performances on a custom system based on a TQMA6A module (based on an iMX6Q), [1st link below].
I eventually correlated my observations with a contention at the NIC level when in concurrency with the graphics pipeline. Troubleshooting this in the kernel lead to disabling DMA bursts accesses made by the IPU in order to avoid triggering the QoS at the interconnect level, reducing from 50 to 10% the drop rate on eth0, [2nd link below]. The solution worked on my setup but not on others, which still suffered from abnormally high drop rates even with this "fix".
After looking a while into TQ Systems BSP I figured out a number of differences in recent U-Boot out-of-tree patches they had in their repository [3rd link]. Parsing the differences one after the other lead me to this final solution.
The reset pad of the DDR controller was apparently misconfigured, Bit 18-19 picturing the "DDR select field". The current value b11 is reserved. The only defined value as of version 6 of the iMX6Q manual was b00 "DDR3 and LPDDR2 mode". In practice no register difference has been spotted after changing this configuration but all issues tracked thus far just vanished. All previous fixes have been proven irrelevant. Just clearing this field solved all our network issues and the drop rate as measured by iperf3 felt back to 0%.
Link: https://lore.kernel.org/netdev/20231012193410.3d1812cf@xps-13/
Link: https://lists.freedesktop.org/archives/dri-devel/2023-October/428251.html
Link: https://github.com/tq-systems/u-boot-tqmaxx/commit/15eb6abbefbf6916c28467b85...
Signed-off-by: Miquel Raynal < miquel.raynal@bootlin.com
board/tq/tqma6/tqma6q.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/board/tq/tqma6/tqma6q.cfg b/board/tq/tqma6/tqma6q.cfg index a49489aed3f..a345c4de93d 100644 --- a/board/tq/tqma6/tqma6q.cfg +++ b/board/tq/tqma6/tqma6q.cfg @@ -36,7 +36,7 @@ DATA 4, MX6_IOM_DRAM_SDCLK_1, 0x00008030 DATA 4, MX6_IOM_DRAM_CAS, 0x00008030 DATA 4, MX6_IOM_DRAM_RAS, 0x00008030 DATA 4, MX6_IOM_GRP_ADDDS, 0x00000030 -DATA 4, MX6_IOM_DRAM_RESET, 0x000C3030 +DATA 4, MX6_IOM_DRAM_RESET, 0x00003030
Thank you for pointing this out. Originally this error came from an older/ancient reference manual. Sorry that we missed to bring this upstream. We will send the changes for DCD data in the next days.
No problem, I'm glad this can now be solved. By any chance, could you point to the relevant location of the manual (ddr or imx6 ?) explaining what this is actually about? Because I failed to bring any real explanation to my observations so far.
Thanks, Miquèl
Hi Miquel,
Am Donnerstag, 23. November 2023, 14:12:33 CET schrieb Miquel Raynal:
Hi Markus,
Markus.Niebel@ew.tq-group.com wrote on Thu, 23 Nov 2023 14:04:43 +0100:
Hello Miquel,
Initially investigating a Linux network issue causing a lot of drop and poor network performances on a custom system based on a TQMA6A module (based on an iMX6Q), [1st link below].
I eventually correlated my observations with a contention at the NIC level when in concurrency with the graphics pipeline. Troubleshooting this in the kernel lead to disabling DMA bursts accesses made by the IPU in order to avoid triggering the QoS at the interconnect level, reducing from 50 to 10% the drop rate on eth0, [2nd link below]. The solution worked on my setup but not on others, which still suffered from abnormally high drop rates even with this "fix".
After looking a while into TQ Systems BSP I figured out a number of differences in recent U-Boot out-of-tree patches they had in their repository [3rd link]. Parsing the differences one after the other lead me to this final solution.
The reset pad of the DDR controller was apparently misconfigured, Bit 18-19 picturing the "DDR select field". The current value b11 is reserved. The only defined value as of version 6 of the iMX6Q manual was b00 "DDR3 and LPDDR2 mode". In practice no register difference has been spotted after changing this configuration but all issues tracked thus far just vanished. All previous fixes have been proven irrelevant. Just clearing this field solved all our network issues and the drop rate as measured by iperf3 felt back to 0%.
Link: https://lore.kernel.org/netdev/20231012193410.3d1812cf@xps-13/
Link: https://lists.freedesktop.org/archives/dri-devel/2023-October/428251.htm l
Link: https://github.com/tq-systems/u-boot-tqmaxx/commit/15eb6abbefbf6916c2846 7b85485911dad3da6bc
Signed-off-by: Miquel Raynal < miquel.raynal@bootlin.com
board/tq/tqma6/tqma6q.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/board/tq/tqma6/tqma6q.cfg b/board/tq/tqma6/tqma6q.cfg index a49489aed3f..a345c4de93d 100644 --- a/board/tq/tqma6/tqma6q.cfg +++ b/board/tq/tqma6/tqma6q.cfg @@ -36,7 +36,7 @@ DATA 4, MX6_IOM_DRAM_SDCLK_1, 0x00008030
DATA 4, MX6_IOM_DRAM_CAS, 0x00008030 DATA 4, MX6_IOM_DRAM_RAS, 0x00008030 DATA 4, MX6_IOM_GRP_ADDDS, 0x00000030
-DATA 4, MX6_IOM_DRAM_RESET, 0x000C3030 +DATA 4, MX6_IOM_DRAM_RESET, 0x00003030
Thank you for pointing this out. Originally this error came from an older/ancient reference manual. Sorry that we missed to bring this upstream. We will send the changes for DCD data in the next days.
No problem, I'm glad this can now be solved. By any chance, could you point to the relevant location of the manual (ddr or imx6 ?) explaining what this is actually about? Because I failed to bring any real explanation to my observations so far.
It's a bit hidden but the comment above that list indicates these settings are iomuxc configurations. In this case the register IOMUXC_SW_PAD_CTL_PAD_DRAM_RESET. See i.MX6Q RM Rev.6 05/2020 section 36.4.347. Your patch configures the field "DDR Select Field" from reserved3 to DDR3_LPDDR2. It seems this field has to be set to 0 in every case.
Thanks again and best regards, Alexander
Thanks, Miquèl
Hi Alexander,
--- a/board/tq/tqma6/tqma6q.cfg +++ b/board/tq/tqma6/tqma6q.cfg @@ -36,7 +36,7 @@ DATA 4, MX6_IOM_DRAM_SDCLK_1, 0x00008030
DATA 4, MX6_IOM_DRAM_CAS, 0x00008030 DATA 4, MX6_IOM_DRAM_RAS, 0x00008030 DATA 4, MX6_IOM_GRP_ADDDS, 0x00000030
-DATA 4, MX6_IOM_DRAM_RESET, 0x000C3030 +DATA 4, MX6_IOM_DRAM_RESET, 0x00003030
Thank you for pointing this out. Originally this error came from an older/ancient reference manual. Sorry that we missed to bring this upstream. We will send the changes for DCD data in the next days.
No problem, I'm glad this can now be solved. By any chance, could you point to the relevant location of the manual (ddr or imx6 ?) explaining what this is actually about? Because I failed to bring any real explanation to my observations so far.
It's a bit hidden but the comment above that list indicates these settings are iomuxc configurations. In this case the register IOMUXC_SW_PAD_CTL_PAD_DRAM_RESET. See i.MX6Q RM Rev.6 05/2020 section 36.4.347. Your patch configures the field "DDR Select Field" from reserved3 to DDR3_LPDDR2. It seems this field has to be set to 0 in every case.
Yes, that's also what I get from reading the iMX6Q manual. But I'm sorry I don't understand why a runtime change of a reset pin configuration can be so impacting. It's not like it only affects the power-on sequence because I can reproduce the strange QoS/NIC issues with a devmem once the kernel has started. I believe there is a bit more than just a pad configuration behind this field.
Thanks, Miquèl
participants (3)
-
Alexander Stein -
Markus Niebel -
Miquel Raynal