Hi Albert,
Hi Marek,
On Wed, 10 Jul 2013 02:30:29 +0200, Marek Vasut marex@denx.de wrote:
The flush_dcache_range() on arm926 did not work as expected on i.MX28.
This can be observed during the operation of the FEC ethernet driver where the driver did occasionally fail with timeout trying to transmit a frame. The FEC ethernet driver uses DMA for transmitting the frame in the following fashion:
- Set bits in DMA descriptor
- Write DMA descriptor into aligned DRAM address
- Flush D-Cache over the descriptor
- Start DMA
- Invalidate D-Cache over the descriptor
- Test if certain bits in DMA descriptor are unset
Very not so often it happened that the bits in the DMA descriptor were still set even after hardware register -- which is not cached -- indicated otherwise.
Here I will theoreticise, Albert, can you please correct me if I'm wrong?
This leads me to believe the DMA descriptor was still in the cacheline after being flushed in step 2) and during the DMA gets evicted into DRAM therefore corrupting the result of readback in 5) . By reading the ARM926 datasheet DDI0198E_arm926ejs_r0p5_trm.pdf page 50 table 2-17, it is not clear whether cacheline that is Valid+Clean will be invalidated in the D-Cache using the "mcr p15, 0, <Rx>, c7, c14, 1" instruction or whether only Valid+Dirty lines are cleaned+invalidated. The other thing that is unclear to me is whether a cacheline that is Valid+Clear is written back into DRAM when it is evicted from cache.
The way I see table 2-17, the "if valid and dirty" condition only applies to the "write" part of the description, not to the "marked not valid" one; this reading makes the most sense and is the most consistent with the cache functioning as a whole.
So basically all Valid+Dirty get cleaned . Nothing is done on Valid+Clean . Finally, as they are now all Valid+Clean , they all get Invalidated .
This is how it'd make the most sense, but I'm afraid this is not what I observe.
As for a valid and clean (rather than "clear", I assume) line being written if evicted, I think it is not. Eviction is not flushing: the line is kicked out, plain and brutal. And even if we're talking about flushing, if the line is clean then it is coherent with the RAM area it caches, and writing it back there is unneeded.
Yes, this _does_ make full sense to me.
Interestingly enough, running invalidate_dcache_range() after doing the flush_dcache_range() over the descriptor solved the problem and I see no occasional timeout anymore. This confirms my opinion that the descriptor might remain in the cache and be written back during the DMA operation.
Can you point me to the source code location(s) where the sequence above is implemented?
See:
738 flush_dcache_range(addr, addr + size);
in drivers/net/fec_mxc.c . If I put invalidate_dcache_range(addr, addr + size); , all seems to work well. Note it also happens with multiple compilers, Debian gcc 4.8.0, ELDK 5.2 and ELDK 5.3 .
Best regards, Marek Vasut