I'm not sure how to run proper timing tests on the process, but stopwatch timing just between seeing "Trying to boot" and "U-Boot 2023.04-rc2" showed the return to BootROM under 1 second, and the direct from SPI around 4 seconds. I thought the goal of loading from SPI directly was speed, but returning to BootROM is significantly faster on this board.
You should check SPI speed in DTS file and also in the defconfig.
I think we have probably seen this slowdown before. There is a TODO in the way the DTS nodes are parsed by DM uclass. So this config must be set in defconfig to get around that bug: CONFIG_SF_DEFAULT_SPEED=50000000
That works and is much faster now. I'll submit it in a V2 for these two patches once Pali's mvebu changes are accepted. Only question I have is: should the faster speed be applied to all three defconfigs? CONFIG_SF_DEFAULT_SPEED=1000000 (50x less) is implicitly added to the MMC and SATA configs at the moment.