Has this ever been resolved? I'm currently seeing the same issue on an i.MX6UL board.
CCM registers look identical for both boot sources. Using the MMDC profiling feature, I've found out that RAM access is so slow because something is causing high numbers of memory accesses:
After booting from SD card, there are very few RAM accesses when idle:
=> mw 21B0410 a; mw 21B0410 1; sleep 1; md 21B0418 4
021b0418: 179b2e3f 00000856 00000017 00000002
Booted from USB serial downloader:
=> mw 21B0410 a; mw 21B0410 1; sleep 1; md 21B0418 4
021b0418: 179b450d 06d02898 000279bf 004e81b4
So we're seeing ~160k read accesses and ~5M write accesses every second. There numbers are fairly stable.
I assume it must be something set up by the boot ROM, but how can I debug further what is causing these accesses?