Hi, I guess it'll be rather issue with I-cache than D-cache. After POR cache content is invalid, by first code execution the code is being loaded to the cache by bursts, what could cause initial delay (in dependence of the code, with linear code you should see no impact) plus initialization of cache itself (invalidation takes some cycles). On the other hand during runtime having enabled caches greatly increase execution performance.
But if there is some difference in execution time I would not expect much. Do you have some numbers?
You may consider enabling of I-cache in the main application. Another point is whether you are configuring PLL clocks before MCU init, this may significantly increase initialization time.