So we found that it happens on only on one for our boards. It does not happen after running a memory test from the boot loader, the difference being the pattern left in DDR memory - 0 with memory test, deadbeef when not run.
We have not seen it happen on Linux 6.4.
Looking at the Linux log to see if there was anything that may have fixed the problem