I think this is a clear indication that this is a signal deterioration issue.
I think you are familiar with Fourier analysis, which allows you to specify a signal as a superposition of a fundamental frequency and it's harmonics. For a rectangular wave with 20MHz, you quickly get beyond 200MHz of bandwidth requirements to get a decent signal form.
PCB layout and ground layers become important. And series resistors help to attenuate reflections and have an acceptable line impedance.
Even measuring with a scope gets difficult at this frequencies, the probe will definitely influence the signal, and improper probe impedance might totally mess it up.
Though I am no hardware guy, I must add.
A pragmatic solution would be to lower the SPI clock until you get stable & correct results. I would test this at the maximum allowable ambient temperature for your device, since heat tends to worsen such effects.