Hi,
The fluctuation might be caused by the hardware/interface. Test from my side with a real motor and resolver, the error is little. Please see the snapshot.

Please check the hardware part, especially:
1. The phase between sin & cos waves from the resolver output to SDADC's pins. If there are phase shifts caused by the circuit.
2. The amplitude of the sin & cos waves. If the amplitude has a large difference when they are processed by the hardware circuit. And if the maximum peak value is over the design limit.
BRs,
Vincent