Mike,
Firstly, thank you for such a prompt and well-detailed reply.
The fact that the NXP MCUs do the calibration makes sense--I actually don't feel like this is our biggest issue. I've used MotionCal to obtain what seem like appropriate soft- and hard-iron matrix coefficients.
The robot was indeed built to address the issue of producing adverse fields that could affect the magnetometer. The sensor is mounted on an all-aluminum arm about half a meter away from the body of the robot. The main body is ABS plastic with a single servo, battery pack, and ESP32. The arm with the sensor does rotate continuously, however, the base remains stationary and the sensor, robot body (including servo and batteries) and arm all rotate as a single unit. This way, any field produced from the robot remains constant to the sensor; the sensor's position does not change in relation to this field.
By now, I have gotten the compass (getHeading()) to function very nicely. The issue is with measuring the pitch. For our orientation, the pitch of the arm is measured from (-1) * getRoll(). When the arm is stationary, the values are quite good. However, once it begins rotating, the pitch tends to drift up or down, even while the actual pitch isn't changing.
I've read as many docs as I can and have a general understanding of how SensorFusion.cpp works, and the difficulty in decoupling these values due to how they are calculated from accelerometer and gyro inputs. I realize we might not be able to fully decouple the heading/roll, but do you have any tips for anything that could help me? We are assuming a constant rotational velocity of the arm (in the x-y plane), with gentle fluctuations in the pitch (z-direction).
EDIT: One clarifying detail, the pitch will never exceed +/- 5 degrees.