Hi,
No, it is not possible to make |A|=10, even if you assume that attenuation is 40% or 60%, reason is mentioned in the paragraph you sent telling that it is physically impossible to separate out the magnitude of the geomagnetic field strength from the soft iron, reason of why in the NXP software, we always set the determinant of the sift iron matrix |A|=1.
You can download the complete source code and license for the Hard-Iron and Soft-Iron Calibration from: https://www.nxp.com/support/developer-resources/evaluation-and-development-boards/freedom-developmen...
We provide full optimized MCU software with full source code including 10 element magnetic calibration models for Hard-Iron and Soft-Iron on version 7.2 of sensor fusion library via KDSK download from MCUXpresso for FRDM_K64F and FRDM_K22F.
*Please check the ISSDK checkbox when building your KSDK.
Once downloaded you can find specifically the 10 element code can be found in the file magnetic.c in the projects.