Question asked by Tom Tatakis on Nov 1, 2013
Latest reply on Nov 4, 2013 by Tomas Vaverka

Hello,

I'm having trouble figuring out the example in the data sheet for reading and calculating the c12 coefficient.  The data sheet indicates there are 13 fractional bits, plus a sign bit, and it has 9 decimal points of pad built in.  In the example reading and calculation it shows a raw reading of the MSB and LSB of the coefficient as follows:

c12 coefficient MSB = 0x38

c12 coefficient LSB = 0xCC c12 coefficient = 0x38CC = 0.00086665

First, the 0x38CC value is 14 bits.  Even if you assume the 14th bit is actually the sign bit, then the value calculated should be negative.

From the description of 'dec point zero pad', I assume that means that each fractional bit actually has the resolution of 1/(2**(9 + 13))= 1/4194304.

If you take the straight 0x38cc= 14540, then c12= 14540/4194304 = 0.003466 (not quite the value in the example).

If you take 0x38cc and assume the 14th bit is the sign bit, then after the 2's compliment you get 0x733= 1843 for the absolute value.  This yields

c12= 1843/4194304 = 0.0004394 (about half of the example value- but the wrong sign).  So- what am I missing?  Or is the example incorrect?