AnsweredAssumed Answered

MLIB FRAC16 Edgecase

Question asked by Adrian Esser on Apr 24, 2019
Latest reply on May 6, 2019 by Felipe García



So I noticed a weird bug / edgecase in the FRAC16 macro defined in "rtcesl/mlib/mlib_types.h" header file. The nested ternary expression appears to implicitly cast the type of the result as the TRUE case for the ternary expression. So when the input is a number less than -1, the default return value of 0x8000 is first cast to a double (32768) and then to a signed short (32767). But we want 0x8000 to be directly cast to a signed short (-32768). 


Pretty certain that the fix is to just cast the floating point expression inside the nested ternary expression, ( x )*0x8000, to a frac16_t so that FALSE case of 0x8000 isn't cast to a double first. I've attached a screenshot of a little test in an online C++ shell demonstrating the problem and the proposed fix. The casting shouldn't matter for the positive edge case (0x7FFF) because that resolves to 32767. 


Since the FRAC32, FRAC64, ACC16, ACC32, ACC64 macros are written the same, they should have this problem as well. 


Thanks for your time! 


- Adrian