Hi,
yes, when RXFIFO is enabled number of available MBs decreases. See more in chapters 55.5.6 Rx FIFO structure and 55.5.2.12 Control 2 Register (CTRL2) of the device RM. Some words from there...
When MCR[RFEN] is set, the memory area 80h–DCh (which is normally occupied by MBs 0–5) is used by the reception FIFO engine.
An additional memory area, which starts at E0h and may extend up to 2DCh (normally occupied by MBs 6–37) depending on the CTRL2[RFFN] field setting, contains the ID filter table (configurable from 8 to 128 table elements).
As FlexCAN0 has up to 32MBs then there is not possible to have 128 filter elements in RXFIFO table using Format A, max is 104. But you can config filters in Format B, which doubles number of IDs.
BR, Petr