Hi,
even if you set the memory buffer area to 64 payload, you can easily send 8 bytes only, just set the MB's DLC (Data Length Code) properly. Of course you have less number of MBs available.
If the payload configuration is changed using FDCTRL.MBDSR0, starting address of each MB is different, so you must be sure correct addresses are used. You can refer to chapter 53.4.5 of the RM for a MB's memory map for each payload configuration.
Do you use SDK driver or some own code?
BR, Petr