Hi,
be sure FlexCAN operates using the ISO CAN FD protocol so you have CTRL2[ISOCANFDEN] bit set.
Or configure it according the CAN tool setting, but this appears using ISO CAN FD.
For the CAN bit setting configure it to have the same sample point as CAN tool. It is recommended to have same prescaler and TDC is not needed for this rates. You can use e.g.
CANx_CBT = 0x806B29EB; // bitrate=500kbps, CPI clk=80 MHz; Prescaler= 4, PROPSEG=11, PSEG1=16, PSEG2=12, RJW=12, smp=70%
CANx_FDCBT = 0x003514E5; // bitrateFD=1000kbps, CPI clk=80 MHz; fPrescaler= 4, fPROPSEG=5, fPSEG1=8, fPSEG2=6, fRJW=6, smp=70%
BR, Petr