Alright, so it seems like the problem is not in my software nor hardware, but in FlexCAN. I isolated the problem to how the tq segments are set. Apparently FlexCAN either cannot handle a too small phase2 segment, or it makes strange assumptions over how the tq segments should be. Whatever is the case, it is seemingly not documented anywhere.
I just fiddled around with the tq settings and got very strange results. With one tq setting, everything works just fine. With another, FlexCAN runs amok and sets the baudrate to a rubbish value. I don't know why this happens.
The values in the table below work, tested with trial & error. If I cange the size of one segment or the other, FlexCAN might apparently run amok. Min & Max is the allowed range for the location of the sample point, in CANopen DS301. This is part of a larger spread sheet where I also check if it is possible to use a certain amount of tq with the given oscillator clock, 8MHz in this case, and within the 1% inaccuracy tolerance.
Baudrate kHz tq_tot tq_sync tq_prop tq_pseg1 tq_pseg2 rjw sample_point Min Max
1000 8 1 3 2 2 2 75,0% 75,0% 90,0%
800 10 1 3 4 2 2 80,0% 75,0% 90,0%
500 16 1 8 5 2 2 87,5% 85,0% 90,0%
250 16 1 8 5 2 2 87,5% 85,0% 90,0%
125 16 1 8 5 2 2 87,5% 85,0% 90,0%
50 16 1 8 5 2 2 87,5% 85,0% 90,0%
20 16 1 8 5 2 2 87,5% 85,0% 90,0%
10 16 1 8 5 2 2 87,5% 85,0% 90,0%
For example, I tried to use fewer tq. I tried 250kbps, 8tq, where prop=4, phase1=2, phase2=1, rjw=2. This gives a sample point at exactly 87.5%. Prescaler = 8000kHz / (250kbps * 8) = 4. PRESDIV=3.
Based on this, my algorithm writes 0348 0003 hex to the CTRL register. This causes FlexCAN to go berserk on the CAN bus, picking a completely wrong baudrate of around 40 kbps and ignoring bus arbitration, killing the bus.