I'm currently working on the MCAN module for MPC5777C using MPC57xx motherboard. My problem is under the same code, same configuration, the MCAN module sometimes working, sometimes not. When it not working, the error from MCAN0_PSR LEC field is 0x05, which indicate during the transmission of a message, the device wanted to send a dominant level but the monitored bus value was recessive.
The error only happens at system boot up, after it successfully boot, if MCAN is working, it always working.
I hooked up oscilloscope with MCAN_0 pins and trigger the edge when it boot up, and
here's the image when it working (Yellow - MCAN0_TX; Green - MCAN0_RX):
here's the image when it not working (Yellow - MCAN0_TX; Green - MCAN0_RX)
Both case are using same code and same configuration, and when it boot up, whether is working or not, seems pretty random.
your MCAN is going to bus off state, most probably the transceiver detects some local failure and disables transmitter.
Try to remove EN jumper (J23(21) in position 5-6) and place it back. It should start communicating if the MCAN is not in INIT mode.