Hi,
here is what comes into my mind:
1. The base address of downloaded code must be 0x4000_0100 or higher. It is not possible to use area 0x4000_0000 - 0x4000_00FF.
2. If it works when debugger was connected, it could be caused by watchdog or ECC errors in RAM. Have you disabled watchdog in your bootloader? The RAM memory is initialized when the BAM copies user code to RAM memory. If you want to use RAM memory outside this area (outside the image which is copied to RAM) then it must be initialized explicitly by SW.
3. If it helps, here you can find description how to create RAM image which is downloaded by BAM:
Re: Boot Assist Module (BAM) with FlexCAN MPC5604P
Lukas