I found the bug!!!
The main code and the bootloader code had the ram set up in two different places. So after my bootloader got done and tried to jump to his code, his code would try to set the ram at a different place. Problem is that the register that does that can only be set once (and after the bootloader code set it at the beginning, the main code would try to do it again). To fix the problem I changed the position of my ram to the same place as the main code. That meant that I put the ram on top of my vector table, but since the bootloader does not need any interrupts, I should be fine. Easy fix!
Now, the datasheet also says that those registers can be modified more then once if the microcontroller is running in special mode, for example in DEBUG! So thats why the whole thing would work in debug but not afterwards...
Thanks for everyone that spent some time trying to figure this out. If you need more information about anything just reply to this thread.
Andre
P.S.: Hooray! Bootloader works!!!