Stepping through the assembly code, i've found it does a compare on the 0xEB value of the argument, which passes. Then look is the "serial downloader" bit is set, if so, it jumps to 0x0020037C.
This is the last call where it jumps to CPSID:

After looking into it, it writes 0xE000ED0C to register R0, 0x5FA0004 to register R1, and then stores the value of R1 into the register at address R0. Looking into the Arm Cortex-M7 Processor Technical Reference Manual (pjrc.com), this is Application Interrupt and Reset Control Register.
According the ARMv7-M Architecture Reference Manual when written with the value 0x5FA0004 will request a reset:
Writing 1 to this bit asserts a signal to the external system to request a Local reset.
Ofcourse this should be done, and my guess is that is also does this and restart, because my application stops (pins are also reset).
Tracing the things the call to runBootloader does; I've found it does the following
- Compare 0xEB with the high byte of the given argument and if equal
- Check if serial downloaders is requested by AND with 0x100000 and if so
- Load the value of 0xeb100000 into register 0x400f8000+0x2C
- Call a reset
When doing this before the scheduler has started, it seems to work. This would mean there is an issue with my OS or application.
I'll dive into this and report back when I know more.
Thanks
Marcel