Hello everyone
I was able to solve the problem with a workaround by using bJPT_RadioSetChannel() instead of vMMAC_SetChannel(). The Contiki radio driver for the JN516x is using the MMAC and JPT API to control the radio settings and that might be a problem (this should be confirmed or denied by NXP). For now I'm happy with the solution, but it still doesn't fix the real problem. Here is the disassembly of both functions, maybe someone of the NXP developers can take a look.
Disassembly of section .text.vMMAC_SetChannel:
00000000 <vMMAC_SetChannel>:
0: 47 a4 40 b.entri 0x2,0x2
3: 05 43 b.mov r10,r3
5: 30 61 20 b.addi r3,r1,0x4
8: 00 80 b.movi r4,0x0
a: 48 00 00 b.jal a <vMMAC_SetChannel+0xa>
d: cc f0 43 69 b.lwz r7,-0x6940(r16)
11: 00 c0 b.movi r6,0x0
13: cc d0 03 69 b.sw -0x6940(r16),r6
17: cc b0 7e 29 b.lwz r5,-0x6b84(r16)
1b: 34 a5 f0 b.andi r5,r5,0xf
1e: cc d0 60 79 b.lwz r6,-0x61fc(r16)
22: 34 c6 f0 b.andi r6,r6,0xf
25: 60 c6 29 b.or r6,r6,r5
28: 41 06 f7 b.bnei r6,0x0,17 <vMMAC_SetChannel+0x17>
2b: cd 50 10 79 b.sw -0x61f8(r16),r10
2f: cc d0 51 29 b.lwz r6,-0x6b78(r16)
33: 34 c6 a0 b.andi r6,r6,0x5
36: 41 06 e0 b.bnei r6,0x0,3d <vMMAC_SetChannel+0x3d>
39: cc f0 03 69 b.sw -0x6940(r16),r7
3d: 30 61 20 b.addi r3,r1,0x4
40: 48 00 00 b.jal 40 <vMMAC_SetChannel+0x40>
43: 47 b4 40 b.reti 0x2,0x2
Disassembly of section .text.bJPT_RadioSetChannel:
00000000 <bJPT_RadioSetChannel>:
0: 3c e3 50 b.sfleui r3,0xa
3: 47 27 00 b.bf 11 <bJPT_RadioSetChannel+0x11>
6: 38 e0 58 b.ori r7,r0,0x1a
9: 3d 23 d8 b.sfltui r3,0x1b
c: 64 63 3a b.cmov r3,r3,r7
f: 0e 80 b.j 14 <bJPT_RadioSetChannel+0x14>
11: 38 60 d0 b.ori r3,r0,0xb
14: 8c 60 10 78 00 40 b.sw 0x2001e08(r0),r3
1a: 0c 00 b.j 1a <bJPT_RadioSetChannel+0x1a>