4.9.11 kernel crash in imx serial driver

[anitathomas]

We are still seeing the same crash on the 4.9.11 (yocto-morty) kernel wit the IMX7d. Our NXP FAE suggested we post this on here.

Did see some new imx serial improvement patches and a similar crash at 

http://lists.infradead.org/pipermail/linux-arm-kernel/2017-June/516845.html

Is NXP looking to incorporate these in 4.9 or later kernels?

[248275.244910] Unhandled fault: external abort on non-linefetch (0x1008) at 0xa00b0000 [248275.252669] pgd = 893a0000 [248275.255465] [a00b0000] *pgd=88006811, *pte=30880653, *ppte=30880453 [248275.261865] Internal error: : 1008 [#1] PREEMPT SMP ARM [248275.267178] Modules linked in: [248275.270336] CPU: 0 PID: 5581 Comm: getstats Tainted: G        W       4.9.11-1.0.0 #1 [248275.278252] Hardware name: Freescale i.MX7 Dual (Device Tree) [248275.284085] task: 88620640 task.stack: 892d2000 [248275.288712] PC is at imx_rxint+0x4c/0x2ac [248275.292816] LR is at _raw_spin_lock_irqsave+0x18/0x5c [248275.297956] pc : [<804115d0>]    lr : [<80736028>]    psr: 200b0193 [248275.297956] sp : 892d3ea8  ip : 860000d8  fp : 80998848 [248275.309607] r10: 600b0193  r9 : 00000000  r8 : 80c02d00 [248275.314919] r7 : 8805f1f8  r6 : 00000001  r5 : 00002150  r4 : 88390010 [248275.321534] r3 : 128d164e  r2 : a00b0000  r1 : 00000000  r0 : 600b0193 [248275.328150] Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none [248275.335459] Control: 10c53c7d  Table: 893a006a  DAC: 00000051 [248275.341292] Process getstats (pid: 5581, stack limit = 0x892d2210) [248275.347560] Stack: (0x892d3ea8 to 0x892d4000) [248275.352006] 3ea0:                   400b0093 00000000 00000000 88390010 00002150 00005089 [248275.360274] 3ec0: 00000030 00000030 881ea100 80c11b32 80998848 8041336c 883f8340 881ea100 [248275.368542] 3ee0: 00000000 892d3f2c 00000030 80177250 00000000 00000000 00000010 80998870 [248275.376809] 3f00: 8673fc88 881ea100 881ea100 80c0a1b0 00000000 00000001 892d3fb0 88008000 [248275.385076] 3f20: 004411d4 801772f4 88620640 00000000 004411d4 881ea100 881ea160 80177368 [248275.393343] 3f40: 881ea100 881ea160 80c0a1b0 8017a7f0 80b6edec 00000000 00000030 00000000 [248275.401610] 3f60: 00000001 8017652c 80b6edec 80176a54 80c1aec0 80c034a0 a000200c 892d3fb0 [248275.409878] 3f80: a0002000 a0003000 000afb70 801014c4 00044608 200b0010 ffffffff 10c53c7d [248275.418145] 3fa0: 10c53c7d 00443934 000afb70 80736db0 00000000 00000000 004439bc 00443943 [248275.426412] 3fc0: 000afadc 00000000 00000004 0043f1f0 000b1c7c 00443934 000afb70 004411d4 [248275.434679] 3fe0: 000af430 7ec5f140 00043ee8 00044608 200b0010 ffffffff 00000000 00000000 [248275.442958] [<804115d0>] (imx_rxint) from [<8041336c>] (imx_int+0xe8/0x1b8) [248275.450016] [<8041336c>] (imx_int) from [<80177250>] (__handle_irq_event_percpu+0x9c/0x124) [248275.458460] [<80177250>] (__handle_irq_event_percpu) from [<801772f4>] (handle_irq_event_percpu+0x1c/0x58) [248275.468205] [<801772f4>] (handle_irq_event_percpu) from [<80177368>] (handle_irq_event+0x38/0x5c) [248275.477168] [<80177368>] (handle_irq_event) from [<8017a7f0>] (handle_fasteoi_irq+0xd0/0x1a8) [248275.485784] [<8017a7f0>] (handle_fasteoi_irq) from [<8017652c>] (generic_handle_irq+0x24/0x34) [248275.494488] [<8017652c>] (generic_handle_irq) from [<80176a54>] (__handle_domain_irq+0x7c/0xec) [248275.503279] [<80176a54>] (__handle_domain_irq) from [<801014c4>] (gic_handle_irq+0x48/0x8c) [248275.511722] [<801014c4>] (gic_handle_irq) from [<80736db0>] (__irq_usr+0x50/0x80) [248275.519291] Exception stack(0x892d3fb0 to 0x892d3ff8) [248275.524431] 3fa0:                                     00000000 00000000 004439bc 00443943 [248275.532698] 3fc0: 000afadc 00000000 00000004 0043f1f0 000b1c7c 00443934 000afb70 004411d4 [248275.540964] 3fe0: 000af430 7ec5f140 00043ee8 00044608 200b0010 ffffffff [248275.547671] Code: e5943070 e5942008 e2833001 e5843070 (e5925000)  [248275.553863] ---[ end trace ad4a27aeb28285be ]--- [248275.558571] Kernel panic - not syncing: Fatal exception in interrupt [248276.440014] SMP: failed to stop secondary CPUs [248276.444549] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

Similar crash on a 4.1.15 kernel with one core disabled

[33849.821632] Unhandled fault: external abort on non-linefetch (0x1008) at 0xa0100000
[33849.829310] pgd = 88740000
[33849.832023] [a0100000] *pgd=88006811, *pte=30880653, *ppte=30880453
[33849.838343] Internal error: : 1008 [#1] PREEMPT SMP ARM
[33849.843572] Modules linked in:
[33849.846648] CPU: 0 PID: 943 Comm: zigbee-stress-t Not tainted 4.1.15-1.2.7 #1
[33849.853786] Hardware name: Freescale i.MX7 Dual (Device Tree)
[33849.859537] task: 88afd7c0 ti: 886f2000 task.ti: 886f2000
[33849.864949] PC is at imx_rxint+0x54/0x2ac
[33849.868971] LR is at _raw_spin_lock_irqsave+0x18/0x5c
[33849.874029] pc : [<802fe604>] lr : [<80692f50>] psr: 20000193
[33849.874029] sp : 886f3db0 ip : 8600067c fp : 020912e4
[33849.885510] r10: 00000000 r9 : 8099e100 r8 : 00000113
[33849.890740] r7 : 8099e504 r6 : 880589e8 r5 : 00002550 r4 : 880b4410
[33849.897271] r3 : 02258618 r2 : a0100000 r1 : 00000000 r0 : 60000193
[33849.903805] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[33849.911031] Control: 10c53c7d Table: 8874006a DAC: 00000015
[33849.916781] Process zigbee-stress-t (pid: 943, stack limit = 0x886f2210)
[33849.923486] Stack: (0x886f3db0 to 0x886f4000)
[33849.927850] 3da0: 0000bb75 60000193 ffffffff 00000001
[33849.936036] 3dc0: 00000000 880b4410 00002550 00004089 00000113 00000113 88138780 809fa76a
[33849.944223] 3de0: 020912e4 802ffce8 88071800 881387e0 00000000 00000000 00000113 80066484
[33849.952409] 3e00: 020912e4 80692f50 88138780 881387e0 88071800 00000000 00000001 886f3e88
[33849.960595] 3e20: 88008000 8006657c 88138780 881387e0 809a398c 80069238 00000113 00000000
[33849.968781] 3e40: 00000113 80065b24 80997be8 80065ddc a000200c 8099e830 886f3e88 a0002000
[33849.976967] 3e60: 86008c38 885b5a80 00000002 8000941c 806932c8 60000013 ffffffff 886f3ebc
[33849.985153] 3e80: 86008c38 800132c0 880b4410 a0000013 0000d9d8 0000d9d9 880589e8 88731200
[33849.993338] 3ea0: a0000013 880b4410 86008c38 885b5a80 00000002 020912e4 00000000 886f3ed0
[33850.001525] 3ec0: 802fbc84 806932c8 60000013 ffffffff 880589e8 802fbc84 88731200 00000000
[33850.009710] 3ee0: 00000000 884e86d0 86008c38 885b5a80 00000002 802e2754 881ebe28 8012427c
[33850.017896] 3f00: 880bb015 00000005 882af214 a0000013 00000008 80692f50 00000002 00000000
[33850.026082] 3f20: 885b5a88 00000001 885b5a88 00000001 00000000 00000000 00000000 885b5a80
[33850.034268] 3f40: 881ebe28 00000000 884e86d0 86008c38 00000008 885b5a88 020912e4 800ee694
[33850.042453] 3f60: 00000000 00000000 88afdbb8 00000000 809ff0a8 88afd7c0 8000f684 886f2000
[33850.050639] 3f80: 00000000 80044f38 886f2000 8000f684 886f3fb0 0000003f 8000f684 800123d4
[33850.058825] 3fa0: 0000000b 0000000b 02091408 8000f54c 00000001 00000001 80000001 00000001
[33850.067010] 3fc0: 0000000b 0000000b 02091408 0000003f 00000000 00000000 00000000 020912e4
[33850.075196] 3fe0: 000af728 7eae2e84 0003e968 76ec78ac a0010010 0000000b 00000000 00000000
[33850.083394] [<802fe604>] (imx_rxint) from [<802ffce8>] (imx_int+0x144/0x1bc)
[33850.090457] [<802ffce8>] (imx_int) from [<80066484>] (handle_irq_event_percpu+0x78/0x134)
[33850.098646] [<80066484>] (handle_irq_event_percpu) from [<8006657c>] (handle_irq_event+0x3c/0x5c)
[33850.107530] [<8006657c>] (handle_irq_event) from [<80069238>] (handle_fasteoi_irq+0xe0/0x198)
[33850.116066] [<80069238>] (handle_fasteoi_irq) from [<80065b24>] (generic_handle_irq+0x2c/0x3c)
[33850.124687] [<80065b24>] (generic_handle_irq) from [<80065ddc>] (__handle_domain_irq+0x7c/0xec)
[33850.133396] [<80065ddc>] (__handle_domain_irq) from [<8000941c>] (gic_handle_irq+0x24/0x5c)
[33850.141758] [<8000941c>] (gic_handle_irq) from [<800132c0>] (__irq_svc+0x40/0x74)
[33850.149243] Exception stack(0x886f3e88 to 0x886f3ed0)
[33850.154303] 3e80: 880b4410 a0000013 0000d9d8 0000d9d9 880589e8 88731200
[33850.162489] 3ea0: a0000013 880b4410 86008c38 885b5a80 00000002 020912e4 00000000 886f3ed0
[33850.170671] 3ec0: 802fbc84 806932c8 60000013 ffffffff
[33850.175734] [<800132c0>] (__irq_svc) from [<806932c8>] (_raw_spin_unlock_irqrestore+0x20/0x54)
[33850.184357] [<806932c8>] (_raw_spin_unlock_irqrestore) from [<802fbc84>] (uart_close+0x1a4/0x1f0)
[33850.193244] [<802fbc84>] (uart_close) from [<802e2754>] (tty_release+0xf0/0x4b4)
[33850.200653] [<802e2754>] (tty_release) from [<800ee694>] (__fput+0x80/0x1d0)
[33850.207717] [<800ee694>] (__fput) from [<80044f38>] (task_work_run+0xac/0xe4)
[33850.214868] [<80044f38>] (task_work_run) from [<800123d4>] (do_work_pending+0x7c/0xa4)
[33850.222797] [<800123d4>] (do_work_pending) from [<8000f54c>] (work_pending+0xc/0x20)
[33850.230550] Code: e594306c e5942008 e2833001 e584306c (e592b000) 
[33850.236657] ---[ end trace cbc6694aa496ca26 ]---
[33850.241281] Kernel panic - not syncing: Fatal exception in interrupt
[33850.247642] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
-----

[arnoutdiels]

Seen this too on 4.9.144 

root@signco-baseboard:~# microcom -p /dev/ttymxc2
connected to /dev/ttymxc2
Escape character: Ctrl-\
Type the escape character followed by c to get to the menu or q to quit
▒[ 56.065073] random: crng init done
[ 56.068554] random: 7 urandom warning(s) missed due to ratelimiting
▒[ 92.803734] Unhandled fault: external abort on non-linefetch (0x1008) at 0xa08a4000
[ 92.811461] pgd = 95398000
[ 92.814210] [a08a4000] *pgd=94005811, *pte=021ec653, *ppte=021ec453
[ 92.820622] Internal error: : 1008 [#1] SMP ARM
[ 92.825183] Modules linked in: rpcsec_gss_krb5 wl18xx wlcore mac80211 cfg80211 sht3x crc8 wlcore_sdio
[ 92.834725] CPU: 0 PID: 300 Comm: microcom Not tainted 4.9.144 #567
[ 92.841029] Hardware name: Freescale i.MX6 UltraLite (Device Tree)
[ 92.847250] task: 942733c0 task.stack: 94962000
[ 92.851849] PC is at imx_rxint+0x4c/0x1fc
[ 92.855909] LR is at imx_rxint+0x1c/0x1fc
[ 92.859968] pc : [<804ec084>] lr : [<804ec054>] psr: 20080193
[ 92.859968] sp : 94963c40 ip : 00000000 fp : 94963c6c
[ 92.871494] r10: 80d6411c r9 : 9410f200 r8 : 40080193
[ 92.876761] r7 : 00000000 r6 : 00000004 r5 : 943009f8 r4 : 942d0410
[ 92.883330] r3 : 00088087 r2 : a08a4000 r1 : 00003787 r0 : 40080193
[ 92.889904] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
[ 92.897168] Control: 10c5387d Table: 9539806a DAC: 00000051
[ 92.902953] Process microcom (pid: 300, stack limit = 0x94962210)
[ 92.909086] Stack: (0x94963c40 to 0x94964000)
[ 92.913512] 3c40: 00000000 942d0410 00002640 000052a9 00000038 00000038 9410f200 80d6411c
[ 92.921753] 3c60: 94963c94 94963c70 804ee3bc 804ec044 94003380 9410f200 00000000 94963ce4
[ 92.929993] 3c80: 00000038 9410f200 94963cdc 94963c98 8016fd04 804ee2a4 94963cdc 94963ca8
[ 92.938231] 3ca0: 804f0d7c 80a8a99c 80a8a988 80a8a960 00000000 9410f200 9410f200 00000000
[ 92.946471] 3cc0: 00000001 94006000 f4a03000 946b1001 94963cfc 94963ce0 8016fdac 8016fc5c
[ 92.954708] 3ce0: 94963d2c 00000000 9410f200 9410f260 94963d1c 94963d00 8016fe34 8016fd94
[ 92.962948] 3d00: 9410f200 80d0a858 00000000 00000001 94963d3c 94963d20 801735fc 8016fdf4
[ 92.971185] 3d20: 80c55d9c 00000000 00000000 00000001 94963d4c 94963d40 8016ef08 8017354c
[ 92.979426] 3d40: 94963d74 94963d50 8016f498 8016eee8 80d21000 80d0444c f4a0200c 94963da0
[ 92.987664] 3d60: f4a02000 f4a03000 94963d9c 94963d78 801014c8 8016f440 808191b4 60080013
[ 92.995904] 3d80: ffffffff 94963dd4 00000fff 94962000 94963dfc 94963da0 8010d34c 80101484
[ 93.004142] 3da0: 942d0410 a0080013 a08a4000 00003787 00000000 943009f8 00000000 00000001
[ 93.012382] 3dc0: 00000fff 942d0410 946b1001 94963dfc 94963e00 94963df0 804e8f00 808191b4
[ 93.020621] 3de0: 60080013 ffffffff 00000051 7f000000 94963e34 94963e00 804e8f00 80819198
[ 93.028860] 3e00: a0080013 9469c400 9469c400 9469c400 00000001 946b1000 809346f4 945466c0
[ 93.037099] 3e20: a2244270 00000000 94963e8c 94963e38 804d13a8 804e8e0c 9469c58c 94962000
[ 93.045337] 3e40: 946b1000 9469c480 94962000 00000000 942733c0 80164a14 9469c590 9469c590
[ 93.053575] 3e60: 7e92d68c 9469c400 00000001 00000001 7e92d68c 94962000 00000001 00000051
[ 93.061815] 3e80: 94963ed4 94963e90 804ccd48 804d11f0 804d5624 94874040 804d11e4 945466c0
[ 93.070053] 3ea0: 946b1000 00000000 94963f78 945466c0 804ccb5c 94963f78 00000001 7e92d68c
[ 93.078293] 3ec0: 00000001 00000004 94963f44 94963ed8 8020b554 804ccb68 0000e000 80d02084
[ 93.086531] 3ee0: 94963f4c 94963ef0 8012b0f8 00000001 944f1548 00000001 9401d000 941ccdf8
[ 93.094769] 3f00: 80d02d00 ffffaeb9 0000000a 809021f8 00000000 00000000 94963ef0 00000000
[ 93.103007] 3f20: 00000001 945466c0 7e92d68c 94963f78 7e92d68c 00000001 94963f74 94963f48
[ 93.111246] 3f40: 8020c448 8020b52c 00000000 00000000 945466c0 945466c0 00000000 00000000
[ 93.119483] 3f60: 7e92d68c 00000001 94963fa4 94963f78 8020d50c 8020c3a8 00000000 00000000
[ 93.127722] 3f80: 00000001 7e92d68d 00000001 00000004 80108804 94962000 00000000 94963fa8
[ 93.135959] 3fa0: 80108620 8020d4d4 00000001 7e92d68d 00000003 7e92d68c 00000001 7e92d68d
[ 93.144198] 3fc0: 00000001 7e92d68d 00000001 00000004 0237a008 004b8000 7e92d688 7e92d60c
[ 93.152436] 3fe0: 00000000 7e92d5c4 004a5f65 76ec73f6 20080030 00000003 00000000 00000000
[ 93.160641] Backtrace:
[ 93.163191] [<804ec038>] (imx_rxint) from [<804ee3bc>] (imx_int+0x124/0x168)
[ 93.170309] r10:80d6411c r9:9410f200 r8:00000038 r7:00000038 r6:000052a9 r5:00002640
[ 93.178178] r4:942d0410 r3:00000000
[ 93.181832] [<804ee298>] (imx_int) from [<8016fd04>] (__handle_irq_event_percpu+0xb4/0x138)
[ 93.190249] r9:9410f200 r8:00000038 r7:94963ce4 r6:00000000 r5:9410f200 r4:94003380
[ 93.198061] [<8016fc50>] (__handle_irq_event_percpu) from [<8016fdac>] (handle_irq_event_percpu+0x24/0x60)
[ 93.207778] r10:946b1001 r9:f4a03000 r8:94006000 r7:00000001 r6:00000000 r5:9410f200
[ 93.215640] r4:9410f200
[ 93.218244] [<8016fd88>] (handle_irq_event_percpu) from [<8016fe34>] (handle_irq_event+0x4c/0x70)
[ 93.227159] r5:9410f260 r4:9410f200
[ 93.230798] [<8016fde8>] (handle_irq_event) from [<801735fc>] (handle_fasteoi_irq+0xbc/0x19c)
[ 93.239373] r7:00000001 r6:00000000 r5:80d0a858 r4:9410f200
[ 93.245096] [<80173540>] (handle_fasteoi_irq) from [<8016ef08>] (generic_handle_irq+0x2c/0x3c)
[ 93.253758] r7:00000001 r6:00000000 r5:00000000 r4:80c55d9c
[ 93.259482] [<8016eedc>] (generic_handle_irq) from [<8016f498>] (__handle_domain_irq+0x64/0xbc)
[ 93.268246] [<8016f434>] (__handle_domain_irq) from [<801014c8>] (gic_handle_irq+0x50/0x94)
[ 93.276659] r9:f4a03000 r8:f4a02000 r7:94963da0 r6:f4a0200c r5:80d0444c r4:80d21000
[ 93.284460] [<80101478>] (gic_handle_irq) from [<8010d34c>] (__irq_svc+0x6c/0x90)
[ 93.291980] Exception stack(0x94963da0 to 0x94963de8)
[ 93.297094] 3da0: 942d0410 a0080013 a08a4000 00003787 00000000 943009f8 00000000 00000001
[ 93.305332] 3dc0: 00000fff 942d0410 946b1001 94963dfc 94963e00 94963df0 804e8f00 808191b4
[ 93.313549] 3de0: 60080013 ffffffff
[ 93.317099] r9:94962000 r8:00000fff r7:94963dd4 r6:ffffffff r5:60080013 r4:808191b4
[ 93.324916] [<8081918c>] (_raw_spin_unlock_irqrestore) from [<804e8f00>] (uart_write+0x100/0x188)
[ 93.333855] [<804e8e00>] (uart_write) from [<804d13a8>] (n_tty_write+0x1c4/0x464)
[ 93.341401] r10:00000000 r9:a2244270 r8:945466c0 r7:809346f4 r6:946b1000 r5:00000001
[ 93.349264] r4:9469c400
[ 93.351865] [<804d11e4>] (n_tty_write) from [<804ccd48>] (tty_write+0x1ec/0x2f0)
[ 93.359324] r10:00000051 r9:00000001 r8:94962000 r7:7e92d68c r6:00000001 r5:00000001
[ 93.367186] r4:9469c400
[ 93.369789] [<804ccb5c>] (tty_write) from [<8020b554>] (__vfs_write+0x34/0x120)
[ 93.377162] r10:00000004 r9:00000001 r8:7e92d68c r7:00000001 r6:94963f78 r5:804ccb5c
[ 93.385025] r4:945466c0
[ 93.387623] [<8020b520>] (__vfs_write) from [<8020c448>] (vfs_write+0xac/0x1b8)
[ 93.394993] r9:00000001 r8:7e92d68c r7:94963f78 r6:7e92d68c r5:945466c0 r4:00000001
[ 93.402803] [<8020c39c>] (vfs_write) from [<8020d50c>] (SyS_write+0x44/0x98)
[ 93.409913] r9:00000001 r8:7e92d68c r7:00000000 r6:00000000 r5:945466c0 r4:945466c0
[ 93.417723] [<8020d4c8>] (SyS_write) from [<80108620>] (ret_fast_syscall+0x0/0x48)
[ 93.425357] r9:94962000 r8:80108804 r7:00000004 r6:00000001 r5:7e92d68d r4:00000001
[ 93.433157] Code: e5943070 e5942008 e2833001 e5843070 (e5929000)
[ 93.439300] ---[ end trace 4ea9116bce85a11a ]---
[ 93.443961] Kernel panic - not syncing: Fatal exception in interrupt
[ 93.450377] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

[magnusstenkvist]

Hi Arnout, I'm not working for the specific customer anymore but I remember that I resolved this problem (note, probably not the same as the OP) by backporting the 4.17 version of the serial/imx driver.

When looking at the stacktrace it seemed like a imx_rxint() occurs while calling uart_write(). Uwe Klein-König has provided a lot of fixes upstream and I found the following two patches from him to be of extra interest:

4377689 "serial: imx: Only handle irqs that are actually enabled"

76821e2 "serial: imx: ensure that RX irqs are off if RX is off"

Anyway, we backported the 4.17 driver that included these patches and have not had any more problems with it since.

[arnoutdiels]

Hi,

Thanks. After trying to cherry-pick some commits, I also just ended up backporting the upstream v5.2 driver (only a small change needed in a new timer api). This indeed seems to fix the crashes.

Kind regards,

Arnout

[magnusstenkvist]

Am helping out a customer who has a i.MX51 system using the imx serial driver on a 4.14.28 kernel. 
The serial port in this test setup is used for a radio modem communications link and is at regular intervals sending/receiving data (i.e. no stress test setup).
They have been running a system test for a couple of days and got a crash after ~114 hours.
Found this thread while looking into the problem and thought it might be useful to post the dump as it seems related to the original post.

<1>[410624.222991] Unhandled fault: external abort on non-linefetch (0x1008) at 0xc88e4000
<1>[410624.223021] pgd = c6da4000
<1>[410624.223031] [c88e4000] *pgd=97856811, *pte=73fbc653, *ppte=73fbc453
<0>[410624.223062] Internal error: : 1008 [#1] PREEMPT ARM
<4>[410624.223074] Modules linked in: ramoops pstore reed_solomon
<4>[410624.223107] CPU: 0 PID: 2934 Comm: SccFacade Not tainted 4.14.28-xyz600-1 #1
<4>[410624.223117] Hardware name: Freescale i.MX51 (Device Tree Support)
<4>[410624.223126] task: c7bb3600 task.stack: c6fbc000
<4>[410624.223153] PC is at imx_rxint+0x58/0x240
<4>[410624.223164] LR is at imx_rxint+0x2c/0x240
<4>[410624.223173] pc : [<c03aab90>] lr : [<c03aab64>] psr: 00000193
<4>[410624.223182] sp : c6fbdc78 ip : c6fbdc78 fp : c6fbdca4
<4>[410624.223191] r10: c6fbdd1c r9 : 00000000 r8 : 40000193
<4>[410624.223200] r7 : 00000000 r6 : 00000004 r5 : c7b2d000 r4 : c7a38010
<4>[410624.223210] r3 : 010fc8dc r2 : c88e4000 r1 : c7a38010 r0 : 00000001
<4>[410624.223222] Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
<4>[410624.223232] Control: 10c5387d Table: 96da4019 DAC: 00000051
<0>[410624.223241] Process SccFacade (pid: 2934, stack limit = 0xc6fbc208)
<0>[410624.223251] Stack: (0xc6fbdc78 to 0xc6fbe000)
<0>[410624.223263] dc60: 00000000 c7a38010
<0>[410624.223280] dc80: 00002610 000054c9 0000002f c6fbc000 00000000 c6fbdd1c c6fbdccc c6fbdca8
<0>[410624.223295] dca0: c03abed8 c03aab44 c03abd44 c7a934c0 c7857a80 0000002f 00000001 c6fbc000
<0>[410624.223311] dcc0: c6fbdd14 c6fbdcd0 c015f510 c03abd50 c0146364 c0147818 c6cd9b00 c7857a80
<0>[410624.223327] dce0: c0b57d48 c0b57d5c c6c3a8c8 c7857a80 c7857a80 00000000 00000001 c6fbddc8
<0>[410624.223343] dd00: c785e000 00000000 c6fbdd34 c6fbdd18 c015f71c c015f4b0 c6fbdd3c 00000000
<0>[410624.223358] dd20: c7857a80 00000000 c6fbdd4c c6fbdd38 c015f7b4 c015f6fc c7857a80 00000000
<0>[410624.223374] dd40: c6fbdd64 c6fbdd50 c0163310 c015f764 c016324c c0b53138 c6fbdd74 c6fbdd68
<0>[410624.223390] dd60: c015e648 c0163258 c6fbdd9c c6fbdd78 c015ecc8 c015e620 c0b5e9a4 00000000
<0>[410624.223405] dd80: 00000001 c6fbddc8 00000000 00000000 c6fbddc4 c6fbdda0 c01014d0 c015ec54
<0>[410624.223421] dda0: c03a6904 60000013 ffffffff c6fbddfc c7a38010 c6fbc000 c6fbde4c c6fbddc8
<0>[410624.223437] ddc0: c010d50c c0101464 00000000 00000000 c88e4000 a0000013 00000000 c7b2d000
<0>[410624.223453] dde0: 0000001e c6c1b000 c7a38010 c6cb3c1e 00000000 c6fbde4c c6fbdde8 c6fbde18
<0>[410624.223469] de00: c03aaa20 c03a6904 60000013 ffffffff 00000051 bf000000 c6fbde70 a0000013
<0>[410624.223484] de20: c6fbde4c c6c1b000 0000001e c6cb3c00 c073fa0c c71d0c00 c8b83264 00000051
<0>[410624.223500] de40: c6fbdea4 c6fbde50 c038e0f4 c03a6810 c6c1b140 c6fbc000 c6cb3c00 c6c1b058
<0>[410624.223516] de60: c78de000 00000000 c7bb3600 c01516f4 c6fbfd3c c6c1b140 b6108a2e 0000001e
<0>[410624.223532] de80: c6c1b000 0000001e b6108a2e c6fbc000 0000001e 00000051 c6fbdeec c6fbdea8
<0>[410624.223547] dea0: c0389e4c c038dce0 c6cd9b00 c6d6d540 c038dcd4 c71d0c00 c6cb3c00 00000000
<0>[410624.223563] dec0: c06358b0 c71d0c00 c6fbdf80 b6108a2e c6fbdf80 0000001e c6fbc000 00000000
<0>[410624.223579] dee0: c6fbdf5c c6fbdef0 c021f3d4 c0389cf4 0000005d c6fbdf00 0193aea8 0000005d
<0>[410624.223594] df00: 00000001 00000000 00000000 c6fbdf00 00000000 c023f0d8 00000000 c71d0c00
<0>[410624.223610] df20: c6fbdf54 c6fbdf30 c023f0d8 c021f5ac c6cc0600 b6108600 b6108a2e 0000001e
<0>[410624.223626] df40: c71d0c00 b6108a2e c6fbdf80 c0108344 c6fbdf7c c6fbdf60 c021f5ec c021f3a8
<0>[410624.223641] df60: c71d0c00 c71d0c01 0000001e b6108a2e c6fbdfa4 c6fbdf80 c021f790 c021f548
<0>[410624.223657] df80: 00000000 00000000 b6108600 b6108a2e 0000042c 00000004 00000000 c6fbdfa8
<0>[410624.223673] dfa0: c0108160 c021f748 b6108600 b6108a2e 0000001a b6108a2e 0000001e 00000000
<0>[410624.223689] dfc0: b6108600 b6108a2e 0000042c 00000004 b2c21f90 00000000 00000002 b2c21f8c
<0>[410624.223704] dfe0: 00000000 b2c21bf0 b2c22910 b6db4238 80000010 0000001a 00000000 00000000
<4>[410624.223735] [<c03aab90>] (imx_rxint) from [<c03abed8>] (imx_int+0x194/0x2a4)
<4>[410624.223763] [<c03abed8>] (imx_int) from [<c015f510>] (__handle_irq_event_percpu+0x6c/0x24c)
<4>[410624.223783] [<c015f510>] (__handle_irq_event_percpu) from [<c015f71c>] (handle_irq_event_percpu+0x2c/0x68)
<4>[410624.223800] [<c015f71c>] (handle_irq_event_percpu) from [<c015f7b4>] (handle_irq_event+0x5c/0x94)
<4>[410624.223820] [<c015f7b4>] (handle_irq_event) from [<c0163310>] (handle_level_irq+0xc4/0x168)
<4>[410624.223838] [<c0163310>] (handle_level_irq) from [<c015e648>] (generic_handle_irq+0x34/0x44)
<4>[410624.223854] [<c015e648>] (generic_handle_irq) from [<c015ecc8>] (__handle_domain_irq+0x80/0xec)
<4>[410624.223872] [<c015ecc8>] (__handle_domain_irq) from [<c01014d0>] (tzic_handle_irq+0x78/0x9c)
<4>[410624.223890] [<c01014d0>] (tzic_handle_irq) from [<c010d50c>] (__irq_svc+0x6c/0xa8)
<4>[410624.223899] Exception stack(0xc6fbddc8 to 0xc6fbde10)
<4>[410624.223912] ddc0: 00000000 00000000 c88e4000 a0000013 00000000 c7b2d000
<4>[410624.223928] dde0: 0000001e c6c1b000 c7a38010 c6cb3c1e 00000000 c6fbde4c c6fbdde8 c6fbde18
<4>[410624.223939] de00: c03aaa20 c03a6904 60000013 ffffffff
<4>[410624.223955] [<c010d50c>] (__irq_svc) from [<c03a6904>] (uart_write+0x100/0x1ac)
<4>[410624.223981] [<c03a6904>] (uart_write) from [<c038e0f4>] (n_tty_write+0x420/0x464)
<4>[410624.223999] [<c038e0f4>] (n_tty_write) from [<c0389e4c>] (tty_write+0x164/0x334)
<4>[410624.224019] [<c0389e4c>] (tty_write) from [<c021f3d4>] (__vfs_write+0x38/0x138)
<4>[410624.224036] [<c021f3d4>] (__vfs_write) from [<c021f5ec>] (vfs_write+0xb0/0x10c)
<4>[410624.224051] [<c021f5ec>] (vfs_write) from [<c021f790>] (SyS_write+0x54/0xb0)
<4>[410624.224072] [<c021f790>] (SyS_write) from [<c0108160>] (ret_fast_syscall+0x0/0x54)
<0>[410624.224091] Code: e5943070 e5942004 e2833001 e5843070 (e5929000)
<4>[410624.224106] ---[ end trace 63db20977fee9aff ]---
<0>[410624.230968] Kernel panic - not syncing: Fatal exception in interrupt

[foosechek]

HI Anita,

The developer came back with a patch that I don't have much confidence in, but I need to have you check it out anyway. They reported that it still failed on 4.1.15, but passed on 4.9  (I did tell them 4.9 was not failing on the Sabre).   

-Ben

[anitathomas]

thanks Ben, will try this with both 4.9 and 4.1.15 kernels. Will keep you posted

[foosechek]

Hi Anita,

I know you guys have been focused on the new project, but did you get a chance to run your re-pro test with the patch?

Thanks,

-Ben

[anitathomas]

Ben, I know its been a while. But we will need to revisit this. We are still seeing the crash on 4.9.88 and 4.1.15 with the patch

[foosechek]

Hi Anita,  

Are the details of the 4.9.88 crash the same as the 4.1.15?  If not, please add a kernel log to this thread for 4.9.88.

Also, I think last time we spoke about this you mentioned that the 4.1.15 time-to-crash was much shorter than the 4.9.88, can you just give me an estimate of the typical failure time so the AE can wait long enough during test.?

Thanks

-Ben

[anitathomas]

The traces with 4.9.88 kernel looks pretty much the same on both 7d and 6ull. QA saw the crash anywhere between 1- 6hrs, when running zigbee uart stress tests. The foll trace is from 7d

 uname -a Linux MFG-BHBS11804PG003582 4.9.88-2.0.0 #1 SMP PREEMPT Wed Nov 7 15:25:32 UTC 2018 armv7l GNU/Linux

[ 5697.733402] Unhandled fault: external abort on non-linefetch (0x1008) at 0xa00b0000 [ 5697.741074] pgd = 80004000 [ 5697.743784] [a00b0000] *pgd=88006811, *pte=30880653, *ppte=30880453 [ 5697.750098] Internal error: : 1008 [#1] PREEMPT SMP ARM [ 5697.755324] Modules linked in: [ 5697.758397] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W       4.9.88-2.0.0 #1 [ 5697.766053] Hardware name: Freescale i.MX7 Dual (Device Tree) [ 5697.771800] task: 80c06400 task.stack: 80c00000 [ 5697.776341] PC is at imx_rxint+0x58/0x298 [ 5697.780356] LR is at _raw_spin_lock_irqsave+0x18/0x5c [ 5697.785411] pc : [<80409154>]    lr : [<8072c2f8>]    psr: 200f0193 [ 5697.785411] sp : 80c01e20  ip : 860000d8  fp : 883ed810 [ 5697.796890] r10: 8099e5a8  r9 : 80c4b3a2  r8 : 00000004 [ 5697.802116] r7 : 80c02d00  r6 : 00000000  r5 : 8805f1f8  r4 : 00000001 [ 5697.808645] r3 : 000011df  r2 : 007b0c62  r1 : a00b0000  r0 : 400f0193 [ 5697.815176] Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none [ 5697.822399] Control: 10c53c7d  Table: 8931006a  DAC: 00000051 [ 5697.828147] Process swapper/0 (pid: 0, stack limit = 0x80c00210) [ 5697.834155] Stack: (0x80c01e20 to 0x80c02000) [ 5697.838518] 1e20: 400f0193 00000000 80c032e4 883ed810 00002650 00005089 00000030 00000030 [ 5697.846701] 1e40: 80c4b3a2 8099e5a8 8099e580 8040af74 883faa80 881e8800 00000000 80c01ea4 [ 5697.854883] 1e60: 00000030 8017771c 80c01e7c 89365140 9fa24b38 881e8800 888e9d4a 881e8800 [ 5697.863064] 1e80: 881e8800 80c0a110 00000000 00000001 80c01f28 88008000 80c03158 80177804 [ 5697.871246] 1ea0: 9c5ed6a6 00000000 80c03158 881e8800 881e8860 80177878 881e8800 881e8860 [ 5697.879428] 1ec0: 80c0a110 8017acc0 80b6ee4c 00000000 00000030 00000000 00000001 80176a28 [ 5697.887610] 1ee0: 80b6ee4c 80176f3c 80c034a0 80c1ab00 a000200c a0002000 80c01f28 a0003000 [ 5697.895791] 1f00: 00000000 8010144c 805000a0 200f0013 ffffffff 80c01f5c 9b4bf662 80c00000 [ 5697.903973] 1f20: 00000000 8072cc8c 00000000 00000002 00000001 80c00000 9b5fc89a 0000052e [ 5697.912154] 1f40: 9fa24000 00000001 9b4bf662 0000052e 00000000 80c03158 00000000 80c01f78 [ 5697.920337] 1f60: 8072c624 805000a0 200f0013 ffffffff 00000051 00000000 9fa24000 ffffe000 [ 5697.928518] 1f80: 80c030ec 00000001 80c03150 9fa24000 80c097ac 8099c598 80c03158 80168d1c [ 5697.936700] 1fa0: 80c030c0 80b6eff8 809d9450 80c4b39f 00000000 ffffffff 00000000 80b00c5c [ 5697.944882] 1fc0: ffffffff ffffffff 00000000 80b00698 00000000 80b55a30 80c5f294 80c030d8 [ 5697.953064] 1fe0: 80b55a2c 80c076a0 8000406a 410fc075 00000000 8000807c 00000000 00000000 [ 5697.961260] [<80409154>] (imx_rxint) from [<8040af74>] (imx_int+0xe8/0x1b8) [ 5697.968232] [<8040af74>] (imx_int) from [<8017771c>] (__handle_irq_event_percpu+0x50/0x11c) [ 5697.976592] [<8017771c>] (__handle_irq_event_percpu) from [<80177804>] (handle_irq_event_percpu+0x1c/0x58) [ 5697.986252] [<80177804>] (handle_irq_event_percpu) from [<80177878>] (handle_irq_event+0x38/0x5c) [ 5697.995132] [<80177878>] (handle_irq_event) from [<8017acc0>] (handle_fasteoi_irq+0xd0/0x1a0) [ 5698.003665] [<8017acc0>] (handle_fasteoi_irq) from [<80176a28>] (generic_handle_irq+0x24/0x34) [ 5698.012283] [<80176a28>] (generic_handle_irq) from [<80176f3c>] (__handle_domain_irq+0x7c/0xec) [ 5698.020989] [<80176f3c>] (__handle_domain_irq) from [<8010144c>] (gic_handle_irq+0x48/0x8c) [ 5698.029347] [<8010144c>] (gic_handle_irq) from [<8072cc8c>] (__irq_svc+0x6c/0xa8) [ 5698.036830] Exception stack(0x80c01f28 to 0x80c01f70) [ 5698.041886] 1f20:                   00000000 00000002 00000001 80c00000 9b5fc89a 0000052e [ 5698.050067] 1f40: 9fa24000 00000001 9b4bf662 0000052e 00000000 80c03158 00000000 80c01f78 [ 5698.058247] 1f60: 8072c624 805000a0 200f0013 ffffffff [ 5698.063310] [<8072cc8c>] (__irq_svc) from [<805000a0>] (cpuidle_enter_state+0x13c/0x2cc) [ 5698.071411] [<805000a0>] (cpuidle_enter_state) from [<80168d1c>] (cpu_startup_entry+0x168/0x228) [ 5698.080205] [<80168d1c>] (cpu_startup_entry) from [<80b00c5c>] (start_kernel+0x378/0x384) [ 5698.088389] Code: e59b2070 e59b1008 e2822001 e58b2070 (e591a000) [ 5698.094494] ---[ end trace 935022bc8e4fb5d1 ]--- [ 5698.099114] Kernel panic - not syncing: Fatal exception in interrupt [ 5698.980472] SMP: failed to stop secondary CPUs [ 5698.984920] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

[anitathomas]

Also do note these tests were run with dma off

[foosechek]

Got it. I'm using my zigbee uart stress simulator, with the same parameters from last time.  Using 4.9 in the past on the 7D EVK, both my testing and the work done by our AE was unable to repro after 72 of test.   This would leave me to believe that the simulation is not accurate enough, or that there is another contributing at the hardware/OS level.

I'll make the test more aggressive on the UART, right now it taxes the interface at about 10% total possible bandwidth.

[anitathomas]

Still seeing the crash with 4.1.15

[Bio_TICFSL]

Hi Anita,

Which serial are you trying to use? it is kernel enable? do you check your device tree?

Regards