topic Re: Synchronous External Abort: synchronous external abort in i.MX Processors

Synchronous External Abort: synchronous external abort

xiaojuan_qi — Mon, 06 Dec 2021 11:32:51 GMT

Hello,

External abort happened during our test.

Please help to check.

Linux kernel：Linux version 4.14.98

HW platform: NXP IMX8DX Abort detail is as below. Thank you.

< 36.833886> 2019-03-31 16:00:26 [ 30.147569] [c1: kworker/1:2H: 1805]Timeout for IPC response! < 36.941883> 2019-03-31 16:00:26 [ 30.255566] [c1: kworker/1:2H: 1805]Timeout for IPC response! < 36.951571> 2019-03-31 16:00:26 [ 30.265255] [c1: kworker/1:2H: 1805]Failed power operation on resource 248 sc_err 3 < 36.967904> 2019-03-31 16:00:26 [ 30.281587] [c1: kworker/1:2H: 1805]Synchronous External Abort: synchronous external abort (0x96000210) at 0xffff00000d4e0024 < 36.982979> 2019-03-31 16:00:26 [ 30.296662] [c1: kworker/1:2H: 1805]Internal error: : 96000210 [#1] PREEMPT SMP < 36.996409> 2019-03-31 16:00:26 [ 30.310092] [c1: kworker/1:2H: 1805]Modules linked in: < 37.006710> 2019-03-31 16:00:26 [ 30.320393] [c1: kworker/1:2H: 1805] crc32_ce < 37.017528> 2019-03-31 16:00:26 [ 30.331211] [c1: kworker/1:2H: 1805] crct10dif_ce < 37.043174> 2019-03-31 16:00:26 [ 30.356858] [c1: kworker/1:2H: 1805]CPU: 1 PID: 1805 Comm: kworker/1:2H Not tainted 4.14.98-akso-v1.0+g9cb413aec97a #1 < 37.051201> 2019-03-31 16:00:26 [ 30.364884] [c1: kworker/1:2H: 1805]Hardware name: Hirain i.MX8DX AKSO (DT) < 37.059340> 2019-03-31 16:00:26 [ 30.373024] [c1: kworker/1:2H: 1805]Workqueue: kblockd blk_mq_run_work_fn < 37.082539> 2019-03-31 16:00:26 [ 30.396223] [c1: kworker/1:2H: 1805]task: ffff80001923c380 task.stack: ffff00000dd58000 < 37.095275> 2019-03-31 16:00:26 [ 30.408958] [c1: kworker/1:2H: 1805]PC is at esdhc_readl_le+0xc/0x150 < 37.107834> 2019-03-31 16:00:26 [ 30.421517] [c1: kworker/1:2H: 1805]LR is at sdhci_send_command+0xb0/0xab8 < 37.123527> 2019-03-31 16:00:26 [ 30.437211] [c1: kworker/1:2H: 1805]pc : [<ffff0000089c29ec>] lr : [<ffff0000089b5900>] pstate: 200001c5 < 37.137302> 2019-03-31 16:00:26 [ 30.450985] [c1: kworker/1:2H: 1805]sp : ffff00000dd5bad0 < 37.149519> 2019-03-31 16:00:26 [ 30.463202] [c1: kworker/1:2H: 1805]x29: ffff00000dd5bad0 < 37.162159> 2019-03-31 16:00:26 [ 30.475843] [c1: kworker/1:2H: 1805]x28: 0000000000000000 < 37.188585> 2019-03-31 16:00:26 [ 30.502268] [c1: kworker/1:2H: 1805]x27: 0000000000000001 < 37.199845> 2019-03-31 16:00:26 [ 30.513529] [c1: kworker/1:2H: 1805]x26: 0000000000000000 < 37.216438> 2019-03-31 16:00:26 [ 30.530122] [c1: kworker/1:2H: 1805]x25: ffff8000191f8a00 < 37.225265> 2019-03-31 16:00:26 [ 30.538946] [c1: kworker/1:2H: 1805]x24: ffff800018ec9000 < 37.244307> 2019-03-31 16:00:26 [ 30.557991] [c1: kworker/1:2H: 1805]x23: 0000000000418958 < 37.261129> 2019-03-31 16:00:26 [ 30.574813] [c1: kworker/1:2H: 1805]x22: 0000000000000003 < 37.264538> 2019-03-31 16:00:26 [ 30.578222] [c1: kworker/1:2H: 1805]x21: ffff8000174a6388 < 37.267949> 2019-03-31 16:00:26 [ 30.581633] [c1: kworker/1:2H: 1805]x20: 000000000000000b < 37.271364> 2019-03-31 16:00:26 [ 30.585047] [c1: kworker/1:2H: 1805]x19: ffff800018ec9600 < 37.274778> 2019-03-31 16:00:26 [ 30.588462] [c1: kworker/1:2H: 1805]x18: 00000000000007e2 < 37.278194> 2019-03-31 16:00:26 [ 30.591877] [c1: kworker/1:2H: 1805]x17: 0000ffffa405a6a0 < 37.281605> 2019-03-31 16:00:26 [ 30.595289] [c1: kworker/1:2H: 1805]x16: ffff000008239e60 < 37.285014> 2019-03-31 16:00:26 [ 30.598698] [c1: kworker/1:2H: 1805]x15: 0000000000000013 < 37.288425> 2019-03-31 16:00:26 [ 30.602108] [c1: kworker/1:2H: 1805]x14: 000000000000003b < 37.291831> 2019-03-31 16:00:26 [ 30.605515] [c1: kworker/1:2H: 1805]x13: 0000000000000000 < 37.295244> 2019-03-31 16:00:26 [ 30.608928] [c1: kworker/1:2H: 1805]x12: 00000007035eee5d < 37.298653> 2019-03-31 16:00:26 [ 30.612336] [c1: kworker/1:2H: 1805]x11: 0000000000000000 < 37.302066> 2019-03-31 16:00:26 [ 30.615749] [c1: kworker/1:2H: 1805]x10: 00000000000009b0 < 37.305480> 2019-03-31 16:00:26 [ 30.619163] [c1: kworker/1:2H: 1805]x9 : ffff800016420a88 < 37.308894> 2019-03-31 16:00:26 [ 30.622578] [c1: kworker/1:2H: 1805]x8 : ffff8000174a6458 < 37.312309> 2019-03-31 16:00:26 [ 30.625993] [c1: kworker/1:2H: 1805]x7 : 0000000000000001 < 37.315722> 2019-03-31 16:00:26 [ 30.629406] [c1: kworker/1:2H: 1805]x6 : 0000000000000100 < 37.319138> 2019-03-31 16:00:26 [ 30.632821] [c1: kworker/1:2H: 1805]x5 : 000000000000ffff < 37.322549> 2019-03-31 16:00:26 [ 30.636233] [c1: kworker/1:2H: 1805]x4 : 0000000000000000 < 37.325956> 2019-03-31 16:00:26 [ 30.639639] [c1: kworker/1:2H: 1805]x3 : ffff800018ec9600 < 37.329360> 2019-03-31 16:00:26 [ 30.643044] [c1: kworker/1:2H: 1805]x2 : ffff00000d4e0024 < 37.332768> 2019-03-31 16:00:26 [ 30.646452] [c1: kworker/1:2H: 1805]x1 : 0000000000000024 < 37.336180> 2019-03-31 16:00:26 [ 30.649863] [c1: kworker/1:2H: 1805]x0 : ffff800018ec9600 < 37.339589> 2019-03-31 16:00:26 [ 30.653272] [c1: kworker/1:2H: 1805]Process kworker/1:2H (pid: 1805, stack limit = 0xffff00000dd58000) < 37.346822> 2019-03-31 16:00:26 [ 30.660506] [c1: kworker/1:2H: 1805]Call trace: < 37.349276> 2019-03-31 16:00:26 [ 30.662959] [c1: kworker/1:2H: 1805]Exception stack(0xffff00000dd5b990 to 0xffff00000dd5bad0) < 37.355731> 2019-03-31 16:00:26 [ 30.669414] [c1: kworker/1:2H: 1805]b980: ffff800018ec9600 0000000000000024 < 37.363575> 2019-03-31 16:00:26 [ 30.677259] [c1: kworker/1:2H: 1805]b9a0: ffff00000d4e0024 ffff800018ec9600 0000000000000000 000000000000ffff < 37.371420> 2019-03-31 16:00:26 [ 30.685103] [c1: kworker/1:2H: 1805]b9c0: 0000000000000100 0000000000000001 ffff8000174a6458 ffff800016420a88 < 37.379267> 2019-03-31 16:00:26 [ 30.692950] [c1: kworker/1:2H: 1805]b9e0: 00000000000009b0 0000000000000000 00000007035eee5d 0000000000000000 < 37.387110> 2019-03-31 16:00:26 [ 30.700793] [c1: kworker/1:2H: 1805]ba00: 000000000000003b 0000000000000013 ffff000008239e60 0000ffffa405a6a0 < 37.394955> 2019-03-31 16:00:26 [ 30.708638] [c1: kworker/1:2H: 1805]ba20: 00000000000007e2 ffff800018ec9600 000000000000000b ffff8000174a6388 < 37.402799> 2019-03-31 16:00:26 [ 30.716483] [c1: kworker/1:2H: 1805]ba40: 0000000000000003 0000000000418958 ffff800018ec9000 ffff8000191f8a00 < 37.410646> 2019-03-31 16:00:26 [ 30.724329] [c1: kworker/1:2H: 1805]ba60: 0000000000000000 0000000000000001 0000000000000000 ffff00000dd5bad0 < 37.418489> 2019-03-31 16:00:26 [ 30.732173] [c1: kworker/1:2H: 1805]ba80: ffff0000089b5900 ffff00000dd5bad0 ffff0000089c29ec 00000000200001c5 < 37.426334> 2019-03-31 16:00:26 [ 30.740018] [c1: kworker/1:2H: 1805]baa0: ffff8000174a62c8 ffff800018f4c800 ffffffffffffffff ffff00000dd5bbee < 37.434176> 2019-03-31 16:00:26 [ 30.747859] [c1: kworker/1:2H: 1805]bac0: ffff00000dd5bad0 ffff0000089c29ec < 37.439074> 2019-03-31 16:00:26 [ 30.752757] [c1: kworker/1:2H: 1805][<ffff0000089c29ec>] esdhc_readl_le+0xc/0x150 < 37.444484> 2019-03-31 16:00:26 [ 30.758167] [c1: kworker/1:2H: 1805][<ffff0000089b6370>] sdhci_request+0x68/0xe8 < 37.449810> 2019-03-31 16:00:26 [ 30.763493] [c1: kworker/1:2H: 1805][<ffff000008999518>] __mmc_start_request+0x78/0x1f0 < 37.455742> 2019-03-31 16:00:26 [ 30.769425] [c1: kworker/1:2H: 1805][<ffff00000899a040>] mmc_start_request+0x60/0x88 < 37.461414> 2019-03-31 16:00:26 [ 30.775097] [c1: kworker/1:2H: 1805][<ffff0000089adecc>] mmc_blk_mq_issue_rq+0x394/0x3c0 < 37.467433> 2019-03-31 16:00:26 [ 30.781117] [c1: kworker/1:2H: 1805][<ffff0000089aea50>] mmc_mq_queue_rq+0x100/0x258 < 37.473108> 2019-03-31 16:00:26 [ 30.786791] [c1: kworker/1:2H: 1805][<ffff00000847d64c>] blk_mq_dispatch_rq_list+0x31c/0x3a0 < 37.479477> 2019-03-31 16:00:26 [ 30.793160] [c1: kworker/1:2H: 1805][<ffff000008481f04>] blk_mq_sched_dispatch_requests+0x1ac/0x228 < 37.486453> 2019-03-31 16:00:26 [ 30.800137] [c1: kworker/1:2H: 1805][<ffff00000847ba84>] __blk_mq_run_hw_queue+0xa4/0x118 < 37.492559> 2019-03-31 16:00:26 [ 30.806243] [c1: kworker/1:2H: 1805][<ffff00000847bb58>] blk_mq_run_work_fn+0x60/0x70 < 37.498323> 2019-03-31 16:00:26 [ 30.812007] [c1: kworker/1:2H: 1805][<ffff0000080c7ae8>] process_one_work+0x1d8/0x4a0 < 37.504078> 2019-03-31 16:00:26 [ 30.817762] [c1: kworker/1:2H: 1805][<ffff0000080c7df8>] worker_thread+0x48/0x470 < 37.509491> 2019-03-31 16:00:26 [ 30.823174] [c1: kworker/1:2H: 1805][<ffff0000080ce078>] kthread+0x130/0x138 < 37.514470> 2019-03-31 16:00:26 [ 30.828153] [c1: kworker/1:2H: 1805][<ffff000008084ed0>] ret_from_fork+0x10/0x18 < 37.519795> 2019-03-31 16:00:26 [ 30.833478] [c1: kworker/1:2H: 1805]Code: 17ffffc5 f9400c02 aa0003e3 8b21c042 (b9400040) < 37.525908> 2019-03-31 16:00:26 [ 30.839591] [c1: kworker/1:2H: 1805]---[ end trace eef49fda263e0196 ]--- < 37.530539> 2019-03-31 16:00:26 [ 30.844222] [c1: kworker/1:2H: 1805]Kernel panic - not syncing: Fatal exception < 37.535779> 2019-03-31 16:00:26 [ 30.849462] [c1: kworker/1:2H: 1805]SMP: stopping secondary CPUs < 37.539717> 2019-03-31 16:00:26 [ 30.853400] [c1: kworker/1:2H: 1805]CPU: 1 PID: 1 Comm: systemd State: S (sleeping) < 37.545296> 2019-03-31 16:00:26 [ 30.858980] [c1: kworker/1:2H: 1805]Call trace: < 37.547751> 2019-03-31 16:00:26 [ 30.861434] [c1: kworker/1:2H: 1805][<ffff000008085ebc>] __switch_to+0x94/0xd8 < 37.552905> 2019-03-31 16:00:26 [ 30.866589] [c1: kworker/1:2H: 1805][<ffff000008df721c>] __schedule+0x19c/0x6f8 < 37.558141> 2019-03-31 16:00:26 [ 30.871824] [c1: kworker/1:2H: 1805][<ffff000008df77b0>] schedule+0x38/0xa0 < 37.563035> 2019-03-31 16:00:26 [ 30.876719] [c1: kworker/1:2H: 1805][<ffff000008dfb234>] schedule_hrtimeout_range_clock+0x84/0xe8 < 37.569834> 2019-03-31 16:00:26 [ 30.883517] [c1: kworker/1:2H: 1805][<ffff000008dfb2a8>] schedule_hrtimeout_range+0x10/0x18 < 37.576118> 2019-03-31 16:00:26 [ 30.889801] [c1: kworker/1:2H: 1805][<ffff00000828a354>] SyS_epoll_wait+0x2d4/0x400 < 37.581701> 2019-03-31 16:00:26 [ 30.895385] [c1: kworker/1:2H: 1805][<ffff00000828a58c>] SyS_epoll_pwait+0x10c/0x148 < 37.587371> 2019-03-31 16:00:26 [ 30.901054] [c1: kworker/1:2H: 1805]Exception stack(0xffff00000806bec0 to 0xffff00000806c000) < 37.593829> 2019-03-31 16:00:26 [ 30.907512] [c1: kworker/1:2H: 1805]bec0: 0000000000000004 0000fffffef99d70 0000000000000046 0000000000003a98 < 37.601674> 2019-03-31 16:00:26 [ 30.915357] [c1: kworker/1:2H: 1805]bee0: 0000000000000000 0000000000000008 8080808080808080 fefefefefefefeff < 37.609520> 2019-03-31 16:00:26 [ 30.923204] [c1: kworker/1:2H: 1805]bf00: 0000000000000016 0000000000000001 0101010101010101 0000000000000000 < 37.617363> 2019-03-31 16:00:26 [ 30.931047] [c1: kworker/1:2H: 1805]bf20: 0000000000000018 000000005ca0e414 003114720c000000 000023e7e5000000 < 37.625208> 2019-03-31 16:00:26 [ 30.938892] [c1: kworker/1:2H: 1805]bf40: 0000ffffb9405bc0 0000ffffb8f02138 0000ffffb8fa1a70 0000000000000004 < 37.633053> 2019-03-31 16:00:26 [ 30.946736] [c1: kworker/1:2H: 1805]bf60: 0000ffffb9407000 20c49ba5e353f7cf 0000fffffef99d70 0000000000000001 < 37.640900> 2019-03-31 16:00:26 [ 30.954584] [c1: kworker/1:2H: 1805]bf80: 0000aaaac31adbd8 0000aaaac31ad1a8 0000fffffef9a370 0000aaaac31abb00 < 37.648743> 2019-03-31 16:00:26 [ 30.962426] [c1: kworker/1:2H: 1805]bfa0: 00000000000009b7 0000fffffef99d30 0000ffffb92c6d54 0000fffffef99d30 < 37.656587> 2019-03-31 16:00:26 [ 30.970271] [c1: kworker/1:2H: 1805]bfc0: 0000ffffb8f01f90 0000000000000000 0000000000000004 0000000000000016 < 37.664433> 2019-03-31 16:00:26 [ 30.978117] [c1: kworker/1:2H: 1805]bfe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 < 37.672278> 2019-03-31 16:00:26 [ 30.985962] [c1: kworker/1:2H: 1805][<ffff000008083ac0>] el0_svc_naked+0x34/0x38 < 37.677602> 2019-03-31 16:00:26 [ 30.991286] [c1: kworker/1:2H: 1805]CPU: 0 PID: 2 Comm: kthreadd State: S (sleeping)

Re: Synchronous External Abort: synchronous external abort

igorpadykov — Tue, 07 Dec 2021 00:13:20 GMT

Hi Xiaojuan

such error may happen due to incompatible revisions of software components

(they are described in Release Notes for each BSP on below link), for example scfw from one linux

release, uboot from other, rootfs binaries or other firmware binaries from some another third linux release

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applications-processors:IMXLINUX?tab=In-Depth_Tab

In general one can try to debug it to find exact place in codes where error happens using

AN12631: Normal and Secure Debug for i.MX8/8X Family of Applications Processors – Application Note

For "Timeout for IPC response!" may be useful to check sect.10.3.2.1 RPC/IPC Reset Request sc_fw_api_qx_b0.pdf document included in SCFW Porting Kit 1.10.0

Best regards
igor

Re: Synchronous External Abort: synchronous external abort

xiaojuan_qi — Wed, 08 Dec 2021 04:07:28 GMT

Dear igor,

Thanks a lot for your reply.

I had checked the release note, it seems our uboot version and linux kernel version is just the same as release note. For other binaries and firmware, I am not sure how to check whether they are imcompatible.

And I also read AN12631: Normal and Secure Debug for i.MX8/8X Family of Applications Processors – Application Note, but we don't have core dump file. We only have attached log file.

Please share your opinion.

Best regards

Xiaojuan Qi.

Re: Synchronous External Abort: synchronous external abort

igorpadykov — Wed, 08 Dec 2021 05:45:28 GMT

>For other binaries and firmware, I am not sure how to check whether they are imcompatible.

from Linux 5.10.52_2.1.0 i.MX Linux Release Notes

From dump.zip "U-Boot 2018.03" was used (it should be used with Linux 4.14.98_2.3.0), it is very old

software and not more supported. May be suggested to use latest L5.x kernels described on

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applications-processors:IMXLINUX?tab=In-Depth_Tab

Best regards
igor

Re: Synchronous External Abort: synchronous external abort

xiaojuan_qi — Wed, 08 Dec 2021 09:22:45 GMT

Dear igor,

Linux 4.14.98_2.3.0 is just the linux version we are using.

I am not sure whether we can update to Linux 5.10.52_2.1.0.

Is there any other thing I can try based on Linux 4.14.98_2.3.0?

Thank you.

Xiaojuan Qi

Re: Synchronous External Abort: synchronous external abort

igorpadykov — Wed, 08 Dec 2021 10:50:16 GMT

unfortunately Linux 4.14.98_2.3.0 (and any software based on it) is not more supported,

support for outdated releases can be provided with https://contact.nxp.com/new-prof-svcs-sw-tech

Best regards
igor