Unhandled fault: imprecise external abort (0x1406) at 0x7669d004

alaganraj · ‎06-20-2023

Hello,

We are seeing "Unhandled fault: imprecise external abort (0x1406)" error on iMX6 while running sleep cycle(suspend/resume). It occurs ~1000 iterations.

problem is, It just print below logs then reboots.

Line 138581: kern.alert: [87658.479069] Unhandled fault: imprecise external abort (0x1406) at 0x7669d004
Line 138582: kern.alert: [87658.486144] pgd = a73d4000
Line 138583: kern.alert: [87658.488856] [7669d004] *pgd=00000000

Not sure which is causing this error. is there any kernel flag to get more info ?

I've referred similar issues reported in this forum, all are having call stack, which are missing in our case.

https://community.nxp.com/t5/i-MX-Processors/Linux-kernel-crashes-at-spin-unlock-irq-when-suspending...

Any input to debug this issue are very much appreciated.

Here is full log:

SOC: iMX6

Line 135850: kern.info: [87403.394608] brcmfmac: brcmf_fw_map_chip_to_name: using brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
Line 135851: kern.info: [87403.730927] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: May 2 2019 02:46:17 version 7.45.189 (r714228 CY) FWID 01-e1db26e2
Line 135852: kern.err: [87403.737170] brcmfmac: brcmf_sdio_ulp_preinit: ulp_sdioctrl iovar returned err = -23
Line 135853: kern.info: [87403.793922] brcmfmac mmc1:0001:1 eth1: renamed from wlan0
Line 135854: kern.warn: [87404.973969] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Line 136018: kern.info: [87407.009258] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
Line 136106: kern.info: [87412.405781] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Line 136809: kern.err: [87486.732448] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing to do.
Line 136810: kern.err: [87486.739733] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-5)
Line 136811: kern.info: [87486.925587] mmc1: card 0001 removed
Line 136812: kern.info: [87487.218723] suspend.sh (21183): drop_caches: 3
Line 136813: kern.info: [87488.249903] mmc0: card 59b4 removed
Line 136814: kern.info: [87489.366181] PM: suspend entry (deep)
Line 136815: kern.info: [87489.366193] PM: Syncing filesystems ... done.
Line 136816: kern.info: [87489.371848] Freezing user space processes ... (elapsed 0.002 seconds) done.
Line 136817: kern.info: [87489.374069] OOM killer disabled.
Line 136818: kern.info: [87489.374080] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
Line 136819: kern.info: [87489.454280] PM: suspend devices took 0.080 seconds
Line 136820: kern.err: [87489.456465] eagle-pm eagle-pm: eagle_pm_suspend_noirq
Line 136821: kern.info: [87489.462053] Disabling non-boot CPUs ...
Line 136822: kern.info: [87489.503407] Enabling non-boot CPUs ...
Line 136823: kern.info: [87489.503988] CPU1 is up
Line 136824: kern.warn: [87489.505968] gpmi-nand 112000.gpmi-nand: use legacy bch geometry
Line 136825: kern.info: [87489.506319] gpmi-nand 112000.gpmi-nand: enable the asynchronous EDO mode 5
Line 136826: kern.err: [87489.541321] eagle-pm eagle-pm: eagle_pm_complete
Line 136827: kern.info: [87489.546383] PM: resume devices took 0.040 seconds
Line 136828: kern.info: [87489.546390] OOM killer enabled.
Line 136829: kern.info: [87489.546395] Restarting tasks ... done.
Line 136830: kern.info: [87489.547587] PM: suspend exit
Line 136831: kern.info: [87491.486258] sdhci-esdhc-imx 2190000.usdhc: Got CD GPIO
Line 136832: kern.info: [87491.486354] sdhci-esdhc-imx 2190000.usdhc: assigned as sdcard host
Line 136833: kern.info: [87491.552458] mmc0: SDHCI controller on 2190000.usdhc [2190000.usdhc] using ADMA
Line 136834: kern.info: [87491.560221] sdhci-esdhc-imx 2194000.usdhc: assigned as wifi host
Line 136835: kern.warn: [87491.599849] mmc0: host does not support reading read-only switch, assuming write-enable
Line 136836: kern.info: [87491.606095] mmc0: new high speed SDHC card at address 59b4
Line 136837: kern.info: [87491.618631] mmcblk0: mmc0:59b4 USD 7.51 GiB
Line 136838: kern.info: [87491.622092] mmc1: SDHCI controller on 2194000.usdhc [2194000.usdhc] using ADMA
Line 136839: kern.info: [87491.622463] mmcblk0: p1
Line 136840: kern.warn: [87491.639838] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
Line 136841: kern.warn: [87491.641587] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
Line 136842: kern.warn: [87491.643362] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
Line 136843: kern.warn: [87491.646626] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
Line 136844: kern.warn: [87491.647698] mmc1: queuing unknown CIS tuple 0x81 (1 bytes)
Line 136845: kern.info: [87491.724493] mmc1: new high speed SDIO card at address 0001
Line 136846: kern.info: [87492.824549] brcmfmac: brcmf_fw_map_chip_to_name: using brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
Line 136847: kern.info: [87493.161527] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: May 2 2019 02:46:17 version 7.45.189 (r714228 CY) FWID 01-e1db26e2
Line 136848: kern.err: [87493.169404] brcmfmac: brcmf_sdio_ulp_preinit: ulp_sdioctrl iovar returned err = -23
Line 136849: kern.info: [87493.232404] brcmfmac mmc1:0001:1 eth1: renamed from wlan0
Line 136850: kern.warn: [87494.415460] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Line 136851: kern.info: [87495.083898] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
Line 137062: kern.info: [87500.640625] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Line 137805: kern.err: [87576.691461] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing to do.
Line 137806: kern.err: [87576.698637] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-5)
Line 137807: kern.info: [87576.856125] mmc1: card 0001 removed
Line 137808: kern.info: [87578.186180] suspend.sh (21344): drop_caches: 3
Line 137809: kern.info: [87579.247814] mmc0: card 59b4 removed
Line 137810: kern.info: [87580.366529] PM: suspend entry (deep)
Line 137811: kern.info: [87580.366540] PM: Syncing filesystems ... done.
Line 137812: kern.info: [87580.371726] Freezing user space processes ... (elapsed 0.002 seconds) done.
Line 137813: kern.info: [87580.373828] OOM killer disabled.
Line 137814: kern.info: [87580.373837] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
Line 137815: kern.info: [87580.448269] PM: suspend devices took 0.070 seconds
Line 137816: kern.err: [87580.450492] eagle-pm eagle-pm: eagle_pm_suspend_noirq
Line 137817: kern.info: [87580.456024] Disabling non-boot CPUs ...
Line 137818: kern.warn: [87580.489543] IRQ17 no longer affine to CPU1
Line 137819: kern.info: [87580.490670] Enabling non-boot CPUs ...
Line 137820: kern.info: [87580.491254] CPU1 is up
Line 137821: kern.warn: [87580.493164] gpmi-nand 112000.gpmi-nand: use legacy bch geometry
Line 137822: kern.info: [87580.493515] gpmi-nand 112000.gpmi-nand: enable the asynchronous EDO mode 5
Line 137823: kern.err: [87580.528195] eagle-pm eagle-pm: eagle_pm_complete
Line 137824: kern.info: [87580.533188] PM: resume devices took 0.040 seconds
Line 137825: kern.info: [87580.533195] OOM killer enabled.
Line 137826: kern.info: [87580.533199] Restarting tasks ... done.
Line 137827: kern.info: [87580.534341] PM: suspend exit
Line 137828: kern.info: [87582.115095] sdhci-esdhc-imx 2190000.usdhc: Got CD GPIO
Line 137829: kern.info: [87582.115185] sdhci-esdhc-imx 2190000.usdhc: assigned as sdcard host
Line 137830: kern.info: [87582.189689] mmc0: SDHCI controller on 2190000.usdhc [2190000.usdhc] using ADMA
Line 137831: kern.info: [87582.195933] sdhci-esdhc-imx 2194000.usdhc: assigned as wifi host
Line 137832: kern.warn: [87582.237440] mmc0: host does not support reading read-only switch, assuming write-enable
Line 137833: kern.info: [87582.243696] mmc0: new high speed SDHC card at address 59b4
Line 137834: kern.info: [87582.256055] mmcblk0: mmc0:59b4 USD 7.51 GiB
Line 137835: kern.info: [87582.258579] mmcblk0: p1
Line 137836: kern.info: [87582.259956] mmc1: SDHCI controller on 2194000.usdhc [2194000.usdhc] using ADMA
Line 137837: kern.warn: [87582.278383] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
Line 137838: kern.warn: [87582.280136] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
Line 137839: kern.warn: [87582.281938] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
Line 137840: kern.warn: [87582.285067] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
Line 137841: kern.warn: [87582.286120] mmc1: queuing unknown CIS tuple 0x81 (1 bytes)
Line 137842: kern.info: [87582.366738] mmc1: new high speed SDIO card at address 0001
Line 137843: kern.info: [87583.500737] brcmfmac: brcmf_fw_map_chip_to_name: using brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
Line 137844: kern.info: [87583.836160] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: May 2 2019 02:46:17 version 7.45.189 (r714228 CY) FWID 01-e1db26e2
Line 137845: kern.err: [87583.845273] brcmfmac: brcmf_sdio_ulp_preinit: ulp_sdioctrl iovar returned err = -23
Line 137846: kern.info: [87583.907988] brcmfmac mmc1:0001:1 eth1: renamed from wlan0
Line 137847: kern.warn: [87585.092312] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Line 138011: kern.info: [87586.965827] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
Line 138097: kern.info: [87592.485339] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Line 138581: kern.alert: [87658.479069] Unhandled fault: imprecise external abort (0x1406) at 0x7669d004
Line 138582: kern.alert: [87658.486144] pgd = a73d4000
Line 138583: kern.alert: [87658.488856] [7669d004] *pgd=00000000

>> system rebooted here:

Line 138947: kern.info: [ 0.000000] Booting Linux on physical CPU 0x0
Line 138948: kern.notice: [ 0.000000] Linux version 4.14.78-1.0.0+g94da7bd (oe-user@oe-host) (gcc version 8.3.0 (GCC)) #1 SMP PREEMPT Sat Jun 10 01:32:44 UTC 2023
Line 138949: kern.info: [ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c53c7d
Line 138950: kern.info: [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Line 138951: kern.info: [ 0.000000] OF: fdt: Machine model: Masimo i.MX6Dual Hawk
Line 138952: kern.info: [ 0.000000] Memory policy: Data cache writealloc
Line 138953: kern.info: [ 0.000000] Reserved memory: created CMA memory pool at 0x3a000000, size 320 MiB
Line 138954: kern.info: [ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
Line 138955: kern.debug: [ 0.000000] On node 0 totalpages: 253952
Line 138956: kern.debug: [ 0.000000] free_area_init_node: node 0, pgdat 80f7f800, node_mem_map a9797000
Line 138957: kern.debug: [ 0.000000] Normal zone: 1984 pages used for memmap
Line 138958: kern.debug: [ 0.000000] Normal zone: 0 pages reserved
Line 138959: kern.debug: [ 0.000000] Normal zone: 253952 pages, LIFO batch:31
Line 138960: kern.info: [ 0.000000] percpu: Embedded 16 pages/cpu @a974e000 s35532 r8192 d21812 u65536
Line 138961: kern.debug: [ 0.000000] pcpu-alloc: s35532 r8192 d21812 u65536 alloc=16*4096
Line 138962: kern.debug: [ 0.000000] pcpu-alloc: [0] 0 [0] 1
Line 138963: kern.info: [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 251968
Line 138964: kern.notice: [ 0.000000] Kernel command line: quiet console=ttymxc0,115200 mtdparts=gpmi-nand:-(ubi) ubi.mtd=ubi root=ubi0:rootfs0 rootfstype=ubifs raptor.serialno=1000301285 raptor.hw=temp,nibp,NorV1 raptor.ethaddr= raptor.radio_ethaddr=b6:10:d4:04:67:d5 raptor.bluetooth_address=b6:10:d4:04:67:d4 no_console_suspend fbmem=0x4e000000,0x02000000
Line 138965: kern.err: [ 0.000000] Framebuffer was pre-defined at 0x4e000000 size:0x2000000
Line 138966: kern.info: [ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
Line 138967: kern.info: [ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Line 138968: kern.info: [ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Line 138969: kern.info: [ 0.000000] Memory: 663072K/1015808K available (9216K kernel code, 514K rwdata, 3292K rodata, 1024K init, 435K bss, 25056K reserved, 327680K cma-reserved, 0K highmem)
Line 138970: kern.notice: [ 0.000000] Virtual kernel memory layout:
Line 138971: kern.notice: [ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
Line 138972: kern.notice: [ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
Line 138973: kern.notice: [ 0.000000] vmalloc : 0xbe800000 - 0xff800000 (1040 MB)
Line 138974: kern.notice: [ 0.000000] lowmem : 0x80000000 - 0xbe000000 ( 992 MB)
Line 138975: kern.notice: [ 0.000000] pkmap : 0x7fe00000 - 0x80000000 ( 2 MB)
Line 138976: kern.notice: [ 0.000000] modules : 0x7f000000 - 0x7fe00000 ( 14 MB)
Line 138977: kern.notice: [ 0.000000] .text : 0x80008000 - 0x80a00000 (10208 kB)
Line 138978: kern.notice: [ 0.000000] .init : 0x80e00000 - 0x80f00000 (1024 kB)
Line 138979: kern.notice: [ 0.000000] .data : 0x80f00000 - 0x80f80980 ( 515 kB)
Line 138980: kern.notice: [ 0.000000] .bss : 0x80f82000 - 0x80feee18 ( 436 kB)
Line 138981: kern.info: [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Line 138982: kern.info: [ 0.000000] Preemptible hierarchical RCU implementation.
Line 138983: kern.info: [ 0.000000] RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
Line 138984: kern.info: [ 0.000000] Tasks RCU enabled.
Line 138985: kern.info: [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
Line 138986: kern.info: [ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
Line 138987: kern.info: [ 0.000000] L2C-310 errata 752271 769419 enabled
Line 138988: kern.info: [ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
Line 138989: kern.info: [ 0.000000] L2C-310 full line of zeros enabled for Cortex-A9
Line 138990: kern.info: [ 0.000000] L2C-310 ID prefetch enabled, offset 16 lines
Line 138991: kern.info: [ 0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
Line 138992: kern.info: [ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
Line 138993: kern.info: [ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x76470001
Line 138994: kern.info: [ 0.000000] Switching to timer-based delay loop, resolution 333ns
Line 138995: kern.info: [ 0.000008] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps every 715827882841ns
Line 138996: kern.info: [ 0.000020] clocksource: mxc_timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 637086815595 ns
Line 138997: kern.info: [ 0.001706] Console: colour dummy device 80x30
Line 138998: kern.info: [ 0.001725] Calibrating delay loop (skipped), value calculated using timer frequency.. 6.00 BogoMIPS (lpj=30000)
Line 138999: kern.info: [ 0.001737] pid_max: default: 32768 minimum: 301
Line 139000: kern.info: [ 0.001865] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
Line 139001: kern.info: [ 0.001883] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
Line 139002: kern.info: [ 0.002384] CPU: Testing write buffer coherency: ok
Line 139003: kern.info: [ 0.002415] CPU0: Spectre v2: using BPIALL workaround
Line 139004: kern.info: [ 0.002691] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Line 139005: kern.info: [ 0.039608] Setting up static identity map for 0x10100000 - 0x10100060
Line 139006: kern.info: [ 0.059555] Hierarchical SRCU implementation.
Line 139007: kern.info: [ 0.099553] smp: Bringing up secondary CPUs ...
Line 139008: kern.info: [ 0.169816] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
Line 139009: kern.info: [ 0.169822] CPU1: Spectre v2: using BPIALL workaround
Line 139010: kern.info: [ 0.169934] smp: Brought up 1 node, 2 CPUs
Line 139011: kern.info: [ 0.169943] SMP: Total of 2 processors activated (12.00 BogoMIPS).
Line 139012: kern.info: [ 0.169948] CPU: All CPU(s) started in SVC mode.

Regards,
Raj

alaganraj · ‎06-26-2023

Hi Qmiller,

Thank you for your response.

Yes, this log appears in normal boot log and system works fine.

problem is we are not seeing complete backtrace when system crash, how to find which module causing this crash?

Line 138581: kern.alert: [87658.479069] Unhandled fault: imprecise external abort (0x1406) at 0x7669d004
Line 138582: kern.alert: [87658.486144] pgd = a73d4000
Line 138583: kern.alert: [87658.488856] [7669d004] *pgd=00000000

Zhiming_Liu · ‎06-26-2023

It's caused by hardware access memory, you can try to remove brcm driver and then test again.

Zhiming_Liu · ‎06-26-2023

Hi @alaganraj

You can try to disable eth and then test again.

Does this log exist in normal boot log?

FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.