i.MX 6 crash after wake from a sleep

[andrzejdziedzic]

Hi,

I have custom devices with i.MX6 and Android 6.

In one of devices I observe a crash every time this device wakes up from deep sleep (a log below).

I guess this may be a hardware problem on this PCB, because other devices with the same SW works correctly.

Do you have any idea what could be a reson of such crash:

# echo mem > /sys/power/state
[   64.964614] PM: suspend entry 1970-01-01 00:01:01.450056673 UTC
[   64.965403] PM: Syncing filesystems ... done.
[   65.020704] Freezing user space processes ... (elapsed 0.001 seconds) done.
[   65.023554] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[   65.044617] Wake-up in: 00h 09m 08s
[   65.052369] lm75 2-0049: suspend [CHRG 275*C/10]
[   65.057322] lm75 2-0048: watching [iMX6 305*C/10]
[   65.059966] bq2589x-1 1-006a: suspend #1 [3664mV 41%]
[   65.207747] PM: suspend of devices complete after 181.285 msecs
[   65.209415] PM: late suspend of devices complete after 0.902 msecs
[   65.211081] PM: noirq suspend of devices complete after 0.873 msecs
[   65.211882] Disabling non-boot CPUs ...
[   65.213100] CPU1: shutdown
[   65.235271] Enabling non-boot CPUs ...
[   65.274987] CPU1 is up
[   65.275799] PM: noirq resume of devices complete after 0.489 msecs
[   65.276791] [galcore]: _AllowAccess(576) External clock off
[   65.277268] PM: early resume of devices complete after 0.482 msecs
[   65.278346] ------------[ cut here ]------------
[   65.278942] kernel BUG at drivers/mxc/gpu-viv/hal/os/linux/kernel/gc_hal_kernel_os.c:2016!
[   65.279990] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[   65.280734] Modules linked in:
[   65.281146] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.1.15-00005-g04637de-dirty #264
[   65.282150] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[   65.282982] task: c0e0ca30 ti: c0e00000 task.ti: c0e00000
[   65.283684] PC is at gckOS_ReadRegisterEx+0xdc/0xe8
[   65.284310] LR is at _AllowAccess+0x60/0x88
[   65.284849] pc : [<c057d97c>]    lr : [<c057d6e8>]    psr: 600001d3
[   65.284849] sp : c0e01e78  ip : 0000002b  fp : c0e7193c
[   65.286298] r10: 00000000  r9 : 00000000  r8 : c0e00000
[   65.286965] r7 : c0e01e94  r6 : d05c4700  r5 : 00000010  r4 : 00000002
[   65.287795] r3 : 0000005b  r2 : 00010001  r1 : 600001d3  r0 : 00000000
[   65.288626] Flags: nZCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment kernel
[   65.289575] Control: 10c53c7d  Table: 20b5804a  DAC: 00000015
[   65.290306]
[   65.290306] PC: 0xc057d8fc:
[   65.290853] d8fc  ebffff61 e3500000 0a00001c e5963004 e0833104 e5932064 e0825005 e5952000
[   65.291981] d91c  f57ff04f e1a0300d e3c33d7f e3c3303f e5872000 e3a05cff e5933004 e340501f
[   65.293106] d93c  e0055003 e3550000 0a000006 e3a00000 e8bd80f8 e0840084 e0860180 e2800094
[   65.294233] d95c  eb0a1311 eaffffe2 e0844084 e0860184 e2800094 eb0a128e e1a00005 e8bd80f8
[   65.295359] d97c  e7f001f2 e3e00000 e8bd80f8 e1a03002 e1a02001 e3a01000 eaffffc1 e92d40f8
[   65.296485] d99c  e281e026 e590c004 e79cc10e e15c0002 9a000035 e1a07003 e1a04001 e1a0300d
[   65.297611] d9bc  e3c31d7f e3c1103f e1a05002 e3a02cff e1a06000 e5913004 e340201f e0022003
[   65.298737] d9dc  e3520000 0a00001c e1a00006 e1a01004 e1a02005 ebffff24 e3500000 0a000021
[   65.299864]
[   65.299864] LR: 0xc057d668:
[   65.300410] d668  1affffef eafffff1 e3049984 e34c90e0 e5993000 e3530000 1affffe9 eaffffeb
[   65.301538] d688  e281303a e92d4010 e7904103 e3540000 0a00000c e3520000 0a000007 e5903004
[   65.302664] d6a8  e0831101 e5912064 e5923000 f57ff04f e2033003 e3530003 0a00000a e3a04001
[   65.303790] d6c8  e1a00004 e8bd8010 e30b0994 e3031f04 e34c00b8 e34c1095 e3a02d09 eb003d75
[   65.304916] d6e8  e1a00004 e8bd8010 e30b09bc e3031f04 e34c00b8 e34c1095 e3002257 e3a04000
[   65.306042] d708  eb003d6c eaffffed e92d4008 e1a02003 e591c028 e5913008 e59de008 e1a0000c
[   65.307169] d728  e58e3000 e59c3008 e5933010 e12fff33 e0000fc0 e8bd8008 e92d4008 e1a02003
[   65.308295] d748  e5910028 e5903008 e5933014 e12fff33 e3a00000 e8bd8008 e3520000 13510000
[   65.309422]
[   65.309422] SP: 0xc0e01df8:
[   65.309969] 1df8  d052cd20 c0162d9c 00000000 d052ec70 c0eec294 600001d3 c0e7193c c057d980
[   65.311095] 1e18  00000000 c010c4d8 00000000 e7100000 c010c12c c010c12c 00000000 600001d3
[   65.312221] 1e38  00010001 0000005b 00000002 00000010 d05c4700 c0e01e94 c0e00000 00000000
[   65.313347] 1e58  00000000 c0e7193c 0000002b c0e01e78 c057d6e8 c057d97c 600001d3 ffffffff
[   65.314473] 1e78  c0e01e94 d0671800 d079a440 00000126 00000000 c05a17b0 ea6ffbc0 c0158fbc
[   65.315599] 1e98  d01c0540 c057b7a0 c057b78c c017a8c0 8e4204f8 00000000 c0e71928 d01c0540
[   65.316724] 1eb8  00000000 d01c0540 d01c05a0 d079a440 00000000 00000001 c0e01f40 d0020000
[   65.317850] 1ed8  c0e01f90 c017ab00 d01c0540 d01c05a0 c0e0f510 c017d65c 00000126 00000000
[   65.318976]
[   65.318976] FP: 0xc0e718bc:
[   65.319522] 18bc  00000000 00000000 c0bb2368 00000000 00000000 00000000 00000000 c0bb235c
[   65.320647] 18dc  00000000 00000000 00000000 00000000 c0bb2398 00000000 00000000 00000000
[   65.321772] 18fc  00000000 c0bb2388 00000000 00000000 00000000 00000000 c0bb2378 00000000
[   65.322897] 191c  00000000 00000000 00000000 c0bb23a8 00000000 00000000 00000000 00000000
[   65.324022] 193c  c0bb23bc 00000000 00000000 00000000 00000000 c0bb23e0 00000000 00000000
[   65.325147] 195c  00000000 00000000 c0bb23d0 00000000 00000000 00000000 00000000 c0bb2420
[   65.326273] 197c  00000000 00000000 00000000 00000000 c0bb2438 00000000 00000000 00000000
[   65.327398] 199c  00000000 c0bb2408 00000000 00000000 00000000 00000000 c0bb23f0 00000000
[   65.328524]
[   65.328524] R6: 0xd05c4680:
[   65.329070] 4680  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.330195] 46a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.331320] 46c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.332445] 46e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.333570] 4700  2020534f d0637000 d05e37c0 d05e3800 d05e3fc0 d0c19b80 00000001 d05e3880
[   65.334696] 4720  d063f670 d063f670 00000001 00000000 00000000 00000000 00000000 03400340
[   65.335821] 4740  00000099 d05e38c0 d0a2ee10 d0a2ee10 00000001 00000000 00000000 00000000
[   65.336946] 4760  00000000 020c020c 0000003b 00000000 d05e3840 d0637200 eac97260 0000076e
[   65.338073]
[   65.338073] R7: 0xc0e01e14:
[   65.338620] 1e14  c057d980 00000000 c010c4d8 00000000 e7100000 c010c12c c010c12c 00000000
[   65.339746] 1e34  600001d3 00010001 0000005b 00000002 00000010 d05c4700 c0e01e94 c0e00000
[   65.340872] 1e54  00000000 00000000 c0e7193c 0000002b c0e01e78 c057d6e8 c057d97c 600001d3
[   65.341997] 1e74  ffffffff c0e01e94 d0671800 d079a440 00000126 00000000 c05a17b0 ea6ffbc0
[   65.343123] 1e94  c0158fbc d01c0540 c057b7a0 c057b78c c017a8c0 8e4204f8 00000000 c0e71928
[   65.344249] 1eb4  d01c0540 00000000 d01c0540 d01c05a0 d079a440 00000000 00000001 c0e01f40
[   65.345375] 1ed4  d0020000 c0e01f90 c017ab00 d01c0540 d01c05a0 c0e0f510 c017d65c 00000126
[   65.346501] 1ef4  00000000 00000126 c0179f6c c0d6bd10 c017a228 f4a0010c 0000001b c0e05760
[   65.347628]
[   65.347628] R8: 0xc0dfff80:
[   65.348174] ff80  2064616f 706f7270 69747265 66207365 206d6f72 7379732f 2f6d6574 2f202b20
[   65.349302] ffa0  74636166 2079726f 65746661 73662072 756f6d20 0a2e746e 6c006e6f 5f64616f
[   65.350428] ffc0  74737973 705f6d65 73706f72 7463615f 006e6f69 20202020 64616f6c 7379735f
[   65.351554] ffe0  5f6d6574 706f7270 6f0a0073 6f6c006e 705f6461 69737265 705f7473 73706f72
[   65.352680] 0000  00000000 00010002 00000000 c0e0ca30 00000000 00000015 ea6ffb80 c0e0ca30
[   65.353805] 0020  00000000 d092b500 d00b2a00 c0e055e4 c0e0ce0c c0e01f74 c0e01f30 c08007dc
[   65.354931] 0040  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.356056] 0060  010011e8 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.357183] Process swapper/0 (pid: 0, stack limit = 0xc0e00210)
[   65.357948] Stack: (0xc0e01e78 to 0xc0e02000)
[   65.358509] 1e60:                                                       c0e01e94 d0671800
[   65.359549] 1e80: d079a440 00000126 00000000 c05a17b0 ea6ffbc0 c0158fbc d01c0540 c057b7a0
[   65.360588] 1ea0: c057b78c c017a8c0 8e4204f8 00000000 c0e71928 d01c0540 00000000 d01c0540
[   65.361626] 1ec0: d01c05a0 d079a440 00000000 00000001 c0e01f40 d0020000 c0e01f90 c017ab00
[   65.362664] 1ee0: d01c0540 d01c05a0 c0e0f510 c017d65c 00000126 00000000 00000126 c0179f6c
[   65.363702] 1f00: c0d6bd10 c017a228 f4a0010c 0000001b c0e05760 c0e01f40 f4a00100 c0e0f064
[   65.364740] 1f20: 00000001 c0101424 c010822c 60000153 ffffffff c0e01f74 ea6fef80 c010c080
[   65.365778] 1f40: 00000001 ea6fef80 00000000 c0119040 c0e00000 c0e04a3c c0e71ef0 c0902580
[   65.366815] 1f60: ea6fef80 c0e0f064 00000001 c0e01f90 01000000 c0e01f88 c0108228 c010822c
[   65.367854] 1f80: 60000153 ffffffff 00000000 c0163a50 c0d69344 c0d6bf78 c0e055e4 c0e04a44
[   65.368893] 1fa0: 00000000 c0e7164a c0d6cb80 c0d6cb80 00000000 ffffffff 00000000 c0d00c70
[   65.369930] 1fc0: ffffffff ffffffff c0d00684 00000000 00000000 c0d61010 c0e78214 c0e049c0
[   65.370969] 1fe0: c0d6100c c0e0de08 1000406a 412fc09a 00000000 1000807c 00000000 00000000
[   65.372028] [<c057d97c>] (gckOS_ReadRegisterEx) from [<c05a17b0>] (gckVGINTERRUPT_Enque+0x1c/0xe8)
[   65.373172] [<c05a17b0>] (gckVGINTERRUPT_Enque) from [<c057b7a0>] (isrRoutineVG+0x14/0x20)
[   65.371e38>] (alloc_contig_range) from [<c022f4e8>] (cma_alloc+0xe4/0x254)
[   65.416869] [<c022f4e8>] (cma_alloc) from [<c0114b30>] (__alloc_from_contiguous+0x38/0x128)
[   65.417932] [<c0114b30>] (__alloc_from_contiguous) from [<c0114d00>] (__dma_alloc+0xe0/0x278)
[   65.419017] [<c0114d00>] (__dma_alloc) from [<c0114fe4>] (arm_dma_alloc+0xa0/0xa8)
[   65.419987] [<c0114fe4>] (arm_dma_alloc) from [<c0425284>] (imx_startup+0x3e8/0x4e4)
[   65.420977] [<c0425284>] (imx_startup) from [<c041fe2c>] (uart_resume_port+0x19c/0x2d0)
[   65.421999] [<c041fe2c>] (uart_resume_port) from [<c0423064>] (serial_imx_resume+0x200/0x230)
[   65.423090] [<c0423064>] (serial_imx_resume) from [<c045bd3c>] (dpm_run_callback+0x3c/0x18c)
[   65.424166] [<c045bd3c>] (dpm_run_callback) from [<c045c334>] (device_resume+0x8c/0x19c)
[   65.425197] [<c045c334>] (device_resume) from [<c045d6f8>] (dpm_resume+0x124/0x368)
[   65.426174] [<c045d6f8>] (dpm_resume) from [<c045dd98>] (dpm_resume_end+0xc/0x18)
[   65.427135] [<c045dd98>] (dpm_resume_end) from [<c016e6b0>] (suspend_devices_and_enter+0xec/0x2dc)
[   65.428276] [<c016e6b0>] (suspend_devices_and_enter) from [<c016ec28>] (pm_suspend+0x388/0x5fc)
[   65.429383] [<c016ec28>] (pm_suspend) from [<c016d274>] (state_store+0x74/0x7c)
[   65.430323] [<c016d274>] (state_store) from [<c0295180>] (kernfs_fop_write+0xb8/0x19c)
[   65.431337] [<c0295180>] (kernfs_fop_write) from [<c02316ec>] (__vfs_write+0x20/0xd8)
[   65.432337] [<c02316ec>] (__vfs_write) from [<c0231f0c>] (vfs_write+0x90/0x164)
[   65.433270] [<c0231f0c>] (vfs_write) from [<c0232734>] (SyS_write+0x44/0x9c)
[   65.434176] [<c0232734>] (SyS_write) from [<c0107640>] (ret_fast_syscall+0x0/0x3c)
[   65.444729] Rebooting in 5 seconds..

[weidong_sun]

Dear Andrzej,

     Yes, I also feel it seems there are some problems on hardware, BUT it is very hard to judge which parts the issue comes from.

     As we know, when device enters into sleep mode, software running before will  be reserved in DDR memory, when device is waken up, software will begin to run again.

     So I think you can check if the issue comes from DDR memory. For example, re-heated it by BGA workbench, or change DDR chip to be a new one.

Hope above advice is helpful for you!

Have a nice day!

BR,

NXP TIC Weidong Sun

[andrzejdziedzic]

Dear weidong.sun,

Thank you for advices. We already trying to re-heat it (no difference), so now will try to change the chip.

Best regards,

Andrzej