i.mx6q rcu_bh_state detected stall on CPU 1 (t=0 jiffies)

[seekwok]

Hi All,

Our board is i.mx6q + 2G DDR3L, we meet the problem reboot randomly. 

Any help is greatly appreciated.

Our boards are supplied to the largest express cabinet operator in China. At present, some boards are randomly restarted on the customer side.We don't have any ideas about the problem. Can you show us the way to solve the problem?or any other advice？

thx；

the log as below：

[37452.456531] usb0: no IPv6 routers present
[37457.006539] eth0: no IPv6 routers present
[63215.935829] usb 2-1.5: USB disconnect, device number 3
[63216.716747] usb 2-1.5: new full speed USB device number 8 using fsl-ehci
[63216.880018] input: ATMEL ASF HID MULTI TOUCH as /devices/platform/fsl-ehci.1/usb2/2-1/2-1.5/2-1.5:1.0/input/input5
[63216.884583] hid-multitouch 0003:1FF7:0013.0003: input,hidraw0: USB HID v1.11 Mouse [ATMEL ASF HID MULTI TOUCH] on usb-fsl-ehci.1-1.5/input0
[63216.936852] input: ATMEL ASF HID MULTI TOUCH as /devices/platform/fsl-ehci.1/usb2/2-1/2-1.5/2-1.5:1.1/input/input6
[63216.941457] cpufreq_interactive_input_connect: connect to ATMEL ASF HID MULTI TOUCH
[63216.943689] hid-multitouch 0003:1FF7:0013.0004: input,hidraw1: USB HID v1.11 Device [ATMEL ASF HID MULTI TOUCH] on usb-fsl-ehci.1-1.5/input1
[84516.415827] usb 2-1.5: USB disconnect, device number 8
[84517.016810] usb 2-1.5: new full speed USB device number 9 using fsl-ehci
[84517.182375] input: ATMEL ASF HID MULTI TOUCH as /devices/platform/fsl-ehci.1/usb2/2-1/2-1.5/2-1.5:1.0/input/input7
[84517.185423] hid-multitouch 0003:1FF7:0013.0005: input,hidraw0: USB HID v1.11 Mouse [ATMEL ASF HID MULTI TOUCH] on usb-fsl-ehci.1-1.5/input0
[84517.233836] input: ATMEL ASF HID MULTI TOUCH as /devices/platform/fsl-ehci.1/usb2/2-1/2-1.5/2-1.5:1.1/input/input8
[84517.234970] cpufreq_interactive_input_connect: connect to ATMEL ASF HID MULTI TOUCH
[84517.236023] hid-multitouch 0003:1FF7:0013.0006: input,hidraw1: USB HID v1.11 Device [ATMEL ASF HID MULTI TOUCH] on usb-fsl-ehci.1-1.5/input1
[120256.132973] binder: release proc 2923, transaction 12663229, not freed
[120260.029902] init: untracked pid 3394 exited
[120265.580827] type=1400 audit(1606412339.004:6): avc: denied { write } for pid=15870 comm=".sf.module.edms" scontext=u:r:untrusted_app:s0 tcontext=u:r:untrusted_app:s0 tclass=netlink_route_socket
[173606.576559] INFO: rcu_bh_state detected stall on CPU 1 (t=0 jiffies)
[173606.576720] Backtrace for cpu 1 (current):
[173606.576836] [<c0049db0>] (unwind_backtrace+0x0/0xf8) from [<c0048df4>] (smp_send_all_cpu_backtrace+0x58/0xd4)
[173606.577005] [<c0048df4>] (smp_send_all_cpu_backtrace+0x58/0xd4) from [<c00c30e8>] (__rcu_pending+0x140/0x3bc)
[173606.577164] [<c00c30e8>] (__rcu_pending+0x140/0x3bc) from [<c00c3c64>] (rcu_check_callbacks+0x11c/0x204)
[173606.577325] [<c00c3c64>] (rcu_check_callbacks+0x11c/0x204) from [<c0088120>] (update_process_times+0x3c/0x6c)
[173606.577487] [<c0088120>] (update_process_times+0x3c/0x6c) from [<c00a5b28>] (tick_sched_timer+0x88/0xd8)
[173606.577583] [<c00a5b28>] (tick_sched_timer+0x88/0xd8) from [<c009986c>] (__run_hrtimer.isra.18+0x44/0xd8)
[173606.577742] [<c009986c>] (__run_hrtimer.isra.18+0x44/0xd8) from [<c009a13c>] (hrtimer_interrupt+0xf4/0x270)
[173606.577905] [<c009a13c>] (hrtimer_interrupt+0xf4/0x270) from [<c003d340>] (do_IPI+0x154/0x214)
[173606.578066] [<c003d340>] (do_IPI+0x154/0x214) from [<c0042a4c>] (__irq_svc+0x4c/0xe8)
[173606.578214] Exception stack(0xce051f90 to 0xce051fd8)
[173606.578299] 1f80: 00000001 800e0093 ce051fc0 f40c4000
[173606.578446] 1fa0: ce050000 c0854944 c080da6c c060c93c c0815a1c 412fc09a 00000000 00000000
[173606.578594] 1fc0: 00000000 ce051fd8 c060a7a0 c0043b3c 400e0013 ffffffff
[173606.578682] [<c0042a4c>] (__irq_svc+0x4c/0xe8) from [<c0043b3c>] (default_idle+0x24/0x28)
[173606.578837] [<c0043b3c>] (default_idle+0x24/0x28) from [<c0043e5c>] (cpu_idle+0xd8/0x11c)
[173606.578994] [<c0043e5c>] (cpu_idle+0xd8/0x11c) from [<10600cd4>] (0x10600cd4)
[173606.579076]
[173606.579078] sending IPI to all other CPUs:
[173606.579297] IPI backtrace for cpu 0
[173606.579378]
[173606.579457] Pid: 0, comm: swapper
[173606.579608] CPU: 0 Not tainted (3.0.35 #9)
[173606.579694] PC is at default_idle+0x24/0x28
[173606.579780] LR is at _raw_spin_unlock_irqrestore+0x24/0x44
[173606.579929] pc : [<c0043b3c>] lr : [<c060a7a0>] psr: 400f0013
[173606.579934] sp : c07f9fb0 ip : 00000000 fp : 00000000
[173606.580153] r10: 00000000 r9 : 412fc09a r8 : c0815a1c
[173606.580233] r7 : c060c93c r6 : c080da6c r5 : c0854944 r4 : c07f8000
[173606.580382] r3 : f40c4000 r2 : c07f9f98 r1 : 800f0093 r0 : 00000001
[173606.580548] Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
[173606.580641] Control: 10c53c7d Table: 1a63c04a DAC: 00000015
[173606.580807]
[173606.580809] PC: 0xc0043abc:
[173606.580980] 3abc c0824c30 c080db60 c060ba78 c0717514 e121f007 e1a00004 e1a0e006 e1a0f005
[173606.581867] 3adc eb00edc1 e320f000 e3043944 e34c3085 e5932000 e2822001 e5832000 e12fff1e
[173606.582725] 3afc e3043944 e34c3085 e5932000 e2422001 e5832000 e12fff1e e12fff1e e92d4008
[173606.583517] 3b1c e1a0200d e3c23d7f e3c3303f e5933000 e3130002 1a000000 eb003827 f1080080
[173606.584313] 3b3c e8bd8008 e92d4070 e24dd050 e1a04000 e1a05001 e28d0008 e3a01048 e1a06002
[173606.585069] 3b5c eb089d9f e3a0c000 e3860502 e1a0100c e28d2008 e1a0300c e3800c01 e58d5018
[173606.585681] 3b7c e303eadc e58d401c e34ce004 e58dc000 e58de020 e3a0e013 e58dc004 e58de024
[173606.586345] 3b9c e303eacc e34ce004 e58de044 e3a0e093 e58de048 eb00dd14 e28dd050 e8bd8070
[173606.587012]
[173606.587015] LR: 0xc060a720:
[173606.587138] a720 f57ff04f e320f004 e3a00001 ebe9a831 e1a00004 e8bd4010 eae9dcc9 e92d4008
[173606.587804] a740 f57ff05f e3a03000 e5803000 f57ff04f e320f004 e3a00001 ebe9a826 e1a0200d
[173606.588414] a760 e3c23d7f e3c3303f e5933000 e3130002 08bd8008 e8bd4008 eafff8fe e92d4008
[173606.589068] a780 f57ff05f e3a03000 e5803000 f57ff04f e320f004 e121f001 e3a00001 ebe9a815
[173606.589679] a7a0 e1a0200d e3c23d7f e3c3303f e5933000 e3130002 08bd8008 e8bd4008 eafff8ed
[173606.590344] a7c0 e92d4008 f57ff05f e3a03000 e5803000 f57ff04f e320f004 f1080080 e3a00001
[173606.591008] a7e0 ebe9a804 e1a0200d e3c23d7f e3c3303f e5933000 e3130002 08bd8008 e8bd4008
[173606.591620] a800 eafff8dc e92d4008 f57ff05f e1903f9f e2433001 e1802f93 e3320000 1afffffa
[173606.592286]
[173606.592289] SP: 0xc07f9f30:
[173606.592464] 9f30 c07f9f68 c0854ad8 00000000 c00442b4 c080dd30 c003d390 ffffffff f2a00100
[173606.593068] 9f50 00000407 00000002 00000001 c07f8000 00000000 c0042a4c 00000001 800f0093
[173606.593731] 9f70 c07f9f98 f40c4000 c07f8000 c0854944 c080da6c c060c93c c0815a1c 412fc09a
[173606.594399] 9f90 00000000 00000000 00000000 c07f9fb0 c060a7a0 c0043b3c 400f0013 ffffffff
[173606.595063] 9fb0 c0043b18 c0043e5c c080dbac 00000000 c0036894 c28851c0 1000406a c00088b0
[173606.595672] 9fd0 c00082f8 00000000 00000000 c0036894 00000000 10c53c7d c080da8c c0036890
[173606.596334] 9ff0 c0815a14 10008040 00000000 00000000 00000000 00000000 00000000 00000000
[173606.596946] a010 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.597548]
[173606.597550] R2: 0xc07f9f18:
[173606.597724] 9f18 00000000 c07f9f68 c003bc3c c07f8000 00000000 00000000 c07f9f68 c0854ad8
[173606.598378] 9f38 00000000 c00442b4 c080dd30 c003d390 ffffffff f2a00100 00000407 00000002
[173606.598989] 9f58 00000001 c07f8000 00000000 c0042a4c 00000001 800f0093 c07f9f98 f40c4000
[173606.599652] 9f78 c07f8000 c0854944 c080da6c c060c93c c0815a1c 412fc09a 00000000 00000000
[173606.600317] 9f98 00000000 c07f9fb0 c060a7a0 c0043b3c 400f0013 ffffffff c0043b18 c0043e5c
[173606.600930] 9fb8 c080dbac 00000000 c0036894 c28851c0 1000406a c00088b0 c00082f8 00000000
[173606.601543] 9fd8 00000000 c0036894 00000000 10c53c7d c080da8c c0036890 c0815a14 10008040
[173606.602206] 9ff8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.602866]
[173606.602868] R3: 0xf40c3f80:
[173606.602991] 3f80 79c179c1 79c179c1 79c179c1 79c179c1 79c179c1 79c179c1 79c179c1 79c179c1
[173606.603660] 3fa0 819a819a 819a819a 819a819a 819a819a 819a819a 819a819a 819a819a 819a819a
[173606.604330] 3fc0 89738973 89738973 89738973 89738973 89738973 89738973 89738973 89738973
[173606.604936] 3fe0 908d908d 908d908d 908d908d 908d908d 908d908d 908d908d 908d908d 908d908d
[173606.605599] 4000 040110ff 00000000 00000038 00000000 00000000 00018d00 00011204 00900002
[173606.606261] 4020 02b92b06 00490b00 0ec102c1 007312c1 33e77f92 00012093 00012090 00010841
[173606.606873] 4040 00010241 00000000 00000000 00000003 00000000 00200038 045a0000 ffffffff
[173606.607483] 4060 010e00db 0002fe62 0040001f 03300c00 01ff3000 3770300f 0000f300 0f3000c1
[173606.608150]
[173606.608152] R4: 0xc07f7f80:
[173606.608275] 7f80 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.608944] 7fa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.609607] 7fc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.610264] 7fe0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.610871] 8000 00000000 00000004 00000000 c0814920 c0821f54 00000000 00000015 c2889100
[173606.611533] 8020 c0814920 ccb920c0 c07f8000 00000000 ca33aa20 00000000 c07f9fb4 c07f9f00
[173606.612198] 8040 c06085fc 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.612805] 8060 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.613470]
[173606.613472] R5: 0xc08548c4:
[173606.613649] 48c4 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.614261] 48e4 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.614921] 4904 00000000 00000000 00000000 00000000 0007a354 00000000 00000001 00000000
[173606.615583] 4924 ca01c1b0 00000000 00000000 00000000 00000003 00000000 00000000 00000000
[173606.616243] 4944 00000000 c00605cc 00000000 00000000 c08156d4 00000000 00000000 f21f1b29
[173606.616852] 4964 250fb1d4 00063015 c005055c 006c3776 00000000 c071bc40 00000000 00000001
[173606.617514] 4984 c0043b3c 400f0013 41ab2ef8 4168cf92 000f0030 42324890 401035c8 800f0010
[173606.618180] 49a4 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000001
[173606.618786]
[173606.618788] R6: 0xc080d9ec:
[173606.618963] d9ec 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.619620] da0c 00000000 00000000 00000000 00000000 00000000 00000001 00000000 0000b0d7
[173606.620237] da2c c0050404 c0050444 c0780000 c00501c0 c005016c c004ff00 c004ffac c004ffc4
[173606.620852] da4c c004ffc4 c004ffc8 c004ffc8 c005004c c0050144 c0050154 c005010c 00000012
[173606.621521] da6c c00506d8 c0050838 c005078c c0050668 c005090c c0050620 c00506a8 00000000
[173606.622182] da8c 00000f8c 00000000 f2a01000 f2a00100 f2a00100 000003e8 00000000 00003c6f
[173606.622786] daac 00000001 00989680 c071e910 c071e928 c071e934 c071e944 c071e954 c071e960
[173606.623448] dacc c071e96c c071e97c c071e988 c071e990 c071e99c c071e9a4 c071e9b0 c071e9c0
[173606.624112]
[173606.624114] R7: 0xc060c8bc:
[173606.624235] c8bc 00000004 00000008 00000010 00000020 00000040 00000080 00000100 00000200
[173606.624845] c8dc 00000400 00000800 00001000 00002000 00004000 00008000 00010000 00020000
[173606.625510] c8fc 00040000 00080000 00100000 00200000 00400000 00800000 01000000 02000000
[173606.626120] c91c 04000000 08000000 10000000 20000000 40000000 80000000 c080db20 c080db18
[173606.626790] c93c c080db14 c080db1c 0000000f 00000063 c00826ac c0082028 c008265c c0081f88
[173606.627455] c95c 00000000 c0122174 c0122294 00000000 00000000 00000000 00000000 00000000
[173606.628117] c97c 00000000 00000000 00000000 c0081f58 00000000 c0121bbc 00000000 00000000
[173606.628722] c99c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.629331]
[173606.629333] R8: 0xc081599c:
[173606.629454] 599c 00000000 c0041e64 00000000 c0041d14 00000000 c0041c9c 00000000 c0041c74
[173606.630116] 59bc 00000000 c0041914 00000000 c004179c 00000000 c00410c4 00000000 00000000
[173606.630778] 59dc 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[173606.631444] 59fc 00000000 00000000 00000000 00000000 00000000 c0042f00 10c53c7d 10c53c7d
[173606.632057] 5a1c 00000068 c0043b18 c0043c10 c0815b24 c0815a44 0fffffff 07f001f0 00000020
[173606.632727] 5a3c 00000000 c0044d58 c0815a28 c0815a60 0000ffff 0000de01 00000020 00000020
[173606.633339] 5a5c c0044d58 c0815a44 c0815b24 0000ffff 0000f7f0 00000020 00000020 c0044ce8
[173606.634000] 5a7c 00000000 00000000 c0717cf8 00000200 00000000 00000000 00000000 1003d000
[173606.634613] IPI backtrace for cpu 2
[173606.634692] Unhandled fault: imprecise external abort (0x1406) at 0x41ab3004
[173606.634702] Internal error: : 1406 [#1] PREEMPT SMP
[173606.634709] Modules linked in: rtk_btusb
[173606.634719] CPU: 0 Not tainted (3.0.35 #9)
[173606.634727] PC is at default_idle+0x24/0x28
[173606.634735] LR is at _raw_spin_unlock_irqrestore+0x24/0x44
[173606.634743] pc : [<c0043b3c>] lr : [<c060a7a0>] psr: 400f0013
[173606.634748] sp : c07f9fb0 ip : 00000000 fp : 00000000
[173606.634754] r10: 00000000 r9 : 412fc09a r8 : c0815a1c
[173606.634761] r7 : c060c93c r6 : c080da6c r5 : c0854944 r4 : c07f8000
[173606.634769] r3 : f40c4000 r2 : c07f9f98 r1 : 800f0093 r0 : 00000001
[173606.634777] Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
[173606.634786] Control: 10c53c7d Table: 1a63c04a DAC: 00000015

[igorpadykov]

Hi seekwok

first may be recommended to run ddr test

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX-6-7-DDR-Stress-Test-Tool/ta-p/11082...

Also Linux version 3.0.35 is very old kernel and " rcu stall" issues with old kernels may

point to gpu & arm errata, which are fixed in latest nxp linux releases described on

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applicat...

"#define CONFIG_ARM_ERRATA"

https://source.codeaurora.org/external/imx/uboot-imx/tree/include/configs/mx6_common.h?h=nxp%2Fimx_v...

Best regards
igor

[seekwok]

Hi igor,

thanks  for your advice;

but our engineer reply the below information:

1. This device has been in the market for a long time. The customer will not agree to the upgrade.

2. The client's application development is based on Android 4.4 version system(which are fixed in latest nxp linux releases described on)

3. Android system upgrade cycle is a little long. Client need to deal with it in time(which are fixed in latest nxp linux releases described on)

4. The higher version system does not run smoothly on imx6. It is not conducive to the terminal customers.(which are fixed in latest nxp linux releases described on)

5. Now, Client need to deal with the problems in time.

6. DDR tests are all OK. Is there an up-to-date testing tool available.(as:first may be recommended to run ddr test)

it's any advice?wait for a good new......

Best regards

seekwok

[igorpadykov]

Hi seekwok

support for old kernels may be provided by Professional Services

https://contact.nxp.com/new-prof-svcs-sw-tech

Best regards
igor

[seekwok]

ok ,thanks 

Best regards

seekwok

[Ryze]

Hi,Seekwok

Our board is i.mx6q + 1G DDR3L, I meet the same problem now.Did you solve the problem?Do you have any idea? 

Best regards

Wenqing

[seekwok]

this is the message from  customer side.thanks