topic Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37 in i.MX Processors

rcu_preempt self-detected stall on CPU and usb xhci event error 37

jeffin_joys — Sun, 09 Feb 2020 08:24:39 GMT

Dear NXP Team,

We are working on kernel 4.14.98 (GA BSP) on iMX8DX dual core NXP processor. We are using this device to connect to LTE using xhci interface via usb using pppd. Currently we are facing two issues

RCU stall warnings and hence system hangs after seeing crashes. The issue occurrence is random but exists. Sometimes we observe RCU stall warnings immediately after device boots up and sometimes we observe after running the device for longs hours (> 12 hours). This issue was not present when we used L4.14.62 Beta BSP.

USB Event Ring Full error. “xhci-cdns3: ERROR unknown event type 37”. We get continuous prints in our serial console making system unstable after keeping devices for long hours.

One the above issues is being produced when the device boots up using GA BSP L4.14.98.

We have tried applying the patch as mentioned in below link for fair.c file in kernel for RCU crash.

https://lore.kernel.org/lkml/20180426103133.GA6953@linaro.org/T/#md99127a919a18d23868e6ffe9c572a38b9587159

For USB event error, below are patches used. However we observe LTE ping command not working sometimes

https://patchwork.kernel.org/patch/10880073/

https://patchwork.kernel.org/patch/11165607/

With above patches, issue is getting reproduced less frequent, but still exist.

Can you please help us in how we can resolve the issue. How to debug and narrow down the root cause. Without resolving this issue, we are blocked.

Attached Logs for more information:

INFO: rcu_preempt self-detected stall on CPU

[ 1076.614300] INFO: rcu_preempt detected stalls on CPUs/tasks:

[ 1076.616581] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=1438

[ 1076.622233] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=1438

[ 1076.629359]

[ 1076.636490]

[ 1076.636493] (t=5253 jiffies g=14886 c=14885 q=74)

[ 1076.636497] Task dump for CPU 0:

[ 1076.638075] (detected by 1, t=5253 jiffies, g=14886, c=14885, q=74)

[ 1076.639646] swapper/0 R running task 0 0 0 0x00000002

[ 1076.660972] Call trace:

[ 1076.663426] [<ffff000008089668>] dump_backtrace+0x0/0x3c8

[ 1076.668831] [<ffff000008089a44>] show_stack+0x14/0x20

[ 1076.673886] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180

[ 1076.679549] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50

[ 1076.684868] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4

[ 1076.690705] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958

[ 1076.696716] [<ffff00000812a1f4>] update_process_times+0x2c/0x58

[ 1076.702643] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48

[ 1076.708913] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90

[ 1076.714490] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168

[ 1076.720502] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230

[ 1076.726256] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48

[ 1076.732441] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138

[ 1076.738711] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38

[ 1076.744462] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8

[ 1076.750299] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174

[ 1076.755789] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)

[ 1076.762236] 3d80: 0000000000000000 ffff000008d07880

[ 1076.770073] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002

[ 1076.777909] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70

[ 1076.785745] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750

[ 1076.793581] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000

[ 1076.801418] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880

[ 1076.809254] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8

[ 1076.817090] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0

[ 1076.824926] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145

[ 1076.832762] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028

[ 1076.840597] 3ec0: ffff000008003ed0 ffff000008081184

[ 1076.845480] [<ffff000008082a30>] el1_irq+0xb0/0x124

[ 1076.850362] [<ffff000008081184>] __do_softirq+0xa4/0x218

[ 1076.855681] [<ffff0000080d0340>] irq_exit+0xd0/0xf0

[ 1076.860561] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8

[ 1076.866399] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174

[ 1076.871888] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)

[ 1076.878327] 3dc0: 0000000000000000 0000000000000000

[ 1076.886164] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00

[ 1076.894001] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90

[ 1076.901836] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000

[ 1076.909673] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000

[ 1076.917509] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000

[ 1076.925345] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000

[ 1076.933181] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10

[ 1076.941017] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145

[ 1076.948853] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc

[ 1076.956688] 3f00: ffff000008c83f10 ffff000008084f00

[ 1076.961570] [<ffff000008082a30>] el1_irq+0xb0/0x124

[ 1076.966454] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18

[ 1076.971770] [<ffff000008106f50>] do_idle+0x120/0x1e0

[ 1076.976739] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28

[ 1076.982406] [<ffff00000886b314>] rest_init+0xcc/0xd8

[ 1076.987377] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8

[ 1076.992774] Task dump for CPU 0:

[ 1076.995997] swapper/0 R running task 0 0 0 0x00000022

[ 1077.003050] Call trace:

[ 1077.005497] [<ffff0000080853ac>] __switch_to+0x94/0xd8

Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37

igorpadykov — Mon, 10 Feb 2020 05:58:19 GMT

Hi Jeffin

such xhci error is described on

Re: kernel: xhci_hcd 0000:00:14.0: ERROR unknown event type 37 - Kernel 4.19.13 — Linux USB

Regarding "kernel 4.14.98 (GA BSP) on iMX8DX" - it is recommended to try latest rev.C0

production part (ended with "AC") and with L4.14.98_2.3.0 kernel (currently only kernel supporting C0)

linux-imx - i.MX Linux kernel

Linux 4.14.98_2.3.0 Documentation

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37

xinyu_chen — Mon, 10 Feb 2020 12:02:10 GMT

Is it possible to provide the full RCU stall warning log?

Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37

igorpadykov — Mon, 10 Feb 2020 12:55:24 GMT

as this is general linux issue please post it on kernel mail list

Majordomo Lists at VGER.KERNEL.ORG

Best regards
igor

Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37

xinyu_chen — Mon, 10 Feb 2020 13:04:43 GMT

No, RCU stall can be caused by various of delay, live-lock in kernel/driver, etc.

Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37

jeffin_joys — Mon, 10 Feb 2020 14:12:04 GMT

Dear NXP team,

Thanks for your analysis and reply. I am attaching the log captured for crash occurred. This behavior we didn't observed when we used L4.14.62 Beta BSP.

Logs:

root@imx8dx_ccu:~#
[ 1076.614300] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1076.616581] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=1438
[ 1076.622233] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=1438
[ 1076.629359]
[ 1076.636490]
[ 1076.636493] (t=5253 jiffies g=14886 c=14885 q=74)
[ 1076.636497] Task dump for CPU 0:
[ 1076.638075] (detected by 1, t=5253 jiffies, g=14886, c=14885, q=74)
[ 1076.639646] swapper/0       R running task        0     0      0 0x00000002
[ 1076.660972] Call trace:
[ 1076.663426] [<ffff000008089668>] dump_backtrace+0x0/0x3c8
[ 1076.668831] [<ffff000008089a44>] show_stack+0x14/0x20
[ 1076.673886] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180
[ 1076.679549] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50
[ 1076.684868] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4
[ 1076.690705] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958
[ 1076.696716] [<ffff00000812a1f4>] update_process_times+0x2c/0x58
[ 1076.702643] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48
[ 1076.708913] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90
[ 1076.714490] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168
[ 1076.720502] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230
[ 1076.726256] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48
[ 1076.732441] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138
[ 1076.738711] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38
[ 1076.744462] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8
[ 1076.750299] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1076.755789] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)
[ 1076.762236] 3d80:                                   0000000000000000 ffff000008d07880
[ 1076.770073] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002
[ 1076.777909] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70
[ 1076.785745] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750
[ 1076.793581] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000
[ 1076.801418] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880
[ 1076.809254] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8
[ 1076.817090] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0
[ 1076.824926] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145
[ 1076.832762] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028
[ 1076.840597] 3ec0: ffff000008003ed0 ffff000008081184
[ 1076.845480] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1076.850362] [<ffff000008081184>] __do_softirq+0xa4/0x218
[ 1076.855681] [<ffff0000080d0340>] irq_exit+0xd0/0xf0
[ 1076.860561] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8
[ 1076.866399] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1076.871888] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)
[ 1076.878327] 3dc0:                                   0000000000000000 0000000000000000
[ 1076.886164] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00
[ 1076.894001] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90
[ 1076.901836] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000
[ 1076.909673] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000
[ 1076.917509] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000
[ 1076.925345] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000
[ 1076.933181] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10
[ 1076.941017] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145
[ 1076.948853] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc
[ 1076.956688] 3f00: ffff000008c83f10 ffff000008084f00
[ 1076.961570] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1076.966454] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18
[ 1076.971770] [<ffff000008106f50>] do_idle+0x120/0x1e0
[ 1076.976739] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28
[ 1076.982406] [<ffff00000886b314>] rest_init+0xcc/0xd8
[ 1076.987377] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8
[ 1076.992774] Task dump for CPU 0:
[ 1076.995997] swapper/0       R running task        0     0      0 0x00000022
[ 1077.003050] Call trace:
[ 1077.005497] [<ffff0000080853ac>] __switch_to+0x94/0xd8

root@imx8dx_ccu:~# uptime
09:44:04 up 18 min, load average: 6.35, 2.54, 1.33
root@imx8dx_ccu:~# [ 1139.642737] INFO: rcu_preempt self-detected stall on CPU
[ 1139.643202] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1139.648077] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=5329
[ 1139.653729] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=5329
[ 1139.660855]
[ 1139.667986]
[ 1139.667989] (t=21008 jiffies g=14886 c=14885 q=473)
[ 1139.667993] Task dump for CPU 0:
[ 1139.669571] (detected by 1, t=21008 jiffies, g=14886, c=14885, q=473)
[ 1139.671142] swapper/0       R running task        0     0      0 0x00000022
[ 1139.692816] Call trace:
[ 1139.695269] [<ffff000008089668>] dump_backtrace+0x0/0x3c8
[ 1139.700674] [<ffff000008089a44>] show_stack+0x14/0x20
[ 1139.705730] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180
[ 1139.711392] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50
[ 1139.716712] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4
[ 1139.722548] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958
[ 1139.728560] [<ffff00000812a1f4>] update_process_times+0x2c/0x58
[ 1139.734486] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48
[ 1139.740757] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90
[ 1139.746334] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168
[ 1139.752345] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230
[ 1139.758099] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48
[ 1139.764284] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138
[ 1139.770554] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38
[ 1139.776305] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8
[ 1139.782143] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1139.787633] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)
[ 1139.794080] 3d80:                                   0000000000000000 ffff000008d07880
[ 1139.801916] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002
[ 1139.809753] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70
[ 1139.817589] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750
[ 1139.825425] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000
[ 1139.833261] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880
[ 1139.841098] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8
[ 1139.848933] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0
[ 1139.856770] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145
[ 1139.864606] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028
[ 1139.872440] 3ec0: ffff000008003ed0 ffff000008081184
[ 1139.877323] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1139.882205] [<ffff000008081184>] __do_softirq+0xa4/0x218
[ 1139.887524] [<ffff0000080d0340>] irq_exit+0xd0/0xf0
[ 1139.892405] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8
[ 1139.898242] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1139.903732] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)
[ 1139.910170] 3dc0:                                   0000000000000000 0000000000000000
[ 1139.918008] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00
[ 1139.925844] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90
[ 1139.933679] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000
[ 1139.941516] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000
[ 1139.949352] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000
[ 1139.957189] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000
[ 1139.965024] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10
[ 1139.972861] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145
[ 1139.980697] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc
[ 1139.988531] 3f00: ffff000008c83f10 ffff000008084f00
[ 1139.993414] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1139.998297] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18
[ 1140.003614] [<ffff000008106f50>] do_idle+0x120/0x1e0
[ 1140.008582] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28
[ 1140.014250] [<ffff00000886b314>] rest_init+0xcc/0xd8
[ 1140.019219] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8
[ 1140.024617] Task dump for CPU 0:
[ 1140.027840] swapper/0       R running task        0     0      0 0x00000022
[ 1140.034894] Call trace:
[ 1140.037340] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1147.864375] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1147.869882] 0-...: (1 GPs behind) idle=e42/2/0 softirq=38269/40214 fqs=1276
[ 1147.877013] (detected by 1, t=5255 jiffies, g=433, c=432, q=3)
[ 1147.882939] Task dump for CPU 0:
[ 1147.886163] swapper/0       R running task        0     0      0 0x00000022
[ 1147.893216] Call trace:
[ 1147.895670] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1202.671740] INFO: rcu_preempt self-detected stall on CPU
[ 1202.672172] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1202.677080] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=9038
[ 1202.682740] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=9038
[ 1202.689866]
[ 1202.696997]
[ 1202.697000] (t=36763 jiffies g=14886 c=14885 q=1115)
[ 1202.697004] Task dump for CPU 0:
[ 1202.698583] (detected by 1, t=36763 jiffies, g=14886, c=14885, q=1115)
[ 1202.700153] swapper/0       R running task        0     0      0 0x00000022
[ 1202.722001] Call trace:
[ 1202.724454] [<ffff000008089668>] dump_backtrace+0x0/0x3c8
[ 1202.729859] [<ffff000008089a44>] show_stack+0x14/0x20
[ 1202.734914] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180
[ 1202.740577] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50
[ 1202.745897] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4
[ 1202.751733] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958
[ 1202.757745] [<ffff00000812a1f4>] update_process_times+0x2c/0x58
[ 1202.763672] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48
[ 1202.769942] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90
[ 1202.775519] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168
[ 1202.781530] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230
[ 1202.787284] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48
[ 1202.793469] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138
[ 1202.799740] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38
[ 1202.805490] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8
[ 1202.811328] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1202.816818] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)
[ 1202.823265] 3d80:                                   0000000000000000 ffff000008d07880
[ 1202.831102] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002
[ 1202.838938] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70
[ 1202.846774] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750
[ 1202.854610] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000
[ 1202.862446] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880
[ 1202.870283] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8
[ 1202.878119] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0
[ 1202.885955] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145
[ 1202.893791] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028
[ 1202.901625] 3ec0: ffff000008003ed0 ffff000008081184
[ 1202.906508] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1202.911390] [<ffff000008081184>] __do_softirq+0xa4/0x218
[ 1202.916709] [<ffff0000080d0340>] irq_exit+0xd0/0xf0
[ 1202.921590] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8
[ 1202.927427] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1202.932917] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)
[ 1202.939355] 3dc0:                                   0000000000000000 0000000000000000
[ 1202.947193] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00
[ 1202.955029] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90
[ 1202.962865] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000
[ 1202.970701] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000
[ 1202.978537] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000
[ 1202.986374] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000
[ 1202.994210] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10
[ 1203.002046] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145
[ 1203.009882] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc
[ 1203.017716] 3f00: ffff000008c83f10 ffff000008084f00
[ 1203.022599] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1203.027483] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18
[ 1203.032799] [<ffff000008106f50>] do_idle+0x120/0x1e0
[ 1203.037767] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28
[ 1203.043434] [<ffff00000886b314>] rest_init+0xcc/0xd8
[ 1203.048405] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8
[ 1203.053803] Task dump for CPU 0:
[ 1203.057025] swapper/0       R running task        0     0      0 0x00000022
[ 1203.064079] Call trace:
[ 1203.066526] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1209.445201] INFO: task kworker/0:2:146 blocked for more than 120 seconds.
[ 1209.452100]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 1209.460007] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1209.467908] kworker/0:2     D    0   146      2 0x00000020
[ 1209.473482] Workqueue: pm pm_runtime_work
[ 1209.477545] Call trace:
[ 1209.480026] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1209.485289] [<ffff00000886c698>] __schedule+0x1a8/0x5e8
[ 1209.490585] [<ffff00000886cb10>] schedule+0x38/0xa0
[ 1209.495534] [<ffff00000887028c>] schedule_timeout+0x74/0x320
[ 1209.501287] [<ffff0000081297b8>] msleep+0x28/0x38
[ 1209.506113] [<ffff00000853ba8c>] xhci_bus_resume+0x40c/0x448
[ 1209.511877] [<ffff000008504890>] hcd_bus_resume+0x50/0x188
[ 1209.517458] [<ffff000008504af8>] hcd_bus_suspend+0x130/0x138
[ 1209.523208] [<ffff000008514984>] generic_suspend+0x4c/0x58
[ 1209.529610] [<ffff000008509864>] usb_suspend_both+0x18c/0x208
[ 1209.535457] [<ffff00000850a89c>] usb_runtime_suspend+0x2c/0x70
[ 1209.541391] [<ffff0000084c32d0>] __rpm_callback+0xe0/0x268
[ 1209.546967] [<ffff0000084c3478>] rpm_callback+0x20/0x80
[ 1209.552247] [<ffff0000084c3734>] rpm_suspend+0xf4/0x4b8
[ 1209.557563] [<ffff0000084c46ec>] __pm_runtime_suspend+0x5c/0xa0
[ 1209.563540] [<ffff00000850a924>] usb_runtime_idle+0x24/0x30
[ 1209.569285] [<ffff0000084c32d0>] __rpm_callback+0xe0/0x268
[ 1209.574782] [<ffff0000084c35c4>] rpm_idle+0xec/0x168
[ 1209.580118] [<ffff0000084c4048>] pm_runtime_work+0xa0/0xb8
[ 1209.585702] [<ffff0000080e4668>] process_one_work+0x188/0x300
[ 1209.592382] [<ffff0000080e482c>] worker_thread+0x4c/0x420
[ 1209.597947] [<ffff0000080ea5a4>] kthread+0x12c/0x130
[ 1209.603074] [<ffff000008084740>] ret_from_fork+0x10/0x18
[ 1209.608475] INFO: task syslogd:1819 blocked for more than 120 seconds.
[ 1209.615797]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 1209.623670] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1209.631688] syslogd         D    0 1819      1 0x00000200
[ 1209.637296] Call trace:
[ 1209.639755] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1209.645071] [<ffff00000886c698>] __schedule+0x1a8/0x5e8
[ 1209.650354] [<ffff00000886cb10>] schedule+0x38/0xa0
[ 1209.655325] [<ffff0000080f5988>] io_schedule+0x18/0x38
[ 1209.660518] [<ffff00000886d4ac>] bit_wait_io+0x14/0x68
[ 1209.665748] [<ffff00000886d140>] __wait_on_bit+0x70/0xe8
[ 1209.671118] [<ffff00000886d224>] out_of_line_wait_on_bit+0x6c/0x78
[ 1209.677612] [<ffff0000082e8d4c>] do_get_write_access+0x2ac/0x4d8
[ 1209.683679] [<ffff0000082e8fc8>] jbd2_journal_get_write_access+0x50/0x70
[ 1209.690488] [<ffff000008297708>] __ext4_journal_get_write_access+0x48/0xa0
[ 1209.697493] [<ffff0000082b1780>] ext4_reserve_inode_write+0xa8/0xf8
[ 1209.703839] [<ffff0000082b19c0>] ext4_mark_inode_dirty+0x28/0x148
[ 1209.710071] [<ffff0000082b5a8c>] ext4_dirty_inode+0x44/0x70
[ 1209.715740] [<ffff000008244efc>] __mark_inode_dirty+0x3c/0x210
[ 1209.721629] [<ffff000008232fa8>] generic_update_time+0x68/0xb0
[ 1209.728134] [<ffff00000823331c>] file_update_time+0xbc/0xf0
[ 1209.733824] [<ffff000008196ce8>] __generic_file_write_iter+0x88/0x1c8
[ 1209.740367] [<ffff0000082a15cc>] ext4_file_write_iter+0x10c/0x408
[ 1209.746568] [<ffff000008213f24>] __vfs_write+0xac/0x118
[ 1209.751870] [<ffff000008214184>] vfs_write+0xa4/0x1b0
[ 1209.756976] [<ffff000008214448>] SyS_write+0x48/0xb0
[ 1209.762108] Exception stack(0xffff00000e843ec0 to 0xffff00000e844000)
[ 1209.768603] 3ec0: 0000000000000004 000000000556b6d8 0000000000000070 0000ffffa0aca190
[ 1209.776553] 3ee0: 0000ffffa0baf300 0000000000000070 0000ffffa0bae000 0000000000071454
[ 1209.784438] 3f00: 0000000000000040 003b9aca00000000 000000005e3a8e20 00000000000ca4bc
[ 1209.792342] 3f20: 0000000000000018 00000003e8000000 002f90d5db6a8e80 0001d2a9c69c6a8c
[ 1209.800254] 3f40: 00000000004984c8 0000ffffa0a37d20 0000fffff8b8552f 0000000000000004
[ 1209.808817] 3f60: 0000000000000070 000000000556b6d8 0000000000000004 0000000000000006
[ 1209.816774] 3f80: 0000000000499000 0000000000000001 000000000556b4d8 000000000000005e
[ 1209.824782] 3fa0: 0000000000000000 0000fffff8b85720 0000000000408a84 0000fffff8b85720
[ 1209.832699] 3fc0: 0000ffffa0a37d4c 0000000020000000 0000000000000004 0000000000000040
[ 1209.840651] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1209.848542] [<ffff000008083318>] __sys_trace_return+0x0/0x4
[ 1209.854235] INFO: task jbd2/mmcblk0p12:1987 blocked for more than 120 seconds.
[ 1209.861513]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 1209.869328] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1209.878794] jbd2/mmcblk0p12 D    0 1987      2 0x00000200
[ 1209.884433] Call trace:
[ 1209.887065] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1209.892260] [<ffff00000886c698>] __schedule+0x1a8/0x5e8
[ 1209.897579] [<ffff00000886cb10>] schedule+0x38/0xa0
[ 1209.902563] [<ffff0000080f5988>] io_schedule+0x18/0x38
[ 1209.907811] [<ffff00000886d4ac>] bit_wait_io+0x14/0x68
[ 1209.913000] [<ffff00000886d140>] __wait_on_bit+0x70/0xe8
[ 1209.918399] [<ffff00000886d224>] out_of_line_wait_on_bit+0x6c/0x78
[ 1209.924638] [<ffff00000824c4e4>] __wait_on_buffer+0x2c/0x38
[ 1209.930303] [<ffff0000082eb5f8>] jbd2_journal_commit_transaction+0xbc8/0x1320
[ 1209.937495] [<ffff0000082ef024>] kjournald2+0xbc/0x230
[ 1209.945355] [<ffff0000080ea5a4>] kthread+0x12c/0x130
[ 1209.950473] [<ffff000008084740>] ret_from_fork+0x10/0x18
[ 1209.956011] INFO: task java:2219 blocked for more than 120 seconds.
[ 1209.962391]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 1209.970291] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1209.978443] java            D    0 2219   2108 0x00000200
[ 1209.984012] Call trace:
[ 1209.986528] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1209.991723] [<ffff00000886c698>] __schedule+0x1a8/0x5e8
[ 1209.997021] [<ffff00000886cb10>] schedule+0x38/0xa0
[ 1210.001954] [<ffff0000080f5988>] io_schedule+0x18/0x38
[ 1210.007167] [<ffff00000886d4ac>] bit_wait_io+0x14/0x68
[ 1210.012429] [<ffff00000886d140>] __wait_on_bit+0x70/0xe8
[ 1210.017857] [<ffff00000886d224>] out_of_line_wait_on_bit+0x6c/0x78
[ 1210.024151] [<ffff0000082e8d4c>] do_get_write_access+0x2ac/0x4d8
[ 1210.030252] [<ffff0000082e8fc8>] jbd2_journal_get_write_access+0x50/0x70
[ 1210.037074] [<ffff000008297708>] __ext4_journal_get_write_access+0x48/0xa0
[ 1210.044024] [<ffff0000082b1780>] ext4_reserve_inode_write+0xa8/0xf8
[ 1210.050362] [<ffff0000082b19c0>] ext4_mark_inode_dirty+0x28/0x148
[ 1210.056548] [<ffff0000082b5a8c>] ext4_dirty_inode+0x44/0x70
[ 1210.062178] [<ffff000008244efc>] __mark_inode_dirty+0x3c/0x210
[ 1210.068125] [<ffff00000824e2c0>] generic_write_end+0x98/0xd8
[ 1210.073895] [<ffff0000082b4810>] ext4_da_write_end+0x118/0x258
[ 1210.082133] [<ffff0000081946bc>] generic_perform_write+0xf4/0x178
[ 1210.088248] [<ffff000008196d60>] __generic_file_write_iter+0x100/0x1c8
[ 1210.095362] [<ffff0000082a15cc>] ext4_file_write_iter+0x10c/0x408
[ 1210.101590] [<ffff000008213f24>] __vfs_write+0xac/0x118
[ 1210.106923] [<ffff000008214184>] vfs_write+0xa4/0x1b0
[ 1210.112023] [<ffff000008214448>] SyS_write+0x48/0xb0
[ 1210.117070] Exception stack(0xffff00000f243ec0 to 0xffff00000f244000)
[ 1210.123599] 3ec0: 0000000000000016 0000ffffb86e7d88 00000000000007d6 0000000000000000
[ 1210.131523] 3ee0: 00000000ffffffbb 0000ffffb86eb0e0 0000000000000000 0000000000000016
[ 1210.139484] 3f00: 0000000000000040 2064616552202020 6e656c2061746164 0a39203a20687467
[ 1210.147447] 3f20: 2020202020202020 6164206461655220 0000000000000008 00000000000007d6
[ 1210.155334] 3f40: 0000ffffba7663e8 0000ffffbb60ad20 000000000000000e 0000000000000016
[ 1210.163297] 3f60: 00000000000007d6 0000ffffb86e7d88 0000000000000016 0000ffffb86e7d88
[ 1210.171190] 3f80: 0000ffffb86e9d90 0000000000000042 0000ffffac0d7480 0000ffffb86e7d90
[ 1210.179116] 3fa0: 0000ffffba765000 0000ffffb86e7cb0 0000ffffbb60ad78 0000ffffb86e7cb0
[ 1210.187020] 3fc0: 0000ffffbb60ad90 0000000080000000 0000000000000016 0000000000000040
[ 1210.194950] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.202895] [<ffff000008083318>] __sys_trace_return+0x0/0x4
[ 1210.208597] INFO: task trace_receiver:2154 blocked for more than 120 seconds.
[ 1210.215790]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 1210.223642] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1210.231577] trace_receiver D    0 2154      1 0x00400200
[ 1210.237141] Call trace:
[ 1210.239662] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1210.244879] [<ffff00000886c698>] __schedule+0x1a8/0x5e8
[ 1210.250191] [<ffff00000886cb10>] schedule+0x38/0xa0
[ 1210.255126] [<ffff0000080f5988>] io_schedule+0x18/0x38
[ 1210.260355] [<ffff00000886d4ac>] bit_wait_io+0x14/0x68
[ 1210.265611] [<ffff00000886d140>] __wait_on_bit+0x70/0xe8
[ 1210.271004] [<ffff00000886d224>] out_of_line_wait_on_bit+0x6c/0x78
[ 1210.277237] [<ffff0000082e8d4c>] do_get_write_access+0x2ac/0x4d8
[ 1210.283980] [<ffff0000082e8fc8>] jbd2_journal_get_write_access+0x50/0x70
[ 1210.290793] [<ffff000008297708>] __ext4_journal_get_write_access+0x48/0xa0
[ 1210.297746] [<ffff0000082b1780>] ext4_reserve_inode_write+0xa8/0xf8
[ 1210.304071] [<ffff0000082b19c0>] ext4_mark_inode_dirty+0x28/0x148
[ 1210.310240] [<ffff0000082b5a8c>] ext4_dirty_inode+0x44/0x70
[ 1210.316615] [<ffff000008244efc>] __mark_inode_dirty+0x3c/0x210
[ 1210.322584] [<ffff000008232fa8>] generic_update_time+0x68/0xb0
[ 1210.328585] [<ffff00000823331c>] file_update_time+0xbc/0xf0
[ 1210.334301] [<ffff000008196ce8>] __generic_file_write_iter+0x88/0x1c8
[ 1210.340797] [<ffff0000082a15cc>] ext4_file_write_iter+0x10c/0x408
[ 1210.347016] [<ffff000008213f24>] __vfs_write+0xac/0x118
[ 1210.352297] [<ffff000008214184>] vfs_write+0xa4/0x1b0
[ 1210.357442] [<ffff000008214448>] SyS_write+0x48/0xb0
[ 1210.362458] Exception stack(0xffff00000f09bec0 to 0xffff00000f09c000)
[ 1210.368967] bec0: 0000000000000015 00000000f609d624 0000000000000059 0000000000000000
[ 1210.376873] bee0: 0000000000000059 00000000f609d624 0000000000000015 0000000000000004
[ 1210.384798] bf00: 00000000f10db440 0000000000000000 0000000000000000 00000000f10daa34
[ 1210.392763] bf20: 0000000000000000 00000000f10daa00 0000000000000000 0000000000000000
[ 1210.400674] bf40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.409441] bf60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.417460] bf80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.425422] bfa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.433469] bfc0: 00000000f5dfa4fc 0000000080000010 0000000000000015 0000000000000004
[ 1210.441344] bfe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.449316] [<ffff000008083318>] __sys_trace_return+0x0/0x4
[ 1210.455009] INFO: task DR:2193 blocked for more than 120 seconds.
[ 1210.461203]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 1210.469065] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1210.476971] DR              D    0 2193      1 0x00400200
[ 1210.483243] Call trace:
[ 1210.485879] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1210.491106] [<ffff00000886c698>] __schedule+0x1a8/0x5e8
[ 1210.496420] [<ffff00000886cb10>] schedule+0x38/0xa0
[ 1210.501494] [<ffff0000080f5988>] io_schedule+0x18/0x38
[ 1210.506690] [<ffff00000886d4ac>] bit_wait_io+0x14/0x68
[ 1210.511902] [<ffff00000886d140>] __wait_on_bit+0x70/0xe8
[ 1210.517374] [<ffff00000886d224>] out_of_line_wait_on_bit+0x6c/0x78
[ 1210.523597] [<ffff0000082e8d4c>] do_get_write_access+0x2ac/0x4d8
[ 1210.529759] [<ffff0000082e8fc8>] jbd2_journal_get_write_access+0x50/0x70
[ 1210.536540] [<ffff000008297708>] __ext4_journal_get_write_access+0x48/0xa0
[ 1210.544271] [<ffff0000082b1780>] ext4_reserve_inode_write+0xa8/0xf8
[ 1210.550697] [<ffff0000082b19c0>] ext4_mark_inode_dirty+0x28/0x148
[ 1210.556847] [<ffff0000082b5a8c>] ext4_dirty_inode+0x44/0x70
[ 1210.562493] [<ffff000008244efc>] __mark_inode_dirty+0x3c/0x210
[ 1210.568407] [<ffff000008232fa8>] generic_update_time+0x68/0xb0
[ 1210.574314] [<ffff000008234d88>] touch_atime+0x90/0xb0
[ 1210.579964] [<ffff0000081968b4>] generic_file_read_iter+0x67c/0x888
[ 1210.586545] [<ffff0000082a1904>] ext4_file_read_iter+0x3c/0x58
[ 1210.592759] [<ffff000008213bec>] __vfs_read+0xac/0x108
[ 1210.598787] [<ffff000008213cd8>] vfs_read+0x90/0x170
[ 1210.603888] [<ffff000008214398>] SyS_read+0x48/0xb0
[ 1210.610821] Exception stack(0xffff00000f1cbec0 to 0xffff00000f1cc000)
[ 1210.617302] bec0: 0000000000000016 00000000f0f9e4b8 0000000000000809 0000000000000000
[ 1210.625341] bee0: 0000000000000809 00000000f0f9e4b8 0000000000000016 0000000000000003
[ 1210.633181] bf00: 00000000f0f0a0c8 00000000f0f070c8 00000000000000fe 00000000f07976cc
[ 1210.641267] bf20: 0000000000000000 00000000f0797698 0000000000000000 0000000000000000
[ 1210.649162] bf40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.657175] bf60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.665066] bf80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.672988] bfa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.680874] bfc0: 00000000f5dfa5f8 00000000800e0010 0000000000000016 0000000000000003
[ 1210.688836] bfe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1210.696729] [<ffff000008083318>] __sys_trace_return+0x0/0x4
[ 1210.893349] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1210.898858] 0-...: (1 GPs behind) idle=e42/2/0 softirq=38269/40214 fqs=4974
[ 1210.905990] (detected by 1, t=21008 jiffies, g=433, c=432, q=3)
[ 1210.912003] Task dump for CPU 0:
[ 1210.915227] swapper/0       R running task        0     0      0 0x00000022
[ 1210.922280] Call trace:
[ 1210.924734] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1224.316086] cgroup: fork rejected by pids controller in /system.slice/ccu_osgi_loader.service
[ 1265.694238] INFO: rcu_preempt self-detected stall on CPU
[ 1265.699573] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=12777
[ 1265.706791]   (t=52517 jiffies g=14886 c=14885 q=1487)
[ 1265.711936] Task dump for CPU 0:
[ 1265.715160] swapper/0       R running task        0     0      0 0x00000022
[ 1265.722213] Call trace:
[ 1265.724667] [<ffff000008089668>] dump_backtrace+0x0/0x3c8
[ 1265.730072] [<ffff000008089a44>] show_stack+0x14/0x20
[ 1265.735127] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180
[ 1265.740790] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50
[ 1265.746109] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4
[ 1265.751946] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958
[ 1265.757958] [<ffff00000812a1f4>] update_process_times+0x2c/0x58
[ 1265.763884] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48
[ 1265.770154] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90
[ 1265.775731] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168
[ 1265.781743] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230
[ 1265.787496] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48
[ 1265.793682] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138
[ 1265.799952] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38
[ 1265.805703] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8
[ 1265.811541] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1265.817030] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)
[ 1265.823477] 3d80:                                   0000000000000000 ffff000008d07880
[ 1265.831314] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002
[ 1265.839150] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70
[ 1265.846986] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750
[ 1265.854822] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000
[ 1265.862659] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880
[ 1265.870495] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8
[ 1265.878331] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0
[ 1265.886167] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145
[ 1265.894003] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028
[ 1265.901838] 3ec0: ffff000008003ed0 ffff000008081184
[ 1265.906721] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1265.911603] [<ffff000008081184>] __do_softirq+0xa4/0x218
[ 1265.916922] [<ffff0000080d0340>] irq_exit+0xd0/0xf0
[ 1265.921802] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8
[ 1265.927640] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1265.933129] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)
[ 1265.939568] 3dc0:                                   0000000000000000 0000000000000000
[ 1265.947405] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00
[ 1265.955242] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90
[ 1265.963077] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000
[ 1265.970914] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000
[ 1265.978750] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000
[ 1265.986586] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000
[ 1265.994422] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10
[ 1266.002258] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145
[ 1266.010094] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc
[ 1266.017929] 3f00: ffff000008c83f10 ffff000008084f00
[ 1266.022811] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1266.027695] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18
[ 1266.033012] [<ffff000008106f50>] do_idle+0x120/0x1e0
[ 1266.037980] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28
[ 1266.043647] [<ffff00000886b314>] rest_init+0xcc/0xd8
[ 1266.048617] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8
[ 1273.942353] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1273.947860] 0-...: (1 GPs behind) idle=e42/2/0 softirq=38269/40214 fqs=8676
[ 1273.954991] (detected by 1, t=36770 jiffies, g=433, c=432, q=3)
[ 1273.961004] Task dump for CPU 0:
[ 1273.964228] swapper/0       R running task        0     0      0 0x00000022
[ 1273.971281] Call trace:
[ 1273.973735] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1328.723114] INFO: rcu_preempt self-detected stall on CPU
[ 1328.726175] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1328.728445] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=16410
[ 1328.728449]
[ 1328.734109] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=16410
[ 1328.741323] (t=68272 jiffies g=14886 c=14885 q=1696)
[ 1328.742894]
[ 1328.750114] Task dump for CPU 0:
[ 1328.755163] (detected by 1, t=68272 jiffies, g=14886, c=14885, q=1696)
[ 1328.756734] swapper/0       R running task        0     0      0 0x00000022
[ 1328.773534] Call trace:
[ 1328.775987] [<ffff000008089668>] dump_backtrace+0x0/0x3c8
[ 1328.781392] [<ffff000008089a44>] show_stack+0x14/0x20
[ 1328.786448] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180
[ 1328.792110] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50
[ 1328.797430] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4
[ 1328.803267] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958
[ 1328.809278] [<ffff00000812a1f4>] update_process_times+0x2c/0x58
[ 1328.815205] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48
[ 1328.821475] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90
[ 1328.827052] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168
[ 1328.833063] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230
[ 1328.838817] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48
[ 1328.845002] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138
[ 1328.851273] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38
[ 1328.857023] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8
[ 1328.862861] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1328.868351] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)
[ 1328.874789] 3d80:                                   0000000000000000 ffff000008d07880
[ 1328.882626] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002
[ 1328.890463] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70
[ 1328.898298] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750
[ 1328.906135] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000
[ 1328.913971] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880
[ 1328.921808] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8
[ 1328.929643] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0
[ 1328.937479] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145
[ 1328.945316] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028
[ 1328.953150] 3ec0: ffff000008003ed0 ffff000008081184
[ 1328.958033] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1328.962915] [<ffff000008081184>] __do_softirq+0xa4/0x218
[ 1328.968234] [<ffff0000080d0340>] irq_exit+0xd0/0xf0
[ 1328.973115] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8
[ 1328.978952] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1328.984442] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)
[ 1328.990880] 3dc0:                                   0000000000000000 0000000000000000
[ 1328.998717] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00
[ 1329.006554] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90
[ 1329.014389] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000
[ 1329.022226] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000
[ 1329.030062] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000
[ 1329.037898] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000
[ 1329.045734] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10
[ 1329.053570] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145
[ 1329.061406] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc
[ 1329.069241] 3f00: ffff000008c83f10 ffff000008084f00
[ 1329.074124] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1329.079007] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18
[ 1329.084324] [<ffff000008106f50>] do_idle+0x120/0x1e0
[ 1329.089292] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28
[ 1329.094959] [<ffff00000886b314>] rest_init+0xcc/0xd8
[ 1329.099929] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8
[ 1329.105327] Task dump for CPU 0:
[ 1329.108550] swapper/0       R running task        0     0      0 0x00000022
[ 1329.115603] Call trace:
[ 1329.118050] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1332.366759] INFO: task kworker/0:2:146 blocked for more than 120 seconds.
[ 1332.373724]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 1332.381685] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1332.389590] kworker/0:2     D    0   146      2 0x00000020
[ 1332.395150] Workqueue: pm pm_runtime_work
[ 1332.399206] Call trace:
[ 1332.401687] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1332.406897] [<ffff00000886c698>] __schedule+0x1a8/0x5e8
[ 1332.412192] [<ffff00000886cb10>] schedule+0x38/0xa0
[ 1332.417126] [<ffff00000887028c>] schedule_timeout+0x74/0x320
[ 1332.422861] [<ffff0000081297b8>] msleep+0x28/0x38
[ 1332.427840] [<ffff00000853ba8c>] xhci_bus_resume+0x40c/0x448
[ 1332.433607] [<ffff000008504890>] hcd_bus_resume+0x50/0x188
[ 1332.439251] [<ffff000008504af8>] hcd_bus_suspend+0x130/0x138
[ 1332.444991] [<ffff000008514984>] generic_suspend+0x4c/0x58
[ 1332.450531] [<ffff000008509864>] usb_suspend_both+0x18c/0x208
[ 1332.456351] [<ffff00000850a89c>] usb_runtime_suspend+0x2c/0x70
[ 1332.462240] [<ffff0000084c32d0>] __rpm_callback+0xe0/0x268
[ 1332.467827] [<ffff0000084c3478>] rpm_callback+0x20/0x80
[ 1332.473107] [<ffff0000084c3734>] rpm_suspend+0xf4/0x4b8
[ 1332.478425] [<ffff0000084c46ec>] __pm_runtime_suspend+0x5c/0xa0
[ 1332.484427] [<ffff00000850a924>] usb_runtime_idle+0x24/0x30
[ 1332.490076] [<ffff0000084c32d0>] __rpm_callback+0xe0/0x268
[ 1332.495616] [<ffff0000084c35c4>] rpm_idle+0xec/0x168
[ 1332.500743] [<ffff0000084c4048>] pm_runtime_work+0xa0/0xb8
[ 1332.506291] [<ffff0000080e4668>] process_one_work+0x188/0x300
[ 1332.512106] [<ffff0000080e482c>] worker_thread+0x4c/0x420
[ 1332.517563] [<ffff0000080ea5a4>] kthread+0x12c/0x130
[ 1332.522643] [<ffff000008084740>] ret_from_fork+0x10/0x18
[ 1332.528048] INFO: task syslogd:1819 blocked for more than 120 seconds.
[ 1332.534670]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 1332.542484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1332.550381] syslogd         D    0 1819      1 0x00000200
[ 1332.555920] Call trace:
[ 1332.558369] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1332.563690] [<ffff00000886c698>] __schedule+0x1a8/0x5e8
[ 1332.568989] [<ffff00000886cb10>] schedule+0x38/0xa0
[ 1332.573956] [<ffff0000080f5988>] io_schedule+0x18/0x38
[ 1332.579149] [<ffff00000886d4ac>] bit_wait_io+0x14/0x68
[ 1332.584360] [<ffff00000886d140>] __wait_on_bit+0x70/0xe8
[ 1332.589751] [<ffff00000886d224>] out_of_line_wait_on_bit+0x6c/0x78
[ 1332.596004] [<ffff0000082e8d4c>] do_get_write_access+0x2ac/0x4d8
[ 1332.602085] [<ffff0000082e8fc8>] jbd2_journal_get_write_access+0x50/0x70
[ 1332.608863] [<ffff000008297708>] __ext4_journal_get_write_access+0x48/0xa0
[ 1332.615793] [<ffff0000082b1780>] ext4_reserve_inode_write+0xa8/0xf8
[ 1332.622134] [<ffff0000082b19c0>] ext4_mark_inode_dirty+0x28/0x148
[ 1332.628866] [<ffff0000082b5a8c>] ext4_dirty_inode+0x44/0x70
[ 1332.634561] [<ffff000008244efc>] __mark_inode_dirty+0x3c/0x210
[ 1332.640481] [<ffff000008232fa8>] generic_update_time+0x68/0xb0
[ 1332.646419] [<ffff00000823331c>] file_update_time+0xbc/0xf0
[ 1332.652089] [<ffff000008196ce8>] __generic_file_write_iter+0x88/0x1c8
[ 1332.658625] [<ffff0000082a15cc>] ext4_file_write_iter+0x10c/0x408
[ 1332.664773] [<ffff000008213f24>] __vfs_write+0xac/0x118
[ 1332.670416] [<ffff000008214184>] vfs_write+0xa4/0x1b0
[ 1332.675529] [<ffff000008214448>] SyS_write+0x48/0xb0
[ 1332.680568] Exception stack(0xffff00000e843ec0 to 0xffff00000e844000)
[ 1332.687086] 3ec0: 0000000000000004 000000000556b6d8 0000000000000070 0000ffffa0aca190
[ 1332.695140] 3ee0: 0000ffffa0baf300 0000000000000070 0000ffffa0bae000 0000000000071454
[ 1332.703022] 3f00: 0000000000000040 003b9aca00000000 000000005e3a8e20 00000000000ca4bc
[ 1332.710949] 3f20: 0000000000000018 00000003e8000000 002f90d5db6a8e80 0001d2a9c69c6a8c
[ 1332.718832] 3f40: 00000000004984c8 0000ffffa0a37d20 0000fffff8b8552f 0000000000000004
[ 1332.726740] 3f60: 0000000000000070 000000000556b6d8 0000000000000004 0000000000000006
[ 1332.734875] 3f80: 0000000000499000 0000000000000001 000000000556b4d8 000000000000005e
[ 1332.742799] 3fa0: 0000000000000000 0000fffff8b85720 0000000000408a84 0000fffff8b85720
[ 1332.750671] 3fc0: 0000ffffa0a37d4c 0000000020000000 0000000000000004 0000000000000040
[ 1332.758706] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1332.766595] [<ffff000008083318>] __sys_trace_return+0x0/0x4
[ 1332.772253] INFO: task jbd2/mmcblk0p9-:1984 blocked for more than 120 seconds.
[ 1332.779576]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 1332.787394] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1332.795297] jbd2/mmcblk0p9- D    0 1984      2 0x00000200
[ 1332.800879] Call trace:
[ 1332.803379] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1332.808592] [<ffff00000886c698>] __schedule+0x1a8/0x5e8
[ 1332.813886] [<ffff00000886cb10>] schedule+0x38/0xa0
[ 1332.818926] [<ffff0000080f5988>] io_schedule+0x18/0x38
[ 1332.824162] [<ffff000008192abc>] wait_on_page_bit+0x10c/0x140
[ 1332.829966] [<ffff000008192bbc>] __filemap_fdatawait_range+0xcc/0x110
[ 1332.836482] [<ffff000008192c44>] filemap_fdatawait_keep_errors+0x1c/0x50
[ 1332.843258] [<ffff0000082eafc8>] jbd2_journal_commit_transaction+0x598/0x1320
[ 1332.850470] [<ffff0000082ef024>] kjournald2+0xbc/0x230
[ 1332.855703] [<ffff0000080ea5a4>] kthread+0x12c/0x130
[ 1332.860740] [<ffff000008084740>] ret_from_fork+0x10/0x18
[ 1332.866103] INFO: task jbd2/mmcblk0p12:1987 blocked for more than 120 seconds.
[ 1332.873417]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1
[ 1332.881294] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1332.889217] jbd2/mmcblk0p12 D    0 1987      2 0x00000200
[ 1332.894794] Call trace:
[ 1332.897244] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1332.902491] [<ffff00000886c698>] __schedule+0x1a8/0x5e8
[ 1332.907830] [<ffff00000886cb10>] schedule+0x38/0xa0
[ 1332.912785] [<ffff0000080f5988>] io_schedule+0x18/0x38
[ 1332.917994] [<ffff00000886d4ac>] bit_wait_io+0x14/0x68
[ 1332.923208] [<ffff00000886d140>] __wait_on_bit+0x70/0xe8
[ 1332.928593] [<ffff00000886d224>] out_of_line_wait_on_bit+0x6c/0x78
[ 1332.934851] [<ffff00000824c4e4>] __wait_on_buffer+0x2c/0x38
[ 1332.940478] [<ffff0000082eb5f8>] jbd2_journal_commit_transaction+0xbc8/0x1320
[ 1332.947804] [<ffff0000082ef024>] kjournald2+0xbc/0x230
[ 1332.953000] [<ffff0000080ea5a4>] kthread+0x12c/0x130
[ 1332.958056] [<ffff000008084740>] ret_from_fork+0x10/0x18
[ 1336.979359] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1336.984873] 0-...: (1 GPs behind) idle=e42/2/0 softirq=38269/40214 fqs=12330
[ 1336.992090] (detected by 1, t=52524 jiffies, g=433, c=432, q=3)
[ 1336.998104] Task dump for CPU 0:
[ 1337.001327] swapper/0       R running task        0     0      0 0x00000022
[ 1337.008381] Call trace:
[ 1337.010835] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1391.769614] INFO: rcu_preempt self-detected stall on CPU
[ 1391.771186] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1391.774956] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=20401
[ 1391.780617] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=20401
[ 1391.787829]
[ 1391.795047]
[ 1391.795050] (t=84031 jiffies g=14886 c=14885 q=2423)
[ 1391.795054] Task dump for CPU 0:
[ 1391.796633] (detected by 1, t=84031 jiffies, g=14886, c=14885, q=2423)
[ 1391.798203] swapper/0       R running task        0     0      0 0x00000022
[ 1391.820051] Call trace:
[ 1391.822504] [<ffff000008089668>] dump_backtrace+0x0/0x3c8
[ 1391.827909] [<ffff000008089a44>] show_stack+0x14/0x20
[ 1391.832965] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180
[ 1391.838627] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50
[ 1391.843947] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4
[ 1391.849783] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958
[ 1391.855795] [<ffff00000812a1f4>] update_process_times+0x2c/0x58
[ 1391.861721] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48
[ 1391.867992] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90
[ 1391.873569] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168
[ 1391.879580] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230
[ 1391.885334] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48
[ 1391.891519] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138
[ 1391.897789] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38
[ 1391.903540] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8
[ 1391.909378] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1391.914868] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)
[ 1391.921306] 3d80:                                   0000000000000000 ffff000008d07880
[ 1391.929143] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002
[ 1391.936979] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70
[ 1391.944815] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750
[ 1391.952651] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000
[ 1391.960488] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880
[ 1391.968324] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8
[ 1391.976160] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0
[ 1391.983996] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145
[ 1391.991832] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028
[ 1391.999666] 3ec0: ffff000008003ed0 ffff000008081184
[ 1392.004549] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1392.009431] [<ffff000008081184>] __do_softirq+0xa4/0x218
[ 1392.014750] [<ffff0000080d0340>] irq_exit+0xd0/0xf0
[ 1392.019631] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8
[ 1392.025468] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1392.030958] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)
[ 1392.037396] 3dc0:                                   0000000000000000 0000000000000000
[ 1392.045234] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00
[ 1392.053071] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90
[ 1392.060906] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000
[ 1392.068742] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000
[ 1392.076579] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000
[ 1392.084415] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000
[ 1392.092251] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10
[ 1392.100087] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145
[ 1392.107923] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc
[ 1392.115758] 3f00: ffff000008c83f10 ffff000008084f00
[ 1392.120640] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1392.125524] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18
[ 1392.130841] [<ffff000008106f50>] do_idle+0x120/0x1e0
[ 1392.135809] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28
[ 1392.141476] [<ffff00000886b314>] rest_init+0xcc/0xd8
[ 1392.146446] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8
[ 1392.151844] Task dump for CPU 0:
[ 1392.155066] swapper/0       R running task        0     0      0 0x00000022
[ 1392.162120] Call trace:
[ 1392.164567] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1400.008369] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1400.013884] 0-...: (1 GPs behind) idle=e42/2/0 softirq=38269/40214 fqs=16313
[ 1400.021102] (detected by 1, t=68281 jiffies, g=433, c=432, q=8)
[ 1400.027115] Task dump for CPU 0:
[ 1400.030339] swapper/0       R running task        0     0      0 0x00000022
[ 1400.037392] Call trace:
[ 1400.039846] [<ffff0000080853ac>] __switch_to+0x94/0xd8

root@imx8dx_ccu:~#
root@imx8dx_ccu:~#
root@imx8dx_ccu:~#
root@imx8dx_ccu:~#
root@imx8dx_ccu:~# [ 1454.790238] INFO: rcu_preempt self-detected stall on CPU
[ 1454.795572] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=24230
[ 1454.802791]   (t=99784 jiffies g=14886 c=14885 q=2787)
[ 1454.807936] Task dump for CPU 0:
[ 1454.811160] swapper/0       R running task        0     0      0 0x00000022
[ 1454.818213] Call trace:
[ 1454.820666] [<ffff000008089668>] dump_backtrace+0x0/0x3c8
[ 1454.826071] [<ffff000008089a44>] show_stack+0x14/0x20
[ 1454.831127] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180
[ 1454.836789] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50
[ 1454.842109] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4
[ 1454.847946] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958
[ 1454.853957] [<ffff00000812a1f4>] update_process_times+0x2c/0x58
[ 1454.859883] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48
[ 1454.866154] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90
[ 1454.871731] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168
[ 1454.877742] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230
[ 1454.883496] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48
[ 1454.889681] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138
[ 1454.895951] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38
[ 1454.901702] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8
[ 1454.907540] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1454.913030] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)
[ 1454.919477] 3d80:                                   0000000000000000 ffff000008d07880
[ 1454.927314] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002
[ 1454.935150] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70
[ 1454.942986] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750
[ 1454.950822] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000
[ 1454.958658] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880
[ 1454.966495] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8
[ 1454.974331] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0
[ 1454.982167] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145
[ 1454.990003] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028
[ 1454.997837] 3ec0: ffff000008003ed0 ffff000008081184
[ 1455.002720] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1455.007602] [<ffff000008081184>] __do_softirq+0xa4/0x218
[ 1455.012922] [<ffff0000080d0340>] irq_exit+0xd0/0xf0
[ 1455.017802] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8
[ 1455.023639] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1455.029129] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)
[ 1455.035567] 3dc0:                                   0000000000000000 0000000000000000
[ 1455.043405] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00
[ 1455.051241] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90
[ 1455.059077] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000
[ 1455.066913] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000
[ 1455.074749] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000
[ 1455.082585] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000
[ 1455.090422] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10
[ 1455.098258] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145
[ 1455.106094] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc
[ 1455.113928] 3f00: ffff000008c83f10 ffff000008084f00
[ 1455.118811] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1455.123695] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18
[ 1455.129011] [<ffff000008106f50>] do_idle+0x120/0x1e0
[ 1455.133980] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28
[ 1455.139647] [<ffff00000886b314>] rest_init+0xcc/0xd8
[ 1455.144616] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8
[ 1463.089388] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1463.094899] 0-...: (1 GPs behind) idle=e42/2/0 softirq=38269/40214 fqs=20134
[ 1463.102117] (detected by 1, t=84047 jiffies, g=433, c=432, q=11)
[ 1463.108217] Task dump for CPU 0:
[ 1463.111441] swapper/0       R running task        0     0      0 0x00000022
[ 1463.118494] Call trace:
[ 1463.120947] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1474.516999] xhci-cdns3: ERROR unknown event type 37
[ 1517.827364] INFO: rcu_preempt self-detected stall on CPU
[ 1517.829208] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1517.832697] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=28108
[ 1517.832702]
[ 1517.838362] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=28108
[ 1517.845576] (t=115541 jiffies g=14886 c=14885 q=2953)
[ 1517.847146]
[ 1517.854367] Task dump for CPU 0:
[ 1517.859503] (detected by 1, t=115541 jiffies, g=14886, c=14885, q=2953)
[ 1517.861073] swapper/0       R running task        0     0      0 0x00000022
[ 1517.877961] Call trace:
[ 1517.880415] [<ffff000008089668>] dump_backtrace+0x0/0x3c8
[ 1517.885819] [<ffff000008089a44>] show_stack+0x14/0x20
[ 1517.890874] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180
[ 1517.896537] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50
[ 1517.901856] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4
[ 1517.907693] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958
[ 1517.913706] [<ffff00000812a1f4>] update_process_times+0x2c/0x58
[ 1517.919632] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48
[ 1517.925902] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90
[ 1517.931479] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168
[ 1517.937490] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230
[ 1517.943243] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48
[ 1517.949429] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138
[ 1517.955699] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38
[ 1517.961450] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8
[ 1517.967288] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1517.972777] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)
[ 1517.979225] 3d80:                                   0000000000000000 ffff000008d07880
[ 1517.987061] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002
[ 1517.994898] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70
[ 1518.002734] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750
[ 1518.010570] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000
[ 1518.018406] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880
[ 1518.026243] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8
[ 1518.034078] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0
[ 1518.041915] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145
[ 1518.049751] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028
[ 1518.057585] 3ec0: ffff000008003ed0 ffff000008081184
[ 1518.062468] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1518.067350] [<ffff000008081184>] __do_softirq+0xa4/0x218
[ 1518.072671] [<ffff0000080d0340>] irq_exit+0xd0/0xf0
[ 1518.077550] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8
[ 1518.083387] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1518.088877] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)
[ 1518.095315] 3dc0:                                   0000000000000000 0000000000000000
[ 1518.103153] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00
[ 1518.110989] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90
[ 1518.118824] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000
[ 1518.126661] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000
[ 1518.134497] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000
[ 1518.142333] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000
[ 1518.150169] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10
[ 1518.158006] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145
[ 1518.165842] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc
[ 1518.173676] 3f00: ffff000008c83f10 ffff000008084f00
[ 1518.178559] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1518.183442] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18
[ 1518.188759] [<ffff000008106f50>] do_idle+0x120/0x1e0
[ 1518.193727] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28
[ 1518.199394] [<ffff00000886b314>] rest_init+0xcc/0xd8
[ 1518.204364] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8
[ 1518.209762] Task dump for CPU 0:
[ 1518.212985] swapper/0       R running task        0     0      0 0x00000022
[ 1518.220039] Call trace:
[ 1518.222485] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1526.134402] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1526.139913] 0-...: (1 GPs behind) idle=e42/2/0 softirq=38269/40214 fqs=24000
[ 1526.147130] (detected by 1, t=99809 jiffies, g=433, c=432, q=11)
[ 1526.153231] Task dump for CPU 0:
[ 1526.156454] swapper/0       R running task        0     0      0 0x00000022
[ 1526.163507] Call trace:
[ 1526.165961] [<ffff0000080853ac>] __switch_to+0x94/0xd8
root@imx8dx_ccu:~#
root@imx8dx_ccu:~# cd /logs/
root@imx8dx_ccu:/logs# [ 1580.869737] INFO: rcu_preempt self-detected stall on CPU
[ 1580.870221] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1580.875073] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=31648
[ 1580.880725] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=31648
[ 1580.887938]
[ 1580.895156]
[ 1580.895159] (t=131299 jiffies g=14886 c=14885 q=3089)
[ 1580.895163] Task dump for CPU 0:
[ 1580.896742] (detected by 1, t=131299 jiffies, g=14886, c=14885, q=3089)
[ 1580.898312] swapper/0       R running task        0     0      0 0x00000022
[ 1580.920333] Call trace:
[ 1580.922787] [<ffff000008089668>] dump_backtrace+0x0/0x3c8
[ 1580.928192] [<ffff000008089a44>] show_stack+0x14/0x20
[ 1580.933247] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180
[ 1580.938910] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50
[ 1580.944230] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4
[ 1580.950066] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958
[ 1580.956077] [<ffff00000812a1f4>] update_process_times+0x2c/0x58
[ 1580.962004] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48
[ 1580.968274] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90
[ 1580.973851] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168
[ 1580.979863] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230
[ 1580.985616] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48
[ 1580.991802] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138
[ 1580.998072] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38
[ 1581.003822] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8
[ 1581.009661] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1581.015150] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)
[ 1581.021597] 3d80:                                   0000000000000000 ffff000008d07880
[ 1581.029434] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002
[ 1581.037270] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70
[ 1581.045107] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750
[ 1581.052942] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000
[ 1581.060779] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880
[ 1581.068616] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8
[ 1581.076451] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0
[ 1581.084287] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145
[ 1581.092123] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028
[ 1581.099958] 3ec0: ffff000008003ed0 ffff000008081184
[ 1581.104841] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1581.109723] [<ffff000008081184>] __do_softirq+0xa4/0x218
[ 1581.115041] [<ffff0000080d0340>] irq_exit+0xd0/0xf0
[ 1581.119922] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8
[ 1581.125760] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1581.131250] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)
[ 1581.137688] 3dc0:                                   0000000000000000 0000000000000000
[ 1581.145525] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00
[ 1581.153362] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90
[ 1581.161197] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000
[ 1581.169034] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000
[ 1581.176870] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000
[ 1581.184706] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000
[ 1581.192542] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10
[ 1581.200378] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145
[ 1581.208214] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc
[ 1581.216049] 3f00: ffff000008c83f10 ffff000008084f00
[ 1581.220931] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1581.225815] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18
[ 1581.231132] [<ffff000008106f50>] do_idle+0x120/0x1e0
[ 1581.236100] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28
[ 1581.241767] [<ffff00000886b314>] rest_init+0xcc/0xd8
[ 1581.246737] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8
[ 1581.252135] Task dump for CPU 0:
[ 1581.255358] swapper/0       R running task        0     0      0 0x00000022
[ 1581.262411] Call trace:
[ 1581.264859] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1589.171414] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1589.176925] 0-...: (1 GPs behind) idle=e42/2/0 softirq=38269/40214 fqs=27597
[ 1589.184143] (detected by 1, t=115566 jiffies, g=433, c=432, q=17)
[ 1589.190330] Task dump for CPU 0:
[ 1589.193553] swapper/0       R running task        0     0      0 0x00000022
[ 1589.200607] Call trace:
[ 1589.203061] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1643.898488] INFO: rcu_preempt self-detected stall on CPU
[ 1643.899233] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1643.903820] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=35485
[ 1643.903825]
[ 1643.909485] 0-...: (1 GPs behind) idle=e42/2/0 softirq=40213/40214 fqs=35485
[ 1643.916700] (t=147054 jiffies g=14886 c=14885 q=3160)
[ 1643.918270]
[ 1643.925490] Task dump for CPU 0:
[ 1643.930626] (detected by 1, t=147054 jiffies, g=14886, c=14885, q=3160)
[ 1643.932197] swapper/0       R running task        0     0      0 0x00000022
[ 1643.949084] Call trace:
[ 1643.951538] [<ffff000008089668>] dump_backtrace+0x0/0x3c8
[ 1643.956942] [<ffff000008089a44>] show_stack+0x14/0x20
[ 1643.961998] [<ffff0000080f5cf4>] sched_show_task+0x14c/0x180
[ 1643.967660] [<ffff0000080f6c48>] dump_cpu_task+0x40/0x50
[ 1643.972980] [<ffff000008127288>] rcu_dump_cpu_stacks+0x94/0xd4
[ 1643.978817] [<ffff00000812604c>] rcu_check_callbacks+0x64c/0x958
[ 1643.984828] [<ffff00000812a1f4>] update_process_times+0x2c/0x58
[ 1643.990755] [<ffff000008139660>] tick_sched_handle.isra.4+0x30/0x48
[ 1643.997025] [<ffff0000081396b8>] tick_sched_timer+0x40/0x90
[ 1644.002602] [<ffff00000812ae70>] __hrtimer_run_queues+0xe8/0x168
[ 1644.008614] [<ffff00000812b0f0>] hrtimer_interrupt+0xa8/0x230
[ 1644.014367] [<ffff0000085f2808>] arch_timer_handler_phys+0x28/0x48
[ 1644.020552] [<ffff000008119950>] handle_percpu_devid_irq+0x80/0x138
[ 1644.026823] [<ffff0000081141bc>] generic_handle_irq+0x24/0x38
[ 1644.032573] [<ffff00000811482c>] __handle_domain_irq+0x5c/0xb8
[ 1644.038412] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1644.043901] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)
[ 1644.050348] 3d80:                                   0000000000000000 ffff000008d07880
[ 1644.058185] 3da0: 0000000000000000 000000010002e1a9 000000000000002e 0000000000000002
[ 1644.066021] 3dc0: 0000000000000000 0000000040000000 ffff8000121fc248 ffff000008003c70
[ 1644.073857] 3de0: 00000000960c0650 0000000000000001 0000000000000401 ffff000009205750
[ 1644.081693] 3e00: 0000000000000000 0000000000000000 ffff00000813e220 0000000000000000
[ 1644.089530] 3e20: 0000000000000000 ffff000008b4f000 000000000000001a ffff000008d07880
[ 1644.097366] 3e40: ffff000008b4f018 ffff800012405780 ffff000008d06000 000000010002e1a8
[ 1644.105202] 3e60: ffff000008004000 ffff000008890000 0000000000000001 ffff000008003ed0
[ 1644.113038] 3e80: ffff0000080d0340 ffff000008003ed0 ffff000008081184 0000000040000145
[ 1644.120874] 3ea0: ffff8000125ab0a4 ffff8000125ab000 0000ffffffffffff ffff8000125ab028
[ 1644.128708] 3ec0: ffff000008003ed0 ffff000008081184
[ 1644.133591] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1644.138473] [<ffff000008081184>] __do_softirq+0xa4/0x218
[ 1644.143794] [<ffff0000080d0340>] irq_exit+0xd0/0xf0
[ 1644.148673] [<ffff000008114830>] __handle_domain_irq+0x60/0xb8
[ 1644.154511] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174
[ 1644.160000] Exception stack(0xffff000008c83dd0 to 0xffff000008c83f10)
[ 1644.166439] 3dc0:                                   0000000000000000 0000000000000000
[ 1644.174276] 3de0: 0000000000000001 0000000000000000 ffff000008b53108 ffff000008c83f00
[ 1644.182113] 3e00: 000080003740d000 ffff80003ff65b60 ffff000008c952c0 ffff000008c83e90
[ 1644.189948] 3e20: 00000000000008e0 0000000000000000 0000000000000001 0000000000000000
[ 1644.197784] 3e40: 00000000f6045434 0000000000000000 ffff00000813e220 0000000000000000
[ 1644.205621] 3e60: 0000000000000000 ffff000008b4f018 ffff000008c8a000 ffff000008c8a000
[ 1644.213457] 3e80: ffff000008b57a30 ffff000008c8aa60 0000000000000000 0000000000000000
[ 1644.221293] 3ea0: ffff000008c94980 0000000000000400 0000000080cf0018 ffff000008c83f10
[ 1644.229129] 3ec0: ffff000008084efc ffff000008c83f10 ffff000008084f00 0000000000000145
[ 1644.236965] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff0000081397dc
[ 1644.244800] 3f00: ffff000008c83f10 ffff000008084f00
[ 1644.249682] [<ffff000008082a30>] el1_irq+0xb0/0x124
[ 1644.254565] [<ffff000008084f00>] arch_cpu_idle+0x10/0x18
[ 1644.259882] [<ffff000008106f50>] do_idle+0x120/0x1e0
[ 1644.264851] [<ffff0000081071a8>] cpu_startup_entry+0x20/0x28
[ 1644.270518] [<ffff00000886b314>] rest_init+0xcc/0xd8
[ 1644.275488] [<ffff000008af0c1c>] start_kernel+0x3d4/0x3e8
[ 1644.280886] Task dump for CPU 0:
[ 1644.284109] swapper/0       R running task        0     0      0 0x00000022
[ 1644.291162] Call trace:
[ 1644.293609] [<ffff0000080853ac>] __switch_to+0x94/0xd8
[ 1652.200426] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1652.205937] 0-...: (1 GPs behind) idle=e42/2/0 softirq=38269/40214 fqs=31413
[ 1652.213154] (detected by 1, t=131318 jiffies, g=433, c=432, q=17)
[ 1652.219341] Task dump for CPU 0:
[ 1652.222565] swapper/0       R running task        0     0      0 0x00000022
[ 1652.229618] Call trace:
[ 1652.232072] [<ffff0000080853ac>] __switch_to+0x94/0xd8

Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37

xinyu_chen — Tue, 11 Feb 2020 01:47:12 GMT

From the log, seems the USB host runtime pm blocked for long time, please try to disable the runtime pm to try:

diff --git a/drivers/usb/cdns3/host.c b/drivers/usb/cdns3/host.c
index f6633de..aac6158 100644
--- a/drivers/usb/cdns3/host.c
+++ b/drivers/usb/cdns3/host.c
@@ -155,6 +155,7 @@ static int cdns3_host_start(struct cdns3 *cdns)
pm_runtime_set_active(dev);
pm_runtime_no_callbacks(dev);
pm_runtime_enable(dev);
+ pm_runtime_forbid(dev);
host->hcd = __usb_create_hcd(&xhci_cdns3_hc_driver, sysdev, dev,
dev_name(dev), NULL);

Also, please check the top in background in loop before issue happen, see the irq/sirq loading.

Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37

jonas_karlsson — Thu, 13 Feb 2020 07:25:53 GMT

Hi,

We are also experiencing the same problem on iMX8QXP using the 4.14.98 (GA BSP). We have also tested the 4.19.35 BSP which does not solve the problem.

In our case we have a modem connected to USB.

We have also applied the patch [v2,1/1] usb: host: xhci: update event ring dequeue pointer on purpose - Patchwork

which reduces the amount of “xhci-cdns3: ERROR unknown event type 37”, but it does not solve the root cause. When we have applied the patch and the “xhci-cdns3: ERROR unknown event type 37” situation occurs we see that the cpu load increases significantly. According to 'top' the increased cpu load is in irq context. However, we do not see a significant increase in the amount of interrupts when looking in /proc/interrupts. When we unbind the usb port the cpu load goes back to normal.

Occasionally this issue has also led to kernel panics. All kernel panics that have been analyzed are preceded by a time out in the SCU communication. We see "Timeout for IPC response!" in the logs.

I have been trying to capture traces by enabling tracing in the kernel and then enabling the following:

echo 1 > /sys/kernel/debug/tracing/events/xhci-hcd/enable
echo 1 > /sys/kernel/debug/tracing/events/cdns3/enable
echo "(irq == 30) || (irq == 27)" > /sys/kernel/debug/tracing/events/irq/irq_handler_entry/filter
echo "(irq == 30) || (irq == 27)" > /sys/kernel/debug/tracing/events/irq/irq_handler_exit/filter
echo 1 > /sys/kernel/debug/tracing/events/irq/enable

echo 1 > /sys/kernel/debug/tracing/tracing_on

Unfortunately I have not been able to read out the trace afterwards since I get a kernel panic every time the “xhci-cdns3: ERROR unknown event type 37” occurs when traces are enabled.

I will continue trying to capture traces to see what happens.

BR,

Jonas

Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37

jeffin_joys — Sat, 15 Feb 2020 10:01:08 GMT

Dear NXP Team,

Dear Xinyu,

Thanks for your reply.

We had tried applying patch (disable runtime pm of usb) as suggested from your side. Additionally, below are the list of patches that's applied

1. Downgraded USB driver from GA BSP (L 4.14.98 GA) to Beta BSP (L4.14.62)

2. [v2,1/1] usb: host: xhci: update event ring dequeue pointer on purpose - Patchwork

3. usb: xhci: Support running urb giveback in tasklet context - Patchwork

4. Reverted patch present in RCU (rcu: Make need_resched() respond to urgent RCU-QS needs)

commit 077506972ba23772b752e08b1ab7052cf5f04511

Observations:

1. Issue reproducing rate is decreased, but still exists.

Reason:

After fresh flashing, device was booted (Boot time ~40 seconds) and login prompt came. As next step, we had provided device a soft reboot from terminal using reboot command. Immediately upon coming of login prompt in next reboot, we had received RCU crash.

2. If the RCU crash is not reproduced, the device is capable to run for overnight. Max time we have observed is 12 hours. Longer hours still we need to test and see the behavior.

Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37

karina_valencia — Mon, 17 Feb 2020 15:04:43 GMT

XinyuChen‌ please continue with the follow up.

Re: rcu_preempt self-detected stall on CPU and usb xhci event error 37

danwei_luo — Tue, 18 Feb 2020 06:11:20 GMT

Status update:

We checked the difference between L4.14.62 and L4.14.98 on kernel/rcu/, only one commit is added on L4.14.98:

commit 077506972ba23772b752e08b1ab7052cf5f04511

Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Date: Mon Jul 9 13:47:30 2018 -0700

rcu: Make need_resched() respond to urgent RCU-QS needs

After revert this commit and apply the USB patch, the RCU stall issue is fixed on L4.14.98.

Best regards,

Danwei