INFO: rcu_preempt self-detected stall on CPU on imx8 dual core

[yellapu_anishkh]

Hi,

We have LTE module on USB on imx8, when ppp0 is up and when i run the hardware for few hours, we are observing below rcu crash. if i disable LTE, then we are not getting any crash. Please find the crash logs below.

Tried different RCU kernel configs, but no luck. 

INFO: rcu_preempt self-detected stall on CPU

[ 1412.997440]  0-...: (1 GPs behind) idle=c7a/2/0 softirq=30757/37129 fqs=1452

[ 1413.004577]  (detected by 1, t=5256 jiffies, g=338, c=337, q=4)

[ 1413.010504] Task dump for CPU 0:

[ 1413.013736] swapper/0       R  running task        0     0      0 0x00000022

[ 1413.020798] Call trace:

[ 1413.023263] [<ffff0000080853b4>] __switch_to+0x94/0xd8

[ 1432.528731] INFO: rcu_preempt self-detected stall on CPU

[ 1432.534070]  0-...: (1 GPs behind) idle=c7a/2/0 softirq=37128/37129 fqs=20824

[ 1432.541296]   (t=84026 jiffies g=9495 c=9494 q=1975)

[ 1432.546267] Task dump for CPU 0:

[ 1432.549500] swapper/0       R  running task        0     0      0 0x00000022

[ 1432.556561] Call trace:

[ 1432.559018] [<ffff0000080892f0>] dump_backtrace+0x0/0x3c8

[ 1432.564421] [<ffff0000080896cc>] show_stack+0x14/0x20

[ 1432.569476] [<ffff0000080f5164>] sched_show_task+0x14c/0x180

[ 1432.575138] [<ffff0000080f60b8>] dump_cpu_task+0x40/0x50

[ 1432.580459] [<ffff000008126788>] rcu_dump_cpu_stacks+0x94/0xd4

[ 1432.586295] [<ffff00000812554c>] rcu_check_callbacks+0x64c/0x958

[ 1432.592307] [<ffff0000081296f4>] update_process_times+0x2c/0x58

[ 1432.598234] [<ffff000008138b98>] tick_sched_handle.isra.4+0x30/0x48

[ 1432.604504] [<ffff000008138bf0>] tick_sched_timer+0x40/0x90

[ 1432.610081] [<ffff00000812a370>] __hrtimer_run_queues+0xe8/0x168

[ 1432.616092] [<ffff00000812a5f0>] hrtimer_interrupt+0xa8/0x230

[ 1432.621847] [<ffff0000085c70e0>] arch_timer_handler_phys+0x28/0x48

[ 1432.628031] [<ffff000008118e50>] handle_percpu_devid_irq+0x80/0x138

[ 1432.634301] [<ffff000008113684>] generic_handle_irq+0x24/0x38

[ 1432.640052] [<ffff000008113cf4>] __handle_domain_irq+0x5c/0xb8

[ 1432.645890] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174

[ 1432.651379] Exception stack(0xffff000008003d90 to 0xffff000008003ed0)

[ 1432.657826] 3d80:                                   0000000000000000 ffff000008cc5880

[ 1432.665663] 3da0: 0000000000000000 0000000100030999 0000000000000007 0000000000000002

[ 1432.673500] 3dc0: 0000000000000001 0000000040000000 ffff800012358248 ffff000008003c70

[ 1432.681335] 3de0: 00000000960b19d0 0000000000000001 0000000000000400 ffff000008f3dad0

[ 1432.689172] 3e00: 0000000000000000 0000000000000000 ffff00000813d758 0000000000000000

[ 1432.697008] 3e20: 0000000000000000 ffff000008b1e000 000000000000001a ffff000008cc5880

[ 1432.704844] 3e40: ffff000008b1e018 ffff800012405780 ffff000008cc4000 0000000100030998

[ 1432.712680] 3e60: ffff000008004000 ffff000008870000 0000000000000001 ffff000008003ed0

[ 1432.720517] 3e80: ffff0000080cf8e8 ffff000008003ed0 ffff000008081184 0000000040000145

[ 1432.728353] 3ea0: ffff80001259f8a4 ffff80001259f800 0000ffffffffffff ffff80001259f828

[ 1432.736186] 3ec0: ffff000008003ed0 ffff000008081184

[ 1432.741070] [<ffff000008082a30>] el1_irq+0xb0/0x124

[ 1432.745952] [<ffff000008081184>] __do_softirq+0xa4/0x218

[ 1432.751271] [<ffff0000080cf8e8>] irq_exit+0xd0/0xf0

[ 1432.756152] [<ffff000008113cf8>] __handle_domain_irq+0x60/0xb8

[ 1432.761989] [<ffff000008080fe0>] gic_handle_irq+0x78/0x174

[ 1432.767478] Exception stack(0xffff000008c43dd0 to 0xffff000008c43f10)

[ 1432.773926] 3dc0:                                   0000000000000000 0000000000000000

[ 1432.781763] 3de0: 0000000000000001 0000000000000000 ffff000008b22108 ffff000008c43f00

[ 1432.789600] 3e00: 000080003743e000 ffff800012425600 ffff000008c55240 ffff000008c43e90

[ 1432.797435] 3e20: 00000000000008e0 0000000000000000 0000000000000000 0000000000000000

[ 1432.805272] 3e40: 0000000000000000 0000000000000000 ffff00000813d758 0000000000000000

[ 1432.813108] 3e60: 0000000000000000 ffff000008b1e018 ffff000008c4a000 ffff000008c4a000

[ 1432.820944] 3e80: ffff000008b26a30 ffff000008c4a9e0 0000000000000000 0000000000000000

[ 1432.828780] 3ea0: ffff000008c54900 0000000000000400 0000000080cc0018 ffff000008c43f10

[ 1432.836616] 3ec0: ffff000008084f04 ffff000008c43f10 ffff000008084f08 0000000000000145

[ 1432.844452] 3ee0: 0000000000000000 00000000bfe9eb84 ffffffffffffffff ffff000008138d14

[ 1432.852286] 3f00: ffff000008c43f10 ffff000008084f08

[ 1432.857169] [<ffff000008082a30>] el1_irq+0xb0/0x124

[ 1432.862053] [<ffff000008084f08>] arch_cpu_idle+0x10/0x18

[ 1432.867370] [<ffff0000081063e0>] do_idle+0x120/0x1e0

[ 1432.872338] [<ffff00000810663c>] cpu_startup_entry+0x24/0x28

[ 1432.878006] [<ffff000008840fe4>] rest_init+0xcc/0xd8

[ 1432.882976] [<ffff000008ac0b88>] start_kernel+0x37c/0x390

[ 1450.718977] INFO: task jbd2/mmcblk0p12:1993 blocked for more than 120 seconds.

[ 1450.728787]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1

[ 1450.737430] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[ 1450.745701] jbd2/mmcblk0p12 D    0  1993      2 0x00000000

[ 1450.751414] Call trace:

[ 1450.753912] [<ffff0000080853b4>] __switch_to+0x94/0xd8

[ 1450.759336] [<ffff00000884234c>] __schedule+0x18c/0x5c0

[ 1450.764766] [<ffff0000088427b8>] schedule+0x38/0xa0

[ 1450.769892] [<ffff0000080f4df8>] io_schedule+0x18/0x38

[ 1450.775246] [<ffff000008843154>] bit_wait_io+0x14/0x68

[ 1450.780674] [<ffff000008842de8>] __wait_on_bit+0x70/0xe8

[ 1450.786250] [<ffff000008842ecc>] out_of_line_wait_on_bit+0x6c/0x78

[ 1450.792677] [<ffff00000823e884>] __wait_on_buffer+0x2c/0x38

[ 1450.798644] [<ffff0000082dd768>] jbd2_journal_commit_transaction+0x10f8/0x1358

[ 1450.806154] [<ffff0000082e0c9c>] kjournald2+0xbc/0x230

[ 1450.811507] [<ffff0000080e99f4>] kthread+0x12c/0x130

[ 1450.816709] [<ffff000008084748>] ret_from_fork+0x10/0x18

[ 1450.822299] INFO: task kworker/0:2:2409 blocked for more than 120 seconds.

[ 1450.831849]       Tainted: P           O    4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1

[ 1450.839663] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[ 1450.847878] kworker/0:2     D    0  2409      2 0x00000020

[ 1450.853604] Workqueue: pm pm_runtime_work

[ 1450.857819] Call trace:

[ 1450.860621] [<ffff0000080853b4>] __switch_to+0x94/0xd8

[ 1450.865983] [<ffff00000884234c>] __schedule+0x18c/0x5c0

[ 1450.871526] [<ffff0000088427b8>] schedule+0x38/0xa0

[ 1450.876613] [<ffff000008845f34>] schedule_timeout+0x74/0x320

[ 1450.882522] [<ffff000008128cb8>] msleep+0x28/0x38

[ 1450.887478] [<ffff000008510484>] xhci_bus_resume+0x40c/0x448

[ 1450.893380] [<ffff0000084d9300>] hcd_bus_resume+0x50/0x188

[ 1450.899541] [<ffff0000084d9568>] hcd_bus_suspend+0x130/0x138

[ 1450.905455] [<ffff0000084e932c>] generic_suspend+0x4c/0x58

[ 1450.911185] [<ffff0000084de33c>] usb_suspend_both+0x18c/0x208

[ 1450.917141] [<ffff0000084df374>] usb_runtime_suspend+0x2c/0x70

[ 1450.923319] [<ffff000008497ce8>] __rpm_callback+0xe0/0x268

[ 1450.929058] [<ffff000008497e90>] rpm_callback+0x20/0x80

[ 1450.935222] [<ffff00000849814c>] rpm_suspend+0xf4/0x4b8

[ 1450.940513] [<ffff000008499104>] __pm_runtime_suspend+0x5c/0xa0

[ 1450.946728] [<ffff0000084df3fc>] usbs_runtime_idle+0x24/0x30

[ 1450.952550] [<ffff000008497ce8>] __rpm_callback+0xe0/0x268

[ 1450.958249] [<ffff000008497fdc>] rpm_idle+0xec/0x168

[ 1450.963424] [<ffff000008498a60>] pm_runtime_work+0xa0/0xb8

[ 1450.969205] [<ffff0000080e3a10>] process_one_work+0x188/0x300

[ 1450.975165] [<ffff0000080e3bd4>] worker_thread+0x4c/0x420

[ 1450.980776] [<ffff0000080e99f4>] kthread+0x12c/0x130

[ 1450.986124] [<ffff000008084748>] ret_from_fork+0x10/0x18

[igorpadykov]

Hi Anish

one can look at suggestions on such error provided on

https://lkml.org/lkml/2019/1/2/148 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------