Kernel crash after enabling all 4 cores of t1042

[chandrashekhar]

Hi Team,

After enabling all the 4 cores of t1042, I am seeing kernel crash at below point.

----------------------------------------------------------------------------------------------

Bman: BPID allocator includes range 32:32

Qman: FQID allocator includes range 256:512

Qman: FQID allocator includes range 32768:32768

Qman: CGRID allocator includes range 0:256

Qman: pool channel allocator includes range 1025:15

Switched to clocksource timebase

INFO: rcu_sched self-detected stall on CPU { 3}  (t=17160 jiffies g=4294967058 c=4294967057 q=32)

CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.12.19-rt30-gc29fe1a #23

Call Trace:

[e30bfd00] [c0006f74] show_stack+0x44/0x150 (unreliable)

[e30bfd40] [c069bebc] dump_stack+0x78/0xa0

[e30bfd50] [c00a0a60] rcu_check_callbacks+0x320/0x6e0

[e30bfdb0] [c003dd90] update_process_times+0x40/0x70

[e30bfdd0] [c0088f78] tick_sched_timer+0x68/0xb0

[e30bfe00] [c00561c4] __run_hrtimer.isra.27+0x54/0xe0

[e30bfe20] [c00571e4] hrtimer_interrupt+0x1b4/0x4c0

[e30bfea0] [c0009e98] timer_interrupt+0x1b8/0x1f0

[e30bfed0] [c00103f0] ret_from_except+0x0/0x18

--- Exception: 901 at arch_cpu_idle+0x24/0x70

    LR = arch_cpu_idle+0x24/0x70

[e30bff90] [c009fc28] rcu_idle_enter+0xb8/0x100 (unreliable)

[e30bffa0] [c0076604] cpu_startup_entry+0xd4/0x180

[e30bffd0] [c0011d04] start_secondary+0x224/0x240

[e30bfff0] [c0001d58] __secondary_start+0x30/0x84

--------------------------------------------------------------------------------------------------

Can anyone provide any inputs to debug this issue?

Regards,

[yipingwang]

Hello Chandra Shekhar,

Would your please provide u-boot log(including RCW information) and Kernel configuration file(.config)?

Thanks,

Yiping

[chandrashekhar]

Hi Yiping,

I am attaching u-boot log, Linux boot log and config file for your reference.

Thanks,

[yipingwang]

Hello Chandra Shekhar,

I built a Kernel image in SDK 1.7 with your kernel configuration file, and verified it on the demo board, but didn't reproduce your problem.

According your kernel log information, it looks that 4 cores has already been brought up successfully.

e500 family performance monitor hardware support registered

Brought up 4 CPUs

devtmpfs: initialized

I suspect whether this problem is related with Kernel source which you are using, I attached my Kernel image file to you, please try whether your problem remains with the attached uImage file.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[chandrashekhar]

Hi Yiping,

I tried the provided kernel image but unfortunately, this also crashed. I would like to add that we are using custom board designed with t1042 and u-boot is also tweaked from what originally provided for t1042rdb board. With this image I got more logs while crash, which I am attaching here. Does it provide any clue what is going wrong?

Thanks,

Chandra Shekhar

[yipingwang]

Hello Chandra Shekhar,

I discussed with my colleague, please consider about the following points.

* Please check whether timebase of core 1-3 is enabled in the register RCPM_PCTBENR in u-boot. You could refer to T1040 reference manual for detailed information for this register.

*Please check clock setting in RCW.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[chandrashekhar]

Hi Yiping,

I am seeing with option "nohz=off" kernel is booting most of the time. Sometimes it hangs just before Linux prompt. I am attaching the logs while it hangs. I will check the Registers mentioned by you and update.

Regards,

[chandrashekhar]

Hi Yiping,

Also register "RCPM_PCTBENR =0x0000000f" . So it looks ok.

[yipingwang]

Hello Chandra Shekhar,

On your target board, you use u-boot and Linux Kernel from different version Linux SDK, this is what we don't recommended, because u-boot configuration would affect some features in Linux Kernel. Just now I installed Linux SDK QorIQ-SDK-T1040-BSP0.2-20140310, and found the default Linux configuration was different, I attached this default Kernel configuration.

Timers subsystem

#

CONFIG_TICK_ONESHOT=y

CONFIG_NO_HZ=y

CONFIG_HIGH_RES_TIMERS=y

#

# RCU Subsystem

#

CONFIG_TREE_RCU=y

# CONFIG_PREEMPT_RCU is not set

CONFIG_RCU_FANOUT=32

CONFIG_RCU_FANOUT_LEAF=16

# CONFIG_RCU_FANOUT_EXACT is not set

# CONFIG_RCU_FAST_NO_HZ is not set

CONFIG_TREE_RCU_TRACE=y

So I suggest you perform the porting based on the attache Kernel configuration file, if you want to use u-boot from Linux SDK QorIQ-SDK-T1040-BSP0.2-20140310.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[chandrashekhar]

Hi Yiping,

In fact I tried loading Linux kernel image build with  SDK QorIQ-SDK-T1040-BSP0.2-20140310 i.e.(kenel 3.8) but got the same result.

Regards,

Chandra Shekhar

[yipingwang]

Hello Chandra Shekhar,

I verified QorIQ-SDK-T1040-BSP0.2-20140310 u-boot to boot SDK 1.7 uImage without problem.

Please check the following.

Please check your dts file whether cpus properties definition in your dts file is similar as arch/powerpc/boot/dts/fsl/t104xsi-pre.dtsi.

Please u-boot whether you modified timebase related configuration.

Please check clock setting in RCW, you could use QCVS tool to assist you.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[yipingwang]

Hello Chandra Shekhar,

I am considering whether this problem is caused by hardware.

Please check whether Kernel could boot successfully  when use "nohz=off" Kernel boot parameter to disable dyntick-idle mode.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------