cant reboot successfully when When some packets sent to P4080ds

[shanfeifei]

Hi.

I ran lpm-ipfwd(configure fm2-gb3 and fm1-gb2 as share mac) on P4080ds board,When some packets sent to P4080ds,I typed "reboot".But it didnt reboot successfully.

If there were no packets sent to P4080ds,I reboot the board successfully.

So how to solve the problem?

The log is :

root@p4080ds:~# reboot

Broadcast message from root@p4080ds (ttyS0) (Sat Jul 30 15:40:42 2016):

The system is going down for reboot NOW!

INIT: Sending processes the TERM signal

INIT: Stopping Dropbear SSH server: stopped /usr/sbin/dropbear (pid 2239)

dropbear.

Stopping system message bus: dbus.

Stopping network benchmark server: netserverstopped /usr/sbin/netserver (pid 2266)

.

stopping mountd: done

stopping nfsd: nfsd: last server has exited, flushing export cache

.socks: : Network is unreachable

socks: : Network is unreachable

done

Stopping system log daemon...0

Stopping kernel log daemon...0

Stopping tcf-agent: socks: : Network is unreachable

OK

Stopping internet superserver: xinetd.

stopping statd: done

Shutting down MySQL..socks: : Network is unreachable

.[  OK  ]

Stopping crond: OK

Stopping rpcbind daemon...

done.

Deconfiguring network interfaces... done.

Sending all processes the TERM signal...

lpm-ipfwd:handleFatalSignal Number is 15

Received TERM or STOP signal...  shutting down...

Receiver SIGINT, SIGHUP or SIGTERM, Shutting down

select: Interrupted system call

Receiver SIGINT, SIGHUP or SIGTERM, Shutting down

Receiver SIGINT, SIGHUP or SIGTERM, Shutting down

ha_spawn_task:/root/HA/EXE/ha_ipmi_script.sh

******Killimg CMGR childs*****

sh: line 0: kill: (2797) - No such process

plat_type is 6

This palt type is 6

Do Not Support lighting LED

ifconfig eth0 down finish

ifconfig fm1-gb2 down finish

close_fsl_usdpaa_fd close OK

worker_free(worker not first) finish

Sending all processes the KILL signal...

Unmounting remote filesystems...

Deactivating swap...

Unmounting local filesystems...

Rebooting... INFO: rcu_sched self-detected stall on CPU { 0}  (t=5251 jiffies g=1215 c=1214 q=8721)

CPU: 0 PID: 2511 Comm: lpm_ipfwd_app Tainted: G           O 3.12.19-rt30-QorIQ-SDK-V1.6+gc29fe1a #3

INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=5252 jiffies, g=1215, c=1214, q=8721)

Task dump for CPU 0:

lpm_ipfwd_app   R running      0  2511      1 0x00002006

Call Trace:

[c6231a90] [c009d9f0] rcu_check_callbacks+0x320/0x6e0 (unreliable)

[c6231af0] [c003b190] update_process_times+0x40/0x70

[c6231b10] [c00868d8] tick_sched_timer+0x68/0xb0

[c6231b40] [c0053594] __run_hrtimer.isra.28+0x54/0xe0

[c6231b60] [c0054244] hrtimer_interrupt+0x164/0x430

[c6231be0] [c0009e48] timer_interrupt+0x1b8/0x1f0

[c6231c10] [c0010274] ret_from_except+0x0/0x18

--- Exception: 901 at usdpaa_release+0xbc0/0x1010

    LR = usdpaa_release+0x4bc/0x1010

[c6231d70] [c00f5bdc] __fput+0x9c/0x210

[c6231d90] [c004bbf4] task_work_run+0xc4/0x110

[c6231db0] [c002eea0] do_exit+0x280/0x8d0

[c6231e00] [c0030398] do_group_exit+0x48/0xb0

[c6231e20] [c003eaf8] get_signal_to_deliver+0x168/0x4e0

[c6231e70] [c0007ab0] do_signal+0x40/0x210

[c6231f30] [c0007d98] do_notify_resume+0x68/0x80

[c6231f40] [c0010840] do_user_signal+0x74/0xc4

--- Exception: c00 at 0xfef1638

    LR = 0x100052d4

Call Trace:

[c6231a40] [c0006f50] show_stack+0x40/0x150 (unreliable)

[c6231a80] [c067014c] dump_stack+0x78/0xa0

[c6231a90] [c009d9f0] rcu_check_callbacks+0x320/0x6e0

[c6231af0] [c003b190] update_process_times+0x40/0x70

[c6231b10] [c00868d8] tick_sched_timer+0x68/0xb0

[c6231b40] [c0053594] __run_hrtimer.isra.28+0x54/0xe0

[c6231b60] [c0054244] hrtimer_interrupt+0x164/0x430

[c6231be0] [c0009e48] timer_interrupt+0x1b8/0x1f0

[c6231c10] [c0010274] ret_from_except+0x0/0x18

--- Exception: 901 at usdpaa_release+0xbc0/0x1010

    LR = usdpaa_release+0x4bc/0x1010

[c6231d70] [c00f5bdc] __fput+0x9c/0x210

[c6231d90] [c004bbf4] task_work_run+0xc4/0x110

[c6231db0] [c002eea0] do_exit+0x280/0x8d0

[c6231e00] [c0030398] do_group_exit+0x48/0xb0

[c6231e20] [c003eaf8] get_signal_to_deliver+0x168/0x4e0

[c6231e70] [c0007ab0] do_signal+0x40/0x210

[c6231f30] [c0007d98] do_notify_resume+0x68/0x80

[c6231f40] [c0010840] do_user_signal+0x74/0xc4

--- Exception: c00 at 0xfef1638

    LR = 0x100052d4

INFO: rcu_sched self-detected stall on CPU { 0}  (t=21006 jiffies g=1215 c=1214 q=8724)

CPU: 0 PID: 2511 Comm: lpm_ipfwd_app Tainted: G           O 3.12.19-rt30-QorIQ-SDK-V1.6+gc29fe1a #3

INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=21007 jiffies, g=1215, c=1214, q=8724)

Task dump for CPU 0:

lpm_ipfwd_app   R running      0  2511      1 0x00002006

Call Trace:

[c6231a90] [c009d9f0] rcu_check_callbacks+0x320/0x6e0 (unreliable)

[c6231af0] [c003b190] update_process_times+0x40/0x70

[c6231b10] [c00868d8] tick_sched_timer+0x68/0xb0

[c6231b40] [c0053594] __run_hrtimer.isra.28+0x54/0xe0

[c6231b60] [c0054244] hrtimer_interrupt+0x164/0x430

[c6231be0] [c0009e48] timer_interrupt+0x1b8/0x1f0

[c6231c10] [c0010274] ret_from_except+0x0/0x18

--- Exception: 901 at usdpaa_release+0xbc0/0x1010

    LR = usdpaa_release+0x4bc/0x1010

[c6231d70] [c00f5bdc] __fput+0x9c/0x210

[c6231d90] [c004bbf4] task_work_run+0xc4/0x110

[c6231db0] [c002eea0] do_exit+0x280/0x8d0

[c6231e00] [c0030398] do_group_exit+0x48/0xb0

[c6231e20] [c003eaf8] get_signal_to_deliver+0x168/0x4e0

[c6231e70] [c0007ab0] do_signal+0x40/0x210

[c6231f30] [c0007d98] do_notify_resume+0x68/0x80

[c6231f40] [c0010840] do_user_signal+0x74/0xc4

--- Exception: c00 at 0xfef1638

    LR = 0x100052d4

Call Trace:

[c6231a40] [c0006f50] show_stack+0x40/0x150 (unreliable)

[c6231a80] [c067014c] dump_stack+0x78/0xa0

[c6231a90] [c009d9f0] rcu_check_callbacks+0x320/0x6e0

[c6231af0] [c003b190] update_process_times+0x40/0x70

[c6231b10] [c00868d8] tick_sched_timer+0x68/0xb0

[c6231b40] [c0053594] __run_hrtimer.isra.28+0x54/0xe0

[c6231b60] [c0054244] hrtimer_interrupt+0x164/0x430

[c6231be0] [c0009e48] timer_interrupt+0x1b8/0x1f0

[c6231c10] [c0010274] ret_from_except+0x0/0x18

--- Exception: 901 at usdpaa_release+0xbc0/0x1010

    LR = usdpaa_release+0x4bc/0x1010

[c6231d70] [c00f5bdc] __fput+0x9c/0x210

[c6231d90] [c004bbf4] task_work_run+0xc4/0x110

[c6231db0] [c002eea0] do_exit+0x280/0x8d0

[c6231e00] [c0030398] do_group_exit+0x48/0xb0

[c6231e20] [c003eaf8] get_signal_to_deliver+0x168/0x4e0

[c6231e70] [c0007ab0] do_signal+0x40/0x210

[c6231f30] [c0007d98] do_notify_resume+0x68/0x80

[c6231f40] [c0010840] do_user_signal+0x74/0xc4

--- Exception: c00 at 0xfef1638

    LR = 0x100052d4

INFO: rcu_sched self-detected stall on CPU { 0}  (t=36761 jiffies g=1215 c=1214 q=8725)

CPU: 0 PID: 2511 Comm: lpm_ipfwd_app Tainted: G           O 3.12.19-rt30-QorIQ-SDK-V1.6+gc29fe1a #3

INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=36762 jiffies, g=1215, c=1214, q=8725)

Task dump for CPU 0:

lpm_ipfwd_app   R running      0  2511      1 0x00002006

Call Trace:

[c6231a90] [c009d9f0] rcu_check_callbacks+0x320/0x6e0 (unreliable)

[c6231af0] [c003b190] update_process_times+0x40/0x70

[c6231b10] [c00868d8] tick_sched_timer+0x68/0xb0

[c6231b40] [c0053594] __run_hrtimer.isra.28+0x54/0xe0

[c6231b60] [c0054244] hrtimer_interrupt+0x164/0x430

[c6231be0] [c0009e48] timer_interrupt+0x1b8/0x1f0

[c6231c10] [c0010274] ret_from_except+0x0/0x18

--- Exception: 901 at usdpaa_release+0xbc0/0x1010

    LR = usdpaa_release+0x4bc/0x1010

[c6231d70] [c00f5bdc] __fput+0x9c/0x210

[c6231d90] [c004bbf4] task_work_run+0xc4/0x110

[c6231db0] [c002eea0] do_exit+0x280/0x8d0

[c6231e00] [c0030398] do_group_exit+0x48/0xb0

[c6231e20] [c003eaf8] get_signal_to_deliver+0x168/0x4e0

[c6231e70] [c0007ab0] do_signal+0x40/0x210

[c6231f30] [c0007d98] do_notify_resume+0x68/0x80

[c6231f40] [c0010840] do_user_signal+0x74/0xc4

--- Exception: c00 at 0xfef1638

    LR = 0x100052d4

[shanfeifei]

Today this problem happened again,When too many packets sent to P4080ds,I typed "reboot",the system failed to reboot system,the log as follows:

Cannot recover FQ 0x13b, it is scheduled on channel 0x64stopping mountd: done
stopping nfsd: done
Stopping system log daemon...0
Stopping kernel log daemon...0
Stopping tcf-agent:
INFO: rcu_sched detected stalls on CPUs/tasks: { 7} (detected by 4, t=5253 jiffies, g=7178, c=7177, q=3144)
Task dump for CPU 7:
lpm_ipfwd_app R running 0 2534 1 0x00002006
Call Trace:
[e9ff5a90] [c19a1180] 0xc19a1180 (unreliable)
INFO: rcu_sched detected stalls on CPUs/tasks: { 7} (detected by 4, t=21008 jiffies, g=7178, c=7177, q=4289)
Task dump for CPU 7:
lpm_ipfwd_app R running 0 2534 1 0x00002006
Call Trace:
[e9ff5a90] [c19a1180] 0xc19a1180 (unreliable)

the command of lpm-ipfwd is : "lpm_ipfwd_app 7 -c usdpaa_config_p4_serdes_0xe_ok.xml -p usdpaa_policy_hash_ipv4_arp_coarse_classify_ok.xml -d 0x10000000 -b 1600:1600:1600"