Kernel panic when up the pfe interface

[StanleyShen]

Hi

We are using the pfe interface to communicate to 5G module,

when up the PFE interface through ifconfig(time 36.025547), an kernel panic occurs as blow:

（Not 100% reproducible）

[   32.980723] pfeng 46000000.pfe pfe2 (uninitialized): PTP HW addend 0x80000000, max_adj configured to 46566128 ppb
[   32.980736] pfeng 46000000.pfe: IEEE1588: Input Clock: 200000000Hz, Output: 100000000Hz, Accuracy: 10.0ns
[   32.986068] pfeng 46000000.pfe pfe2 (uninitialized): Registered PTP HW clock successfully on EMAC2
[   32.990989] pfeng 46000000.pfe pfe2: registered
[   32.993695] loadpfe: 5G module is not setup,reload_pf2_ko...
[   36.024860] pfeng 46000000.pfe: HIF2 started
[   36.024889] pfeng 46000000.pfe pfe2: configuring for fixed/rgmii link mode
[   36.024963] pfeng 46000000.pfe pfe2: Set TX clock to 125000000Hz
[   36.025232] pfeng 46000000.pfe pfe2: Set TX clock to 125000000Hz
[   36.025257] pfeng 46000000.pfe pfe2: Link is Up - 1Gbps/Full - flow control off
[   36.025547] IPv6: ADDRCONF(NETDEV_CHANGE): pfe2: link becomes ready
[   37.043292] pfeng 46000000.pfe: ERR: (PFENG_DEV) event 19 - TLITE Watchdog trigered: [pfe_wdt_csr.c:126] 
[   37.043504] SError Interrupt on CPU0, code 0x00000000bf000002 -- SError
[   37.043515] CPU: 0 PID: 618 Comm: Global polling  Tainted: G           O      5.15.119-rt65+g96b69aed2 #1
[   37.043526] Hardware name: GRCE DOCAP-STD (DT)
[   37.043531] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   37.043541] pc : el1_abort+0x30/0x6c
[   37.043562] lr : el1_abort+0x24/0x6c
[   37.043571] sp : ffffffc00bea36b0
[   37.043574] x29: ffffffc00bea36b0 x28: ffffff88064d91c0 x27: 0000000000000000
[   37.043589] x26: 0000000000000000 x25: ffffffc0090f6920 x24: ffffff880569d880
[   37.043603] x23: 0000000060000005 x22: ffffffc000bed2ac x21: ffffffc00d090108
[   37.043615] x20: 0000000096000210 x19: ffffffc00bea36f0 x18: ffffffffffffffff
[   37.043629] x17: 0000000000000000 x16: 0000000000000000 x15: ffffffc00bea3d90
[   37.043642] x14: 63c46b65a62b8000 x13: 00ffffc000c28c08 x12: ffffffc000bdb8ac
[   37.043657] x11: 00ffff880543dd00 x10: ffffffc0080d87d4 x9 : 0000000000000000
[   37.043669] x8 : ffffffc00bea3bc8 x7 : 0000000000000000 x6 : ffffff88064d91c0
[   37.043682] x5 : ffffffc00bea36f0 x4 : 0000002000000000 x3 : 0000000000000025
[   37.043695] x2 : 0000002200000000 x1 : 0000000096000210 x0 : 0000000000000000
[   37.043712] Kernel panic - not syncing:
[   37.043715] Asynchronous SError Interrupt
[   37.043719] CPU: 0 PID: 618 Comm: Global polling  Tainted: G           O      5.15.119-rt65+g96b69aed2 #1
[   37.043729] Hardware name: GRCE DOCAP-STD (DT)
[   37.043734] Call trace:
[   37.043736]  dump_backtrace+0x0/0x1b0
[   37.043751]  show_stack+0x18/0x24
[   37.043762]  dump_stack_lvl+0x68/0x84
[   37.043774]  dump_stack+0x18/0x34
[   37.043784]  panic+0xb8/0x378
[   37.043792]  add_taint+0x0/0xc0
[   37.043803]  arm64_serror_panic+0x64/0x74
[   37.043812]  do_serror+0x58/0x60
[   37.043819]  el1h_64_error_handler+0x30/0x50
[   37.043827]  el1h_64_error+0x78/0x7c
[   37.043835]  el1_abort+0x30/0x6c
[   37.043844]  el1h_64_sync_handler+0xa4/0xd0
[   37.043854]  el1h_64_sync+0x78/0x7c
[   37.043861]  pfe_pe_mem_read+0x88/0x18c [pfeng]
[   37.044005]  pfe_pe_memcpy_from_dmem_to_host_32_nolock+0xa4/0x120 [pfeng]
[   37.044104]  pfe_pe_get_fw_messages_nolock+0xc8/0x524 [pfeng]
[   37.044200]  pfe_class_isr+0x90/0x18c [pfeng]
[   37.044297]  pfe_poller_func+0x144/0x25c [pfeng]
[   37.044395]  thread_func+0x28/0x60 [pfeng]
[   37.044494]  kthread+0x160/0x174
[   37.044504]  ret_from_fork+0x10/0x20
[   38.044774] SMP: stopping secondary CPUs
[   41.770704] Kernel Offset: disabled
[   41.774161] CPU features: 0x1,20002001,20000842
[   41.778675] Memory Limit: none

It seem that the watchdog trigered, but i don't know the reason.

Also, in the following log, it was not reported that the watchdog was triggered.

[   20.780716] loadpfe: 5G module is not setup,reload_pf2_ko...
[   20.940209] copy data to lin1 share memory
[   20.940232] copy data successful
[   20.941262] LLCE LIN init success
[   20.941270] LLCE LIN is in sleeping, success.
[   20.942289] LLCE LIN wakeup success
[   20.943285] LLCE LIN is operational, success.
[   23.811487] pfeng 46000000.pfe: HIF2 started
[   23.811516] pfeng 46000000.pfe pfe2: configuring for fixed/rgmii link mode
[   23.811591] pfeng 46000000.pfe pfe2: Set TX clock to 125000000Hz
[   23.811857] pfeng 46000000.pfe pfe2: Set TX clock to 125000000Hz
[   23.811882] pfeng 46000000.pfe pfe2: Link is Up - 1Gbps/Full - flow control off
[   23.812171] IPv6: ADDRCONF(NETDEV_CHANGE): pfe2: link becomes ready
[   23.827428] Internal error: synchronous external abort: 96000210 [#1] PREEMPT SMP	
[   23.827428] SError Interrupt on CPU0, code 0x00000000bf000002 -- SError
[   25.899053] Modules linked in: pfeng(O) brcmfmac(O) cypress_cfg80211(O) compat(O) brcmutil(O) nxp_llce_lin llce_logger llce_can llce_mailbox llce_core [last unloaded: pfeng]
[   25.905648] CPU: 0 PID: 555 Comm: Global polling  Tainted: G           O      5.15.119-rt65+g96b69aed2 #1
[   25.921091] CPU: 2 PID: 24 Comm: kworker/2:0 Tainted: G           O      5.15.119-rt65+g96b69aed2 #1
[   25.930638] Hardware name: GRCE DOCAP-STD (DT)
[   25.939752] Hardware name: GRCE DOCAP-STD (DT)
[   25.944180] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   25.948610] Workqueue: mld mld_ifc_work
[   25.955549] pc : el1_abort+0x30/0x6c
[   25.959370] pstate: 800000c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   25.962926] lr : el1_abort+0x24/0x6c
[   25.969871] pc : pfe_hif_chnl_cfg_irq_mask+0x10/0x38 [pfeng]
[   25.973431] sp : ffffffc00bfab6b0
[   25.979073] lr : pfe_hif_chnl_irq_mask+0x30/0x4c [pfeng]
[   25.982370] x29: ffffffc00bfab6b0 x28: ffffff8809cbc340 x27: 0000000000000000
[   25.987665] sp : ffffffc00913be60
[   25.994782] x26: 0000000000000000 x25: ffffffc0090f6920 x24: ffffff8806ce6980
[   25.998081] x29: ffffffc00913be60 x28: ffffff8800141140 x27: ffffffc008e25008
[   26.005198] x23: 0000000060000005 x22: ffffffc000a5b2ac x21: ffffffc010090108
[   26.012316] x26: ffffff8800959800 x25: ffffffc008e25008 x24: ffffffc008fc8680
[   26.019433] x20: 0000000096000210 x19: ffffffc00bfab6f0 x18: 0000000000000000
[   26.026551] x23: ffffffc00913bf34 x22: 000000000000005e x21: 0000000000000000
[   26.033669] x17: 0000000000000000 x16: 0000000000000000 x15: 0000007f9672e1b0
[   26.040786] x20: ffffff880183d240 x19: ffffff880183d200 x18: ffffffc00910b16c
[   26.047904] x14: 0000000000000000 x13: 00000000000000c8 x12: ffffffc008c69938
[   26.055021] x17: ffffffc856a3c000 x16: ffffffc009138000 x15: 0000000000000000
[   26.062139] x11: 0000000000000028 x10: 0101010101010101 x9 : ffffffc00bfabcb0
[   26.069257] x14: 0000010024000000 x13: 33efbeff01000000 x12: 00000000000002ff
[   26.076374] x8 : 7f7f7f7f7f7f7f7f x7 : 0000000000000000 x6 : ffffff8809cbc340
[   26.083492] x11: 0000000000000040 x10: ffffff8800007268 x9 : ffffff8800007260
[   26.090609] x5 : ffffffc00bfab6f0 x4 : 0000002000000000 x3 : 0000000000000025
[   26.097727] x8 : ffffff88004126e8 x7 : 0000000000000000 x6 : 0000000000000000
[   26.104845] x2 : 0000002200000000 x1 : 0000000096000210 x0 : 0000000000000000
[   26.111962] x5 : ffffff88004126c0 x4 : 0000000000010403 x3 : 0000000000000000
[   26.119081] Kernel panic - not syncing:
[   26.126197] x2 : 0000000000000001 x1 : 0000000000000364 x0 : ffffffc010098364
[   26.130017] Asynchronous SError Interrupt
[   26.137136] Call trace:
[   26.137135] CPU: 0 PID: 555 Comm: Global polling  Tainted: G           O      5.15.119-rt65+g96b69aed2 #1
[   26.153104] Hardware name: GRCE DOCAP-STD (DT)
[   26.157533] Call trace:
[   26.159961]  dump_backtrace+0x0/0x1b0
[   26.163607]  show_stack+0x18/0x24
[   26.166906]  dump_stack_lvl+0x68/0x84
[   26.170551]  dump_stack+0x18/0x34
[   26.173850]  panic+0xb8/0x378
[   26.176801]  add_taint+0x0/0xc0
[   26.179926]  arm64_serror_panic+0x64/0x74
[   26.183918]  do_serror+0x58/0x60
[   26.187130]  el1h_64_error_handler+0x30/0x50
[   26.191383]  el1h_64_error+0x78/0x7c
[   26.194942]  el1_abort+0x30/0x6c
[   26.198153]  el1h_64_sync_handler+0xa4/0xd0
[   26.202320]  el1h_64_sync+0x78/0x7c
[   26.205792]  pfe_pe_mem_read+0x88/0x18c [pfeng]
[   26.210308]  pfe_pe_memcpy_from_dmem_to_host_32_nolock+0xa4/0x120 [pfeng]
[   26.217078]  pfe_pe_get_fw_messages_nolock+0xc8/0x524 [pfeng]
[   26.222807]  pfe_class_isr+0x90/0x18c [pfeng]
[   26.227147]  pfe_poller_func+0x144/0x25c [pfeng]
[   26.231747]  thread_func+0x28/0x60 [pfeng]
[   26.235827]  kthread+0x160/0x174
[   26.239036]  ret_from_fork+0x10/0x20
[   26.242599]  pfe_hif_chnl_cfg_irq_mask+0x10/0x38 [pfeng]
[   26.247891]  pfeng_hif_chnl_direct_isr+0x1c/0x40 [pfeng]
[   26.253187]  __handle_irq_event_percpu+0x60/0x250
[   26.257872]  handle_irq_event+0x64/0xfc
[   26.261693]  handle_fasteoi_irq+0xc0/0x220
[   26.265772]  handle_domain_irq+0x98/0xe4
[   26.269679]  gic_handle_irq+0x54/0x130
[   26.273411]  call_on_irq_stack+0x20/0x4c
[   26.277317]  do_interrupt_handler+0x54/0x60
[   26.281483]  el1_interrupt+0x30/0x80
[   26.285040]  el1h_64_irq_handler+0x18/0x24
[   26.289120]  el1h_64_irq+0x78/0x7c
[   26.292507]  preempt_count_sub+0x8/0xd4
[   26.296326]  sch_direct_xmit+0x104/0x350
[   26.300230]  __dev_queue_xmit+0x4c0/0xb14
[   26.304224]  dev_queue_xmit+0x14/0x20
[   26.307870]  neigh_resolve_output+0xfc/0x1a0
[   26.312122]  ip6_finish_output2+0x198/0x790
[   26.316290]  __ip6_finish_output+0x154/0x280
[   26.320543]  ip6_finish_output+0x34/0xd0
[   26.324449]  ip6_output+0x74/0x1d0
[   26.327833]  NF_HOOK.constprop.0+0xd4/0xe0
[   26.331914]  mld_sendpack+0x24c/0x440
[   26.335560]  mld_ifc_work+0x244/0x4e0
[   26.339205]  process_one_work+0x1d4/0x490
[   26.342824] SMP: stopping secondary CPUs
[   26.343198]  worker_thread+0x13c/0x490
[   26.350836]  kthread+0x160/0x174
[   26.354048]  ret_from_fork+0x10/0x20
[   26.357614] Code: d503233f 53180421 11059021 8b214000 (b9400001) 
[   26.363687] ---[ end trace e96ab85bba3102c9 ]---
[   26.368420] Kernel panic - not syncing:
[   26.372101] synchronous external abort: Fatal exception in interrupt
[   26.372227] Kernel Offset: disabled
[   26.381911] CPU features: 0x1,20002001,20000842
[   26.386424] Memory Limit: none
[   26.389465] Rebooting in 10 seconds..

Env:

BSP38.0  pfe 1.5.0

[irfank]

HEllo

Are there any updates regarding this topic? We are experiencing the same issue .

pc : el1_abort+0x30/0x64
lr : el1_abort+0x24/0x64
sp : ffffffc00a59b6c0
x29: ffffffc00a59b6c0 x28: ffffff8001a56200 x27: 0000000000000000
x26: 0000000000000000 x25: 0000000000000000 x24: ffffff8005911f80
x23: 0000000060000005 x22: ffffffc000c41b1c x21: ffffffc00d090108
x20: 0000000096000210 x19: ffffffc00a59b700 x18: ffffffffffffffff
x17: 7420676f64686374 x16: 6157204554494c54 x15: 0000000000000000
x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
x11: 0000000000000000 x10: 0000000000000000 x9 : ffffffc000c43334
x8 : ffffffc00a59b9e4 x7 : 0000000000000000 x6 : 000000000000003f
x5 : ffffffc00a59b700 x4 : 0000002000000000 x3 : 0000000000000025
x2 : 0000002200000000 x1 : 0000000096000210 x0 : 0000000000000000
Kernel panic - not syncing: Asynchronous SError Interrupt
CPU: 1 PID: 253 Comm: Global polling Tainted: G O 6.1.76-yocto-standard #1
Hardware name: -----
Call trace:
dump_backtrace.part.0+0xc8/0xd4
show_stack+0x20/0x30
dump_stack_lvl+0x6c/0x88
dump_stack+0x18/0x34
panic+0x180/0x33c
add_taint+0x0/0xc0
arm64_serror_panic+0x78/0x84
is_valid_bugaddr+0x0/0x30
el1h_64_error_handler+0x30/0x4c
el1h_64_error+0x64/0x68
el1_abort+0x30/0x64
el1h_64_sync_handler+0xd8/0xe4
el1h_64_sync+0x64/0x68
readl+0x4/0x20 [pfeng]
pfe_pe_memcpy_from_dmem_to_host_32_nolock+0xb4/0x134 [pfeng]
pfe_pe_get_fw_messages_nolock+0xfc/0x5a0 [pfeng]
pfe_class_isr+0xb8/0x1b8 [pfeng]
pfe_poller_func+0x14c/0x264 [pfeng]
thread_func+0x30/0x60 [pfeng]
kthread+0x110/0x114
ret_from_fork+0x10/0x20
SMP: stopping secondary CPUs
Kernel Offset: disabled
CPU features: 0x090000,00800084,0000420b

Secondary Error (Poller Disabled) We attempted to disable the pfe_poller function to isolate the issue. However, the crash persists and has simply shifted to a different code path:

Call trace:
dump_backtrace.part.0+0xc8/0xd4
show_stack+0x20/0x30
dump_stack_lvl+0x6c/0x88
dump_stack+0x18/0x34
panic+0x180/0x33c
add_taint+0x0/0xc0
arm64_serror_panic+0x78/0x84
is_valid_bugaddr+0x0/0x30
el1h_64_error_handler+0x30/0x4c
el1h_64_error+0x64/0x68
el1_abort+0x30/0x64
el1h_64_sync_handler+0xd8/0xe4
el1h_64_sync+0x64/0x68
readl+0x4/0x20 [pfeng]
pfe_pe_memcpy_from_dmem_to_host_32_nolock+0xec/0x174 [pfeng]
pfe_pe_mem_process_lock+0x48/0x1b8 [pfeng]
pfe_pe_memlock_acquire_nolock+0x4c/0x60 [pfeng]
pfe_pe_memcpy_from_host_to_dmem_32+0x7c/0x138 [pfeng]
pfe_class_write_dmem+0x114/0x180 [pfeng]
pfe_phy_if_write_to_class_nostats+0x48/0x88 [pfeng]
pfe_phy_if_promisc_enable+0x4c/0x198 [pfeng]
pfeng_netif_set_rx_mode+0x230/0x320 [pfeng]
__dev_set_rx_mode+0x70/0xb0
dev_set_promiscuity+0x7c/0x9c
vlan_dev_change_rx_flags+0x64/0x7c [8021q]
__dev_set_promiscuity+0x128/0x130
dev_set_promiscuity+0x2c/0x9c
packet_dev_mc+0x34/0xe0
packet_setsockopt+0xbac/0x11ac
__sys_setsockopt+0xa8/0x1a4
__arm64_sys_setsockopt+0x30/0x40
invoke_syscall+0x5c/0x130
el0_svc_common.constprop.0+0x4c/0xf4
do_el0_svc+0x34/0xc0
el0_svc+0x28/0xa0
el0t_64_sync_handler+0xf4/0x120
el0t_64_sync+0x18c/0x190
SMP: stopping secondary CPUs
Kernel Offset: disabled
CPU features: 0x090000,00800084,0000420b

---------------------

Could you please provide insight into what might be causing this sequence?

BSP38, pfe driver:1.5, pfe firmware:1.9.  Depends on pfe frimware release note. pfe frimware has backward compatibility. 

[alejandro_e]

Hello @irfank,

This post was closed due to inactivity. Please create a new post in the community so a member of my team can offer you support.

Thanks 

[StanleyShen]

Here is the panic log

[alejandro_e]

Hello @StanleyShen,

After a long period of inactivity the community posts get closed automatically, please create anew post so I may receive proper notifications. You can include the link to this post and my username in the new post to ensure working with me if that is what you prefer. Sorry for the inconveniences. 

Best regards.

[StanleyShen]

@alejandro_e 

Pls refer to the attachment file 

[StanleyShen]

@alejandro_e 

Hi, Since the same problem reappeared ,can you tell me how to determine the firmware version of PFE and Where to download 1.7.1 version of the firmware?

Auto Linux

[alejandro_e]

Hello @StanleyShen ,

I will need more information about you setup to be able to help you.

• Which BSP are you using?
• Which PFE firmware version are you using?
• What modifications have you added to the BSP?
• Are you using the S32G-VNP-RDB2 board or a custom board?
• Are you using Yocto build or manual build?

Thanks in advance for the information.

[StanleyShen]

@alejandro_e 

Which BSP are you using?

BSP38.0

Which PFE firmware version are you using?

1.5.0

What modifications have you added to the BSP?

The background is that according to the characteristic of  RGMII, the RX clock of PFE needs to be initialized by the clock of the other party (5G module) before it can proceed.
However, we cannot know the startup time of the other party (5G module),
so the current method is to continuously load and unload the PFE driver (rmmod&modprobe pfeng),
and then confirm whether communication with the 5G module is OK through the ping command.
Therefore, during each startup process, the PFE driver will be continuously unloaded and loaded every 3 seconds until communication with 5G is normal.
We have conducted multiple restart tests on this solution and found that it works most of the time,
but there are occasional instances (approximately once in every 20 attempts) where the above crash problem occurs.

Are you using the S32G-VNP-RDB2 board or a custom board?

It's a  custome board 

Are you using Yocto build or manual build?

using Yocto build and the PFE firmware was created at /tmp/work/${machine-name}-fsl-linux/pfe/1.0.0-r0/image/lib/firmware directory

[alejandro_e]

1.5.0 Should be the Linux driver version, the FW should have a different version, can you confirm you are using firmware version 1.7.1? which is the version compatible with with BSP38 as per the release notes of that version.

One thing I can recommend is to use modprobe to remove and add the module, instead of rmmod; or find another mechanism of detecting the 5G module instead of loading and unloading the driver, this way, once the module is detected you can load the driver.

[StanleyShen]

@alejandro_e 

Hi,

> One thing I can recommend is to use modprobe to remove and add the module, instead of rmmod; or find another mechanism > of detecting the 5G module instead of loading and unloading the driver, this way, once the module is detected you can load the > driver.

Currently, pfeng.ko is automatically loaded when the system starts.

Since the same problem reappeared ,can you tell me how to determine the firmware version of PFE and Where to download 1.7.1 version of the firmware?

[StanleyShen]

CPU is  S32G274A