T1024 Ethernet stability

[rdk]

Hi,

We have developed a CPU board the QoIQ T1024. We have 2 Ethernet 10/100 ports, both Ethernet phy are Micrel KSZ9031. When doing regular network operations, such as ssh to the board and low rate network communication everything is fine. When doing "high" rate communication with tools like iperf, the Ethernet port stops working after some seconds and after that only a reset of the CPU can bring back Ethernet communication.

Any hint about how to investigate this issue would be appreciated.

Best regards,  

[yipingwang]

Please try to boot up your custom board with the uImage(https://drive.google.com/file/d/1vG9NUAlxgEA3JjKwbb19x-wKNc6LqIpe/view), which comes from QorIQ SDK 2.0 pre-built image ISO for T1024RDB. Then check whether this problem can be reproduced with this uImage.

.

[rdk]

Hi,

I did have a try with the uImage attached to the reply with no luck, the kernel would panic on PHY detection :

libphy: Fixed MDIO Bus: probed
libphy: Freescale XGMAC MDIO Bus: probed
libphy: Freescale XGMAC MDIO Bus: probed
libphy: Freescale XGMAC MDIO Bus: probed
libphy: Freescale XGMAC MDIO Bus: probed
------------[ cut here ]------------
kernel BUG at /home/jenkins/ci/sdk/master/t1024rdb/sources/linux/drivers/staging/fsl_qbman/qman_high.c:1281!
Oops: Exception in kernel mode, sig: 5 [#1]
SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.8-rt8+gbd51baf #1
task: e9047910 ti: e9048000 task.ti: e9048000
**bleep**: c058ec34 LR: c044ec30 CTR: 00000000
REGS: e9049b30 TRAP: 0700   Not tainted  (4.1.8-rt8+gbd51baf)
MSR: 00029002 <CE,EE,ME>  CR: 42012448  XER: 00000000

GPR00: c044ebd4 e9049be0 e9047910 c0a0d500 00000008 00000020 00000000 e9049ef8  
GPR08: 00000008 00000008 00000001 706f6c69 22012442 00000000 c0002700 00000000  
GPR16: 00000000 00000000 00000000 00000000 00000000 00000000 c09a5540 00000000  
GPR24: 00080000 fff700ff 00000000 00000000 efff5cb4 00000001 e94fe000 e94fe000  
**bleep** [c058ec34] qman_affine_channel+0x64/0x80
LR [c044ec30] ReadFmDevTreeNode+0x750/0xc50
Call Trace:
[e9049be0] [c044ebd4] ReadFmDevTreeNode+0x6f4/0xc50 (unreliable)
[e9049d10] [c044f5a4] fm_probe+0x14/0x16c0
[e9049d60] [c0315870] platform_drv_probe+0x50/0x110
[e9049d80] [c0313928] driver_probe_device+0x1f8/0x330
[e9049da0] [c0313b9c] __driver_attach+0xbc/0xc0
[e9049dc0] [c03115ac] bus_for_each_dev+0x6c/0xc0
[e9049df0] [c0312da8] bus_add_driver+0x178/0x250
[e9049e10] [c0314368] driver_register+0x88/0x140
[e9049e20] [c0450d48] LNXWRP_FM_Init+0x48/0x60
[e9049e30] [c0929e6c] fm_load+0x10/0x60
[e9049e40] [c0001f80] do_one_initcall+0x90/0x1f0
[e9049eb0] [c0909b3c] kernel_init_freeable+0x138/0x208
[e9049f30] [c0002714] kernel_init+0x14/0x110
[e9049f40] [c000f278] ret_from_kernel_thread+0x5c/0x64
Instruction dump:
81290070 812901f4 7d2a2e70 7d4a0194 3c60c0a1 3863d500 554a103a 552806fe  
7d43502e 7d4a4430 554a07fe 694a0001 <0f0a0000> 5529083c 7d234a14 a0690004  
---[ end trace 868e592253a8b124 ]---

Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005

[yipingwang]

Which version Linux Kernel(SDK) do you use?

Would you please send your Kernel Image and configuration file to me to do more verification on my demo board? I need to decide whether this problem is caused by software or hardware.

In addition, please use FMAN ucode provided in SDK 2.0 release.

[rdk]

Hi, 

Thank you for the reply, I'm using the fman micro code from SDK 2.0. Since the SDK 2.0 includes outdated kernel and software, I have switched to yocto-sdk-2.7 which feature a Linux kernel 4.14.122.

Kernel image and config is attached in the zip file.

For your information I have give a try with the latest LTS linux kernel from the repository git://source.codeaurora.org/external/qoriq/qoriq-components/linux (branch linux-5.4). I have integrated it in the yocto-sdk-2.7 

With the kernel 5.4.3 I do not have the behavior I have described with kernel 4.14.122, Ethernet communication is stable. I'm currently validating all our software developments against this kernel.

Best regards,

Renaud

[yipingwang]

Hello Renaud,

I downloaded Yocto 2.7 from the following link, and built Kernel image, the Kernel version should be 4.14.104, not 4.14.122 mentioned by you.

https://source.codeaurora.org/external/qoriq/qoriq-components/yocto-sdk/tree/readme?h=warrior

I have attached Kernel image built from Yocto 2.7, you could try it on your target board.

Thanks,

Yiping

[rdk]

Hello Yiping,

Sorry for the late feedback.

I have try the uImage you have sent but it does panic (see logs here under). I tried it on our board and on the T1024RDB. 

mmcblk0: mmc0:59b4 USD   3.75 GiB  
mmcblk0: p1 p2
EXT4-fs (mmcblk0p2): warning: mounting unchecked fs, running e2fsck is recommended
EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
VFS: Mounted root (ext4 filesystem) on device 179:2.
devtmpfs: mounted
Freeing unused kernel memory: 452K
This architecture does not have kernel memory protection.
request_module: kmod_concurrent_max (0) close to 0 (max_modprobes: 50), for module binfmt-4c46, throttling...
request_module: modprobe binfmt-4c46 cannot be processed, kmod busy with 50 threads for more than 5 seconds now
Starting init: /sbin/init exists but couldn't execute it (error -8)
request_module: kmod_concurrent_max (0) close to 0 (max_modprobes: 50), for module binfmt-4c46, throttling...
request_module: modprobe binfmt-4c46 cannot be processed, kmod busy with 50 threads for more than 5 seconds now
Starting init: /bin/sh exists but couldn't execute it (error -8)
Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.14.104+g7e52442b136c #1
Call Trace:
[e9097eb0] [c0972e78] dump_stack+0x88/0xc0 (unreliable)
[e9097ed0] [c0038010] panic+0x118/0x278
[e9097f30] [c00026a0] kernel_init+0xec/0x108
[e9097f40] [c00123b8] ret_from_kernel_thread+0x5c/0x64
Rebooting in 180 seconds..

Are you sure that kernel from 2.7 sdk is 4.14.104 ? Because I double checked the linux-qoriq_4.14.bb that reference commit e3aede3681a610e6d42c397b96547dc90f6f9e87 of repository git://source.codeaurora.org/external/qoriq/qoriq-components/linux. If I execute le following command it dive me 4.14.122 :

git clone https://source.codeaurora.org/external/qoriq/qoriq-components/linux

cd linux

git checkout e3aede3681a610e6d42c397b96547dc90f6f9e87

head -n 6 Makefile

# SPDX-License-Identifier: GPL-2.0
VERSION = 4
PATCHLEVEL = 14
SUBLEVEL = 122
EXTRAVERSION =
NAME = Petit Gorille

Best regards,

Renaud

[yipingwang]

Hello Renaud,

It seems that there is problem with mounting rootfs filesystem in your log, do you use booting rootfs from SD card on the target board? How do you configure bootargs, would you please provide the whole console log?

Please refer to the following bootargs configuration, I assumed that you had deployed rootfs  filesystem in  mmcblk0p2

setenv bootargs "console=ttyS0,115200 root=/dev/mmcblk0p2  rw rootdelay=5"

In addition, did you download yocto SDK from the following link.

https://source.codeaurora.org/external/qoriq/qoriq-components/yocto-sdk/tree/readme?h=warrior

The linux Kernel commit ID should be "7e52442b136c439c142c4859a70ce92208c481a9" in sources/meta-freescale/recipes-kernel/linux/linux-qoriq_4.14.bb.

Thanks,

Yiping