Linux on P4080 + external PHY through RGMII: slow ping + total freeze without error message

[paramahansapolo]

Hi folks,

I was wandering if anybody have come across the following problem:

I have a customized board based on QorIQ P4080. And, I'm are using linux do bring up.

The board connect to an external PHY (Marvell Alaska 88E1512), and so far I have been able to ping the another computer.  The problem is that:

1).The pings have a big delay (target and host are connected directly)

2). After a few ping packets pass through the kernel freezes without giving any clue about the problem.

In configuring the device tree (see P4080 RGMII Port don't connecting with external PHY ), I needed to set phy-connection-type = "rgmii-id"  instead of  phy-connection-type = "rgmii". Without it, I was able to manage the PHY but no packets where going out or coming in at level L1 (that is, no packets exiting or entering the PHY on the side connected to the network).  We observed the traffic on the target itself by running tcpdump on it (I think, that is, between MAC and PHY). All TX where accounted for; but no packet reached the host.

I will appreciate any suggestion about the problem or how to continue to tackle it.

This question is related to P4080 RGMII Port don't connecting with external PHY 

Logs follow below, full device tree is in attachments as well as dmesg.

* DT network node

-----------------------

(in node   fman0: fman@400000 )

mdio0: mdio@e1120 {
        reg = <0xe1120 0xee0>;
        //interrupts = <0x64 0x1 0x0 0x0>;
        compatible = "fsl,fman-mdio";
        #address-cells = <0x1>;
        #size-cells = <0x0>;
        phandle = <0x4d>;
        linux,phandle = <0x4d>;

        tbi0: tbi-phy@1c {
                reg = <0x1c>;
                device_type = "tbi-phy";
                phandle = <0x18>;
                linux,phandle = <0x18>;
        };

        phy0: ethernet-phy@0 {
                reg = <0x0>;
                compatible = "marvell";
                device_type = "ethernet-phy";
                phandle = <0x19>;
                linux,phandle = <0x19>;
        };
};

fman0_rx0: port@88000 {
        reg = <0x88000 0x1000>;
        compatible = "fsl,fman-port-1g-rx";
        fsl,liodn = <0x2b>;
        cell-index = <0x0>;
        phandle = <0x1a>;
        linux,phandle = <0x1a>;
};

fman0_tx0: port@a8000 {
        reg = <0xa8000 0x1000>;
        compatible = "fsl,fman-port-1g-tx";
        fsl,qman-channel-id = <0x41>;
        cell-index = <0x0>;
        phandle = <0x1b>;
        linux,phandle = <0x1b>;
};

enet0: ethernet@e0000 {
        reg = <0xe0000 0x1000>;
        compatible = "fsl,fman-1g-mac";
        cell-index = <0x0>;
        local-mac-address = [00 04 9f 02 1f bb];
        tbi-handle = <&tbi0>;
        phy-handle = <&phy0>;
        phy-mode = "rgmii-id";
        phy-connection-type = "rgmii-id";
        fsl,port-handles = <&fman0_rx0 &fman0_tx0>;
        ptimer-handle = <&ptp_timer0>;
        phandle = <0x4f>;
        linux,phandle = <0x4f>;
};

* SHELL    LOG

--------------------

Reserved memory: initialized node qman-pfdr, compatible id fsl,qman-pfdr
Reserved memory: initialized node qman-fqd, compatible id fsl,qman-fqd
Reserved memory: initialized node bman-fbpr, compatible id fsl,bman-fbpr
Reserved memory: initialized node pme-pdsr, compatible id fsl,pme-pdsr
Reserved memory: initialized node pme-sre, compatible id fsl,pme-sre
Using CoreNet Generic machine description
Memory CAM mapping: 256/256/256 Mb, residual: 1280Mb
Linux version 4.1.8-rt8+gbd51baf (ppolo@localhost.localdomain) (gcc version 4.9.2 (GCC) ) #34 SMP Wed Nov 23 10:58:51 IST 2016
Found initrd at 0xc1001000:0xc2dc85b5
CPU maps initialized for 1 thread per core
bootconsole [udbg0] enabled
setup_arch: initmem
CoreNet Generic board
arch: exit
Zone ranges:
  DMA      [mem 0x0000000000000000-0x000000002fffffff]
  Normal   empty
  HighMem  [mem 0x0000000030000000-0x000000007fffffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x0000000000000000-0x000000007fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000007fffffff]
MMU: Allocated 1088 bytes of context maps for 255 contexts
PERCPU: Embedded 12 pages/cpu @eef2b000 s19148 r8192 d21812 u49152
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 522752
Kernel command line: root=/dev/ram rw ramdisk_size=10000000 log_buf_len=128K mem=2G console=ttyS0,115200n8
log_buf_len: 131072 bytes
early log buf free: 13304(81%)
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Sorting __ex_table...
Memory: 1887876K/2097152K available (7316K kernel code, 356K rwdata, 2200K rodata, 312K init, 458K bss, 209276K reserved, 0K cma-reserved, 1226752K highmem)
Kernel virtual memory layout:
  * 0xfff5f000..0xfffff000  : fixmap
  * 0xffc00000..0xffe00000  : highmem PTEs
  * 0xffbfd000..0xffc00000  : early ioremap
  * 0xf1000000..0xffbfd000  : vmalloc & ioremap
Hierarchical RCU implementation.
    RCU debugfs-based tracing is enabled.
    Additional per-CPU info printed with stalls.
NR_IRQS:512 nr_irqs:512 16
mpic: Setting up MPIC " OpenPIC  " version 1.2 at 190040000, max 8 CPUs
mpic: ISU size: 512, shift: 9, mask: 1ff
mpic: Initializing for 512 sources
clocksource timebase: mask: 0xffffffffffffffff max_cycles: 0x8a60dd6a9, max_idle_ns: 440795204056 ns
clocksource: timebase mult[1aaaaaab] shift[24] registered
Console: colour dummy device 80x25
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
e500 family performance monitor hardware support registered
smp: smp_85xx_start_cpu: timeout waiting for cpu 1 to reset
smp: failed starting cpu 1 (rc -11)
smp: smp_85xx_start_cpu: timeout waiting for cpu 2 to reset
smp: failed starting cpu 2 (rc -11)
smp: smp_85xx_start_cpu: timeout waiting for cpu 3 to reset
smp: failed starting cpu 3 (rc -11)
smp: smp_85xx_start_cpu: timeout waiting for cpu 4 to reset
smp: failed starting cpu 4 (rc -11)
smp: smp_85xx_start_cpu: timeout waiting for cpu 5 to reset
smp: failed starting cpu 5 (rc -11)
smp: smp_85xx_start_cpu: timeout waiting for cpu 6 to reset
smp: failed starting cpu 6 (rc -11)
smp: smp_85xx_start_cpu: timeout waiting for cpu 7 to reset
smp: failed starting cpu 7 (rc -11)
Brought up 1 CPUs
devtmpfs: initialized
clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
NET: Registered protocol family 16
Bman ver:0a02,01,00
qman-fqd addr 0x7d800000 size 0x800000
qman-pfdr addr 0x7e000000 size 0x2000000
Qman ver:0a01,01,02,00
             
PCI: Probing PCI hardware
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
PTP clock support registered
EDAC MC: Ver: 3.0.0
bman-fbpr addr 0x7c000000 size 0x1000000
Bman err interrupt handler present
Bman portal (shared) initialised, cpu 0
Bman portal (slave) initialised, cpu 1
Bman portal (slave) initialised, cpu 2
Bman portal (slave) initialised, cpu 3
Bman portal (slave) initialised, cpu 4
Bman portal (slave) initialised, cpu 5
Bman portal (slave) initialised, cpu 6
Bman portal (slave) initialised, cpu 7
Bman portals initialised
Qman err interrupt handler present
Revision=0x0102, but portal '/qman-portals@194200000/qman-portal@4000' has 0x0101
Revision=0x0102, but portal '/qman-portals@194200000/qman-portal@8000' has 0x0101
Revision=0x0102, but portal '/qman-portals@194200000/qman-portal@c000' has 0x0101
Revision=0x0102, but portal '/qman-portals@194200000/qman-portal@10000' has 0x0101
Revision=0x0102, but portal '/qman-portals@194200000/qman-portal@14000' has 0x0101
Revision=0x0102, but portal '/qman-portals@194200000/qman-portal@18000' has 0x0101
Revision=0x0102, but portal '/qman-portals@194200000/qman-portal@20000' has 0x0101
Revision=0x0102, but portal '/qman-portals@194200000/qman-portal@24000' has 0x0101
Revision=0x0102, but portal '/qman-portals@194200000/qman-portal@1c000' has 0x0101
Revision=0x0102, but portal '/qman-portals@194200000/qman-portal@0' has 0x0101
Qman portal (shared) initialised, cpu 0
Qman portal (slave) initialised, cpu 1
Qman portal (slave) initialised, cpu 2
Qman portal (slave) initialised, cpu 3
Qman portal (slave) initialised, cpu 4
Qman portal (slave) initialised, cpu 5
Qman portal (slave) initialised, cpu 6
Qman portal (slave) initialised, cpu 7
Qman portals initialised
Bman: BPID allocator includes range 32:32
Qman: FQID allocator includes range 256:256
Qman: FQID allocator includes range 32768:32768
Qman: CGRID allocator includes range 0:256
Qman: pool channel allocator includes range 33:15
No USDPAA memory, no 'fsl,usdpaa-mem' in device-tree
Switched to clocksource timebase
NET: Registered protocol family 2
TCP established hash table entries: 8192 (order: 3, 32768 bytes)
TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
UDP hash table entries: 512 (order: 2, 16384 bytes)
UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Trying to unpack rootfs image as initramfs...
rootfs image is not initramfs (no cpio magic); looks like an initrd
Freeing initrd memory: 30492K (c1001000 - c2dc8000)
futex hash table entries: 2048 (order: 5, 131072 bytes)
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(0.660:1): initialized
HugeTLB registered 4 MB page size, pre-allocated 0 pages
HugeTLB registered 16 MB page size, pre-allocated 0 pages
HugeTLB registered 64 MB page size, pre-allocated 0 pages
HugeTLB registered 256 MB page size, pre-allocated 0 pages
HugeTLB registered 1 GB page size, pre-allocated 0 pages
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered
ntfs: driver 2.1.32 [Flags: R/O].
jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
bounce: pool size: 64 pages
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
console [ttyS0] disabled
serial8250.0: ttyS0 at MMIO 0x19011c500 (irq = 36, base_baud = 18750000) is a 16550A
console [ttyS0] enabled
console [ttyS0] enabled
bootconsole [udbg0] disabled
bootconsole [udbg0] disabled
serial8250.0: ttyS1 at MMIO 0x19011c600 (irq = 36, base_baud = 18750000) is a 16550A
serial8250.0: ttyS2 at MMIO 0x19011d500 (irq = 37, base_baud = 18750000) is a 16550A
serial8250.0: ttyS3 at MMIO 0x19011d600 (irq = 37, base_baud = 18750000) is a 16550A
ePAPR hypervisor byte channel driver
Generic non-volatile memory driver v1.1
brd: module loaded
loop: module loaded
st: Version 20101219, fixed bufsize 32768, s/g segs 256
libphy: Fixed MDIO Bus: probed
libphy: Freescale PowerQUICC MII Bus: probed
Freescale FM module, FMD API version 21.1.0
Freescale FM Ports module
fsl_mac: fsl_mac: FSL FMan MAC API based driver
fsl_mac 1904e0000.ethernet: FMan dTSEC version: 0x08240101
fsl_mac 1904e0000.ethernet: FMan MAC address: 00:04:9f:02:1f:bb
fsl_dpa: FSL DPAA Ethernet driver
fsl_dpa: fsl_dpa: Probed interface eth0
fsl_advanced: FSL DPAA Advanced drivers:
fsl_proxy: FSL DPAA Proxy initialization driver
fsl_dpa_shared: FSL DPAA Shared Ethernet driver
fsl_dpa_macless: FSL DPAA MACless Ethernet driver
fsl_oh: FSL FMan Offline Parsing port driver
e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
e1000: Copyright (c) 1999-2006 Intel Corporation.
e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
e1000e: Copyright(c) 1999 - 2014 Intel Corporation.
SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic channels, max=256).
CSLIP: code copyright 1989 Regents of the University of California.
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
ohci-pci: OHCI PCI platform driver
usbcore: registered new interface driver usb-storage
i2c /dev entries driver
mpc-i2c 190118000.i2c: timeout 1000000 us
mpc-i2c 190118100.i2c: timeout 1000000 us
mpc-i2c 190119000.i2c: timeout 1000000 us
mpc-i2c 190119100.i2c: timeout 1000000 us
lm90 1-004c: Register 0x4 read failed (-6)
lm90 1-004c: Failed to read convrate register!
lm90 1-004c: Register 0x3 read failed (-6)
lm90 1-004c: Initialization failed!
Freescale(R) MPC85xx EDAC driver, (C) 2006 Montavista Software
EDAC MC0: Giving out device to module MPC85xx_edac controller mpc85xx_mc_err: DEV mpc85xx_mc_err (INTERRUPT)
MPC85xx_edac acquired irq 490 for MC
MPC85xx_edac MC err registered
EDAC MC1: Giving out device to module MPC85xx_edac controller mpc85xx_mc_err: DEV mpc85xx_mc_err (INTERRUPT)
MPC85xx_edac acquired irq 489 for MC
MPC85xx_edac MC err registered
qoriq_cpufreq: Freescale QorIQ CPU frequency scaling driver
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pltfm: SDHCI platform and OF driver helper
sdhci-esdhc 190114000.sdhc: No vmmc regulator found
sdhci-esdhc 190114000.sdhc: No vqmmc regulator found
mmc0: SDHCI controller on 190114000.sdhc [190114000.sdhc] using DMA
Freescale USDPAA process driver
fsl-usdpaa: no region found
Freescale USDPAA process IRQ driver
fsl-pme 190316000.pme: ver: 0x00100201
Freescale pme2 db driver
Freescale pme2 scan driver
fsl-pme2-scan: device pme_scan registered
Freescale hypervisor management driver
fsl-hv: no hypervisor found
ipip: IPv4 over IPv4 tunneling driver
Initializing XFRM netlink socket
NET: Registered protocol family 10
sit: IPv6 over IPv4 tunneling driver
NET: Registered protocol family 17
NET: Registered protocol family 15
Key type dns_resolver registered
fsl_generic: FSL DPAA Generic Ethernet driver
hctosys: unable to open rtc device (rtc0)
RAMDISK: gzip image found at block 0
mmc0: MAN_BKOPS_EN bit is not set
mmc0: new high speed MMC card at address 0001
mmcblk0: mmc0:0001 MMC16G 14.5 GiB
mmcblk0boot0: mmc0:0001 MMC16G partition 1 16.0 MiB
mmcblk0boot1: mmc0:0001 MMC16G partition 2 16.0 MiB
 mmcblk0: p1
VFS: Mounted root (ext2 filesystem) on device 1:0.
devtmpfs: mounted
Freeing unused kernel memory: 312K (c094e000 - c099c000)
INIT: version 2.88 booting
Starting udev
udevd[803]: starting version 182
fsl_dpa fsl,dpaa:ethernet@0 fm1-gb0: renamed from eth0
udevd[807]: renamed network interface eth0 to fm1-gb0
bootlogd: cannot allocate pseudo tty: No such file or directory
random: dd urandom read with 11 bits of entropy available
Populating dev cache
hwclock: can't open '/dev/misc/rtc': No such file or directory
Sun Jun  5 08:11:01 UTC 2016
hwclock: can't open '/dev/misc/rtc': No such file or directory
Running postinst /etc/rpm-postinsts/100-sysvinit-inittab...
Running postinst /etc/rpm-postinsts/101-inetutils-inetd...
Running postinst /etc/rpm-postinsts/102-inetutils-ftpd...
update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing)
 Removing any system startup links for run-postinsts ...
  /etc/rcS.d/S99run-postinsts
INIT: Entering runlevel: 5
Configuring network interfaces... done.
Starting OpenBSD Secure Shell server: sshd
  generating ssh RSA key...
  generating ssh ECDSA key...
  generating ssh DSA key...
  generating ssh ED25519 key...
done.
hwclock: can't open '/dev/misc/rtc': No such file or directory
Starting network benchmark server: netserver.
Starting system log daemon...0
Starting kernel log daemon...0
Starting internet superserver: xinetd.

QorIQ SDK (FSL Reference Distro) 2.0 p4080ds /dev/ttyS0

p4080 login: root
root@p4080:~# ifconfig fm1-gb0 10.10.10.40
root@p4080:~# random: nonblocking pool is initialized
      
root@p4080:~# ethtool fm1-gb0
Settings for fm1-gb0:
    Supported ports: [ MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0xffffffff (-1)
                   drv probe link timer ifdown ifup rx_err tx_err tx_queued intr tx_done rx_status pktdata hw wol 0xffff8000
    Link detected: yes
root@p4080:~#
root@p4080:~#
root@p4080:~#
root@p4080:~#
root@p4080:~# ping 10.10.10.200
PING 10.10.10.200 (10.10.10.200) 56(84) bytes of data.
64 bytes from 10.10.10.200: icmp_seq=1 ttl=64 time=996 ms
64 bytes from 10.10.10.200: icmp_seq=2 ttl=64 time=809 ms
64 bytes from 10.10.10.200: icmp_seq=3 ttl=64 time=997 ms
64 bytes from 10.10.10.200: icmp_seq=4 ttl=64 time=2006 ms
64 bytes from 10.10.10.200: icmp_seq=5 ttl=64 time=998 ms
64 bytes from 10.10.10.200: icmp_seq=6 ttl=64 time=2006 ms
64 bytes from 10.10.10.200: icmp_seq=7 ttl=64 time=998 ms

THAT'S   IT  TOTAL  FREEZE  :^?

Please, let me know if you have any clue of what is going on.

Many thanks in advance,

-Paramahansa

Original Attachment has been moved to: myfdt.dts.zip

Original Attachment has been moved to: dmesg.txt.zip

[bpe]

It is not possible to identify the root cause of the problem without
knowing where the execution spins. Thus, the main suggestion is,
attach a reliable JTAG debugger, identify the module and the routine
where it gets stuck. Only a few generic suggestions can be given
with the information you posted:

1. Use the kernel from the SDK that supports your chip revision.

2. Undo all custom patches, if there were any.

3. Use the kernel configuration as close as possible to the SDK default.

4. Make sure you download FMan microcode version that supports your
   chip.
   
5. Test you system memory for stability.

6. Make sure there is enough memory in your system. If unsure,
   install the same amount as the development board has.

Have a great day,
Platon

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------