FAQs
English (US) 日本語 | Japanese 中文 | Chinese (Simplified)

Custom S32G399A board: No frames cross switch-facing RGMII bus in either direction

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Custom S32G399A board: No frames cross switch-facing RGMII bus in either direction

yesterday
102 Views
pcentauri92
Contributor II

Board: Custom S32G399A based module, derived from S32G-VNP-RDB3. PFE_MAC1 connected via RGMII (PE_02–PE_13) to an NXP SJA1110A switch port 2, configured as the DSA CPU port (in-tree sja1105 driver, kernel 6.x BSP43.0).

Topology:
- PFE_MAC0: SGMII via SerDes1 lane1, Mode 1
- PFE_MAC1: RGMII to SJA1110A port 2 (DSA CPU port) — the port in question
- PFE_MAC2: SGMII via SerDes0 lane1

The S32G MACs are configured as follows:
+---------+--------------+------------------+
|                    | LANE 0               | LANE 1                        |
+---------+--------------+------------------+
| SERDES0    | GMAC (SGMII) | PFE_MAC2 (SGMII)  |
| SERDES1      | NOT USED        | PFE_MAC0 (SGMII) |
+---------+--------------+------------------+

Full U-Boot hwconfig:

hwconfig=pcie0:mode=sgmii,clock=ext,fmhz=100,xpcs_mode=both;pcie1:mode=sgmii,clock=ext,fmhz=100,xpcs_mode=0
pfeng_mode=enable,sgmii,rgmii,sgmii


DTS for port@2 (switch side):

			port@2 {
				reg = <2>;
				label = "OBC-1";
				ethernet = <&pfe_netif1>;
				phy-mode = "rgmii";
				rx-internal-delay-ps = <0>;
				tx-internal-delay-ps = <0>;
				fixed-link {
					speed = <1000>;
					full-duplex;
				};
			};


DTS for pfe_netif1 (MAC side):

&pfe_netif1 {
    phy-mode = "rgmii";
    status = "okay";
	fixed-link {
        speed = <1000>;
        full-duplex;
    };
};

 

PFE_MAC1 (pfe1) link state — confirmed up and correctly configured at the Linux/driver level:

dmesg at boot:

[    5.264108] pfeng 46000000.pfe: netif name: pfe1
[    5.274127] pfeng 46000000.pfe: netif(pfe1) linked phyif: 1
[    5.279692] pfeng 46000000.pfe: netif(pfe1) mode: std
[    5.284853] pfeng 46000000.pfe: netif(pfe1) HIFs: count 1 map 02
[    6.012884] pfeng 46000000.pfe pfe1 (uninitialized): Subscribe to HIF1
[    6.019438] pfeng 46000000.pfe pfe1 (uninitialized): Host LLTX disabled
[    6.026270] pfeng 46000000.pfe pfe1 (uninitialized): Enable HIF1
[    6.032374] pfeng 46000000.pfe pfe1 (uninitialized): setting MAC addr: 00:04:9f:be:ef:01
[    6.040545] pfeng 46000000.pfe pfe1 (uninitialized): PTP HW addend 0x80000000, max_adj configured to 46566128 ppb
[    6.060939] pfeng 46000000.pfe pfe1 (uninitialized): Registered PTP HW clock successfully on EMAC1
[    6.070441] pfeng 46000000.pfe pfe1: registered
[    6.207482] pfeng 46000000.pfe pfe1: configuring for fixed/rgmii link mode
[    6.214306] pfeng 46000000.pfe pfe1: Set TX clock to 125000000Hz
[    6.220158] pfeng 46000000.pfe pfe1: Link is Up - 1Gbps/Full - flow control off

[    5.257995] pfeng 46000000.pfe: EMAC0 interface mode: 4
[    5.290707] pfeng 46000000.pfe: EMAC1 interface mode: 9
[    5.323320] pfeng 46000000.pfe: EMAC2 interface mode: 4
[    5.354571] pfeng 46000000.pfe: Interface selected: EMAC0: 0x4 EMAC1: 0x9 EMAC2: 0x4
[    5.382609] pfeng 46000000.pfe: TX clock on EMAC0 for interface sgmii installed
[    5.390050] pfeng 46000000.pfe: RX clock on EMAC0 for interface sgmii installed
[    5.404998] pfeng 46000000.pfe: TX clock on EMAC1 for interface rgmii installed
[    5.419918] pfeng 46000000.pfe: Defer enabling of RX clock on EMAC1 for interface rgmii (ret: -5)
[    5.434235] pfeng 46000000.pfe: TX clock on EMAC2 for interface sgmii installed
[    5.448374] pfeng 46000000.pfe: RX clock on EMAC2 for interface sgmii installed
[    5.667058] pfeng 46000000.pfe: EMAC timestamp external mode bitmap: 0
[    5.998447] pfeng 46000000.pfe pfe0 (uninitialized): Registered PTP HW clock successfully on EMAC0
[    6.060939] pfeng 46000000.pfe pfe1 (uninitialized): Registered PTP HW clock successfully on EMAC1
[    6.130296] pfeng 46000000.pfe pfe2 (uninitialized): Registered PTP HW clock successfully on EMAC2
[    6.215040] pfeng 46000000.pfe: RX clock on EMAC1 for interface rgmii installed

 

Live DTB confirms the kernel matches the source DTS:

# cat /proc/device-tree/soc/pfe@46000000/ethernet@11/phy-mode
rgmii

ip a output:

6: pfe1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1536 qdisc mq state UP group default qlen 1000
link/ether 00:04:9f:be:ef:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::204:9fff:febe:ef01/64 scope link


All SJA1110 DSA slave ports correctly enumerated. This confirms the sja1105 DSA driver bound successfully
to pfe1 as the CPU port/DSA master and parsed the static config without error.

Clock tree: both TX and RX RGMII clocks enabled and attached to the correct consumer:

# cat /sys/kernel/debug/clk/clk_summary | grep pfe1
 pfe1_tx_mii                         0       0        0        125000000   0          0     50000      Y   deviceless                      no_connection_id         
 pfe1_rx_mii                         0       0        0        125000000   0          0     50000      Y   deviceless                      no_connection_id         
 pfe1_tx_rmii                        0       0        0        125000000   0          0     50000      Y   deviceless                      no_connection_id         
 pfe1_rx_rmii                        0       0        0        125000000   0          0     50000      Y   deviceless                      no_connection_id         
 pfe1_tx_rgmii                       1       1        0        125000000   0          0     50000      Y   ethernet@11                     tx_rgmii                 
 pfe1_rx_rgmii                       1       1        0        125000000   0          0     50000      Y   ethernet@11                     rx_rgmii                 
 pfe1_tx_sgmii                       0       0        0        125000000   0          0     50000      Y   deviceless                      no_connection_id         
 pfe1_rx_sgmii                       0       0        0        125000000   0          0     50000      Y   deviceless                      no_connection_id

 

So pfe1 is UP, LOWER_UP, correctly bound to the SJA1110 as DSA master, running in RGMII mode
with both clocks enabled — this rules out pfe1 being down, unbound, or misconfigured at the
Linux/driver level. The open question is specifically whether frames actually cross the
physical RGMII pins between PFE_MAC1 and SJA1110 port 2.

Issue:
No traffic appears to cross the RGMII bus between PFE_MAC1 and SJA1110 port 2 in either direction, despite everything on both sides of that bus being independently up:

Test 1 — S32G -> switch direction

Setup:

ip addr add 192.168.1.100/24 dev EPS-100bt1-9
ethtool -S pfe1 | grep '^ p02_' > before
tcpdump -i pfe1 -e -nn -c 20 > capture.txt &
arping -c 10 -I EPS-100bt1-9 192.168.1.6
ethtool -S pfe1 | grep '^ p02_' > after

 

# arping -c 10 -I EPS-100bt1-9 192.168.1.6
ARPING 192.168.1.6 from 192.168.1.100 EPS-100bt1-9
Sent 10 probes (10 broadcast(s))
Received 0 response(s)

 

$ cat capture.txt
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pfe1, link-type NULL (BSD loopback), snapshot length 262144 bytes
18:08:36.352535 AF Unknown (4294967295), length 64:
        0x0000:  ffff 0004 9fbe ef01 dadb 0c09 0806 0001  ................
        0x0010:  0800 0604 0001 0004 9fbe ef01 c0a8 0164  ...............d
        0x0020:  ffff ffff ffff c0a8 0106 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000            ............
[... 9 more identical ARP frames, all correctly DSA-tagged (dadb 0c09) and well-formed,
     plus one unrelated IPv6 background frame interleaved ...]

 

# diff before after
--- before
+++ after
@@ -1,4 +1,4 @@
-     p02_: 1
+     p02_: 0
      p02_n_runt: 0
      p02_n_soferr: 0
      p02_n_alignerr: 0

 

# grep n_rxfrm before after
before:     p02_n_rxfrm: 0
after:      p02_n_rxfrm: 0


Test 2 — switch -> S32G direction
Setup Partner board is a separate SJA1105 switch based board.

# ping -c 10 -I t1-6 192.168.1.100 (run on a separate SJA1105/1110-family switch board
connected to our port 9 / 100BASE-T1 / EPS-100bt1-9)

 

# diff before after (ethtool -S EPS-100bt1-9)
- n_rxfrm: 0
+ n_rxfrm: 9 <- port 9 physically received 9 frames from the wire

 

 # diff before after 
- p02_n_txfrm: 0
+ p02_n_txfrm: 9 <- switch fabric forwarded all 9 toward the CPU port

 

# tcpdump -i pfe1 -e -nn -c 20 (same window)
listening on pfe1, link-type NULL (BSD loopback), snapshot length 262144 bytes
[-- nothing captured --]


Port 9 received 9 real frames; the fabric forwarded all 9 toward port 2 — but
nothing arrived at pfe1.

So the SJA1110's own fabric counters show all 9 frames successfully forwarded from port 9 to port 2's egress.

But tcpdump -i pfe1 -e -nn on the S32G during this exact test shows NOTHING received.

So the DSA/software layer on the S32G side believes it's sending (case 1). The switch's internal fabric believes it's sending toward the CPU port (case 2). Neither side has any confirmation that the other actually received anything across the physical RGMII bus. Every layer adjacent to this bus works individually; the bus itself has no confirmed successful crossing in either direction.

What's been ruled out so far:
- pfeng_mode / hwconfig (xpcs_mode) — confirmed correct; EMAC1 mode is RGMII (0x9), not SGMII (it was previously misconfigured as SGMII due to xpcs_mode=both on SerDes1 forcing PFE_MAC1's XPCS into SGMII; corrected to xpcs_mode=0 since PFE_MAC0 alone only needs XPCS0)
- PFE_MAC1 TX/RX clock enablement — confirmed enabled at the correct rate (125MHz) in clk_summary
- DSA tagging and CPU port binding — confirmed working (port netdevs exist, frames get tagged with the correct destination port in the DSA header)
- SJA1110 internal fabric/forwarding — confirmed working between two other ports (9 and 2) using real external traffic
- BASE-T1 link partner — confirmed passing real frames into the switch (port 9 n_rxfrm increments from genuine wire traffic)

What hasn't been ruled out / open questions:
- Whether 1000 Mbps RGMII with zero internal delay on both MAC and switch sides (rx/tx-internal-delay-ps=0, plain "rgmii" not "rgmii-id") is compatible without delay added by board trace length — have not yet tried forcing the link down to 100 Mbps as a timing-margin test

1. Is rx/tx-internal-delay-ps=0 on both ends at 1000 Mbps RGMII expected to work, or does this combination typically require delay compensation unless the PCB explicitly accounts for it?
2. Am I missing any other configuration?

Happy to share full register dumps, if required. Appreciate any pointers before we probe the PE_02-13 bus with a logic analyzer (limited probe access due to board layout, so it is not so convenient currently.

Thanks.

Tags (3)
0 Kudos
Reply
3 Replies

yesterday
33 Views
db16122
Contributor IV

any schematics sharing from hardware side for RGMII bus between PFE_MAC1 and SJA1110 port 2 ?

0 Kudos
Reply

yesterday
35 Views
Joey_z
NXP Employee
NXP Employee

Hi，pcentauri92

Thank you for your detail information

According to my understanding, there seems to be a problem with the communication when using PFE_MAC1 RGMAII and Port 2 of SJA1110A on your development board. Is that correct?

The default configuration of S32G-VNP-RDB3 is that PFE_MAC0/1 operates in SGMII mode and is connected to SJA1110. On your development board, why did you consider using RGMII mode? It is recommended to modify the corresponding software configuration.

BR

Joey

 

0 Kudos
Reply

yesterday
20 Views
pcentauri92
Contributor II

Hi @Joey_z ,

Thank you for the response.

The module in question here is a custom design that uses the S32G399A chip along with the NXP SJA1110A ethernet switch. We based this design on the S32G-VNP-RDB3 development platform but we made quite a few changes from the base design. The PFE_MAC1 using RGMII is one of those changes. 

I am also attaching the dts file override where we change the PFE_MAC1 mode and pinmux here.

PFE_MAC1 mode configuration:

/* pfe_mdio1 is already disabled in the base config in s32gxxxa-rdb.dtsi */
&pfe_mdio1 {
	/* occupied by GMAC0 */
	status = "disabled";
};

/*
 * pfe_netif1 = PFE_MAC1 — management port to Switch-A port 2.
 * Overrides the base "sgmii" stub in s32gxxxa-rdb.dtsi.
 * Plain "rgmii" (no -id/-txid) since both MAC and switch add zero delay.
 * No phy-handle: the link partner is the SJA1110A switch, described as a
 * fixed-link on switch port@2. MDIO is not needed for link management here.
 */
&pfe_netif1 {
    phy-mode = "rgmii";
    status = "okay";
	fixed-link {
        speed = <1000>;
        full-duplex;
    };
};


PFE_MAC1 pinmux:

		/*
		* PFE_MAC1 RGMII pinmux — management port to Switch-A.
		*
		* All RX pad SSS values confirmed from S32G3 IOMUX spreadsheet.
		* TX path: output pads only, no IMCR needed.
		* RX path: input pads + IMCR registers to route pads into PFE_MAC1.
		*
		* Note: PE_07 (TXD3) uses FUNC3, not FUNC2. Similarly PE_08 (RX_CLK) output uses FUNC3; its IMCR (CR#859) uses FUNC2.
		*/

		pfe1rgmii_pins: pfe1rgmii_pins {
			/* TX outputs: PE_02=TX_CLK, PE_03=TX_EN, PE_04=TXD0,
			PE_05=TXD1, PE_06=TXD2 PE_07 (TXD3) */
			pfe1rgmii_grp0 {
				pinmux = <S32CC_PINMUX(66, FUNC2)>,   /* PE_02: PFE_MAC1_TX_CLK */
						<S32CC_PINMUX(67, FUNC2)>,   /* PE_03: PFE_MAC1_TX_EN  */
						<S32CC_PINMUX(68, FUNC2)>,   /* PE_04: PFE_MAC1_TXD0   */
						<S32CC_PINMUX(69, FUNC2)>,   /* PE_05: PFE_MAC1_TXD1   */
						<S32CC_PINMUX(70, FUNC2)>,   /* PE_06: PFE_MAC1_TXD2   */
						<S32CC_PINMUX(71, FUNC3)>;   /* PE_07: PFE_MAC1_TXD3 */
				output-enable;
				slew-rate = <S32CC_FAST_SLEW_166MHZ>;
			};

			/* RX inputs — pads set to FUNC0 (input mode); routing into PFE_MAC1
			is handled by the IMCR entries in pfe1rgmii_grp2 below.
			NXP input mux pattern: pad=FUNC0 + IMCR=FUNC2 */
			pfe1rgmii_grp1 {
				pinmux = <S32CC_PINMUX(72, FUNC0)>,   /* PE_08: input */
						<S32CC_PINMUX(73, FUNC0)>,   /* PE_09: input */
						<S32CC_PINMUX(74, FUNC0)>,   /* PE_10: input */
						<S32CC_PINMUX(75, FUNC0)>,   /* PE_11: input */
						<S32CC_PINMUX(76, FUNC0)>,   /* PE_12: input */
						<S32CC_PINMUX(77, FUNC0)>;   /* PE_13: input */
				input-enable;
				slew-rate = <S32CC_FAST_SLEW_166MHZ>;
			};

			/* IMCR input mux — selects which pad drives each PFE_MAC1 RX signal.
			CR#866 routes PE_02 (TX_CLK pad) back into PFE_MAC1_TX_CLK_I; required
			even for RGMII TX because the MAC samples its own TX_CLK internally.
			All entries at FUNC2 per S32G3 IOMUX spreadsheet. */
			pfe1rgmii_grp2 {
				pinmux = <S32CC_PINMUX(866, FUNC2)>,  /* CR#866: PFE_MAC1_TX_CLK_I ← PE_02 */
						<S32CC_PINMUX(859, FUNC2)>,  /* CR#859: PFE_MAC1_RX_CLK_I ← PE_08 */
						<S32CC_PINMUX(865, FUNC2)>,  /* CR#865: PFE_MAC1_RXDV_I   ← PE_09 */
						<S32CC_PINMUX(861, FUNC2)>,  /* CR#861: PFE_MAC1_RXD_I[0] ← PE_10 */
						<S32CC_PINMUX(862, FUNC2)>,  /* CR#862: PFE_MAC1_RXD_I[1] ← PE_11 */
						<S32CC_PINMUX(863, FUNC2)>,  /* CR#863: PFE_MAC1_RXD_I[2] ← PE_12 */
						<S32CC_PINMUX(864, FUNC2)>;  /* CR#864: PFE_MAC1_RXD_I[3] ← PE_13 */
			};
		};


Please let me know if you need any other information. 
 

0 Kudos
Reply