ssh failed, ssh can't send tcp package to PC

OutofSoul · ‎04-12-2023

environment

soc: s32g399ardb3
net device: aux0sl (pfeslave)

phenomenon description

I ssh the PC(ubuntu) from embedded Linux. But after authentication, it can't start shell.

Through the `tcpdump` log, i found the tcp data pkgs (include ack pkgs) can't be sent to PC, after authenticaion.

But before authentication, the data pkgs can be sent to PC from s32g, the reverse is also true.

log

net device infomation

# after boot kernel, aux0sl isn't up, i have to up it manually
root@s32g399ardb3:~# ifconfig aux0sl 192.168.123.120
root@s32g399ardb3:~# ifconfig
aux0sl: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.123.120 netmask 255.255.255.0 broadcast 192.168.123.255
inet6 fe80::204:9fff:febe:ff80 prefixlen 64 scopeid 0x20<link>
ether 00:04:9f:be:ff:80 txqueuelen 1000 (Ethernet)
RX packets 7811 bytes 524882 (512.5 KiB)
RX errors 0 dropped 7376 overruns 0 frame 0
TX packets 19 bytes 1532 (1.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0x46000000-46ffffff

the ssh log

root@s32g399ardb3:~# ssh -v cg@192.168.123.111
OpenSSH_8.3p1, OpenSSL 1.1.1k 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 192.168.123.111 [192.168.123.111] port 22.
debug1: Connection established.
...
...
...
debug1: Local version string SSH-2.0-OpenSSH_8.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.7
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.7 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to 192.168.123.111:22 as 'cg'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
...
...
...
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:EZwQIrHaRi521tPos2AwwtISifFnRSWkJUMaa4fMxUU
debug1: Host '192.168.123.111' is known and matches the ECDSA host key.
debug1: Found key in /home/root/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
...
...
...
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
...
...
...
debug1: Next authentication method: password
cg@192.168.123.111's password:
debug1: Authentication succeeded (password).
Authenticated to 192.168.123.111 ([192.168.123.111]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: exec
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: X11 forwarding requested but DISPLAY not set

tcpdump log

root@s32g399ardb3:~# tcpdump -i aux0sl -n host 192.168.123.111 &
root@s32g399ardb3:~# ssh cg@192.168.123.111
17:26:35.072053 ARP, Request who-has 192.168.123.111 tell 192.168.123.1, length 46
17:26:39.961413 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [S], seq 326276919, win 64240, options [mss 1460,sackOK,TS val 3645630678 ecr 0,nop,wscale 7], length 0
17:26:39.962103 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [S.], seq 1923967336, ack 326276920, win 65160, options [mss 1460,sackOK,TS val 3682564497 ecr 3645630678,nop,wscale 7], length 0
17:26:39.962180 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [.], ack 1, win 502, options [nop,nop,TS val 3645630679 ecr 3682564497], length 0
17:26:39.963144 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1:22, ack 1, win 502, options [nop,nop,TS val 3645630680 ecr 3682564497], length 21
17:26:39.963537 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [.], ack 22, win 509, options [nop,nop,TS val 3682564498 ecr 3645630680], length 0
17:26:39.965709 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [P.], seq 1:42, ack 22, win 509, options [nop,nop,TS val 3682564500 ecr 3645630680], length 41
17:26:39.965731 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [.], ack 42, win 502, options [nop,nop,TS val 3645630682 ecr 3682564500], length 0
17:26:39.966446 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [P.], seq 42:1122, ack 22, win 509, options [nop,nop,TS val 3682564501 ecr 3645630682], length 1080
17:26:39.966465 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [.], ack 1122, win 501, options [nop,nop,TS val 3645630683 ecr 3682564501], length 0
17:26:39.967380 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [.], seq 22:1470, ack 1122, win 501, options [nop,nop,TS val 3645630684 ecr 3682564501], length 1448
17:26:39.967398 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1470:1534, ack 1122, win 501, options [nop,nop,TS val 3645630684 ecr 3682564501], length 64
17:26:39.968045 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [.], ack 1534, win 499, options [nop,nop,TS val 3682564503 ecr 3645630684], length 0
17:26:39.983024 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1534:1582, ack 1122, win 501, options [nop,nop,TS val 3645630700 ecr 3682564503], length 48
17:26:39.983527 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [.], ack 1582, win 501, options [nop,nop,TS val 3682564518 ecr 3645630700], length 0
17:26:39.986271 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [P.], seq 1122:1574, ack 1582, win 501, options [nop,nop,TS val 3682564521 ecr 3645630700], length 452
17:26:39.986288 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [.], ack 1574, win 501, options [nop,nop,TS val 3645630703 ecr 3682564521], length 0

17:26:40.003957 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1582:1598, ack 1574, win 501, options [nop,nop,TS val 3645630720 ecr 3682564521], length 16
17:26:40.004534 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [.], ack 1598, win 501, options [nop,nop,TS val 3682564539 ecr 3645630720], length 0
17:26:40.004556 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1598:1642, ack 1574, win 501, options [nop,nop,TS val 3645630721 ecr 3682564539], length 44
17:26:40.004886 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [.], ack 1642, win 501, options [nop,nop,TS val 3682564540 ecr 3645630721], length 0
17:26:40.004977 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [P.], seq 1574:1618, ack 1642, win 501, options [nop,nop,TS val 3682564540 ecr 3645630721], length 44
17:26:40.004994 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [.], ack 1618, win 501, options [nop,nop,TS val 3645630722 ecr 3682564540], length 0
17:26:40.005062 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1642:1702, ack 1618, win 501, options [nop,nop,TS val 3645630722 ecr 3682564540], length 60
17:26:40.005346 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [.], ack 1702, win 501, options [nop,nop,TS val 3682564540 ecr 3645630722], length 0
17:26:40.005712 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [P.], seq 1618:1670, ack 1702, win 501, options [nop,nop,TS val 3682564540 ecr 3645630722], length 52
17:26:40.052963 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [.], ack 1670, win 501, options [nop,nop,TS val 3645630770 ecr 3682564540], length 0

cg@192.168.123.111's password:
17:26:42.785073 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1786:1870, ack 1722, win 501, options [nop,nop,TS val 3645633502 ecr 3682564888], length 84
17:26:42.788922 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [P.], seq 1722:1750, ack 1870, win 501, options [nop,nop,TS val 3682567322 ecr 3645633502], length 28
17:26:42.788944 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [.], ack 1750, win 501, options [nop,nop,TS val 3645633505 ecr 3682567322], length 0
17:26:42.837026 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1870:1982, ack 1750, win 501, options [nop,nop,TS val 3645633554 ecr 3682567322], length 112
17:26:42.871872 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [P.], seq 1750:2250, ack 1982, win 501, options [nop,nop,TS val 3682567405 ecr 3645633554], length 500
17:26:42.871893 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [.], ack 2250, win 501, options [nop,nop,TS val 3645633588 ecr 3682567405], length 0
17:26:42.872232 IP 192.168.123.111.22 > 192.168.123.120.34874: Flags [P.], seq 2250:2294, ack 1982, win 501, options [nop,nop,TS val 3682567405 ecr 3645633588], length 44
# after here, data pkgs can't be sent out
17:26:42.872251 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [.], ack 2294, win 501, options [nop,nop,TS val 3645633589 ecr 3682567405], length 0
17:26:42.872403 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1982:2358, ack 2294, win 501, options [nop,nop,TS val 3645633589 ecr 3682567405], length 376
17:26:43.080966 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1982:2358, ack 2294, win 501, options [nop,nop,TS val 3645633798 ecr 3682567405], length 376
17:26:43.292969 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1982:2358, ack 2294, win 501, options [nop,nop,TS val 3645634010 ecr 3682567405], length 376
17:26:43.720966 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1982:2358, ack 2294, win 501, options [nop,nop,TS val 3645634438 ecr 3682567405], length 376
17:26:44.584990 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1982:2358, ack 2294, win 501, options [nop,nop,TS val 3645635302 ecr 3682567405], length 376
17:26:46.276975 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1982:2358, ack 2294, win 501, options [nop,nop,TS val 3645636994 ecr 3682567405], length 376
17:26:47.071840 ARP, Request who-has 192.168.123.111 tell 192.168.123.1, length 46
17:26:49.672970 IP 192.168.123.120.34874 > 192.168.123.111.22: Flags [P.], seq 1982:2358, ack 2294, win 501, options [nop,nop,TS val 3645640390 ecr 3682567405], length 376

the successful log

I tired use eth0(gmac) instead of aux0sl(pfeslave), and it can ssh PC successfully.

What causes the difference between these two situations？

root@s32g399ardb3:~# ssh -v cg@192.168.123.111
OpenSSH_8.3p1, OpenSSL 1.1.1k 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 192.168.123.111 [192.168.123.111] port 22.
debug1: Connection established.
debug1: identity file /home/root/.ssh/id_rsa type -1
debug1: identity file /home/root/.ssh/id_rsa-cert type -1
debug1: identity file /home/root/.ssh/id_dsa type -1
debug1: identity file /home/root/.ssh/id_dsa-cert type -1
debug1: identity file /home/root/.ssh/id_ecdsa type -1
debug1: identity file /home/root/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/root/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/root/.ssh/id_ed25519 type -1
debug1: identity file /home/root/.ssh/id_ed25519-cert type -1
debug1: identity file /home/root/.ssh/id_ed25519_sk type -1
debug1: identity file /home/root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/root/.ssh/id_xmss type -1
debug1: identity file /home/root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.7
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.7 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to 192.168.123.111:22 as 'cg'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:EZwQIrHaRi521tPos2AwwtISifFnRSWkJUMaa4fMxUU
debug1: Host '192.168.123.111' is known and matches the ECDSA host key.
debug1: Found key in /home/root/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/root/.ssh/id_rsa
debug1: Will attempt key: /home/root/.ssh/id_dsa
debug1: Will attempt key: /home/root/.ssh/id_ecdsa
debug1: Will attempt key: /home/root/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/root/.ssh/id_ed25519
debug1: Will attempt key: /home/root/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/root/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/root/.ssh/id_rsa
debug1: Trying private key: /home/root/.ssh/id_dsa
debug1: Trying private key: /home/root/.ssh/id_ecdsa
debug1: Trying private key: /home/root/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/root/.ssh/id_ed25519
debug1: Trying private key: /home/root/.ssh/id_ed25519_sk
debug1: Trying private key: /home/root/.ssh/id_xmss
debug1: Next authentication method: password
cg@192.168.123.111's password:
debug1: Authentication succeeded (password).
Authenticated to 192.168.123.111 ([192.168.123.111]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: exec
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: X11 forwarding requested but DISPLAY not set
Welcome to Kylin V10 SP1 (GNU/Linux 5.4.0-144-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

56 updates can be applied immediately.
21 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable

Last login: Tue Apr 11 16:54:41 2023 from 192.168.123.120
cg@laptop:~$

MayanksPatel · ‎04-24-2023

Hi @OutofSoul,

Thanks to inform us. We are closing this issue.

Thanks,

Mayank s Patel

MayanksPatel · ‎04-14-2023

Hi @OutofSoul,

Thank you for reaching out to NXP.

Can you please share tcp/ip stack and PFE Firmware version and Linux BSP version which you are using?

Can you please describe more about the setup?

Thanks,

Mayank s Patel

OutofSoul · ‎04-24-2023

Hello, sorry for taking so long to reply.

Because NXP's techinical support personnel provided a solution brefore, i haven't followed the forum since then.

This is an issue with the TX queue of PFE.