udp packets received are out-of-order on T1040RDB

[wengangzhu]

Hi,

   We are using the T1040RDB on our product. The kernel and dpaa driver are from yocto 2.5 sumo . On our system, the user apps will send/receive the packets via socket from dpaa ethernet device. But we found the packets received from Ethernet device are out of order for those connection-less flows. 

I saw some documents said that the 'Order Restoration' or 'Order Preservation' can help resolve it. But on our system there may be plenty flows,which can be ipv4, ipv6, unicast, mulicast, broadcast and some layer 2 protocol packets,  how can we config the dpaa driver to keep the packets in those flows in order.

 Attachment is the kernel config , the other one is some collection on my production. Hopeful, these are helpful .

Kernel version and CPU info:

root@E7-2:~# uname -a
Linux E7-2 4.14.47+gb09b730 #1 SMP Thu Jun 3 01:31:06 UTC 2021 ppc GNU/Linux
root@E7-2:~# cat /proc/cpuinfo
processor : 0
cpu : e5500
clock : 1200.000000MHz
revision : 2.1 (pvr 8024 1021)
bogomips : 62.50

processor : 1
cpu : e5500
clock : 1200.000000MHz
revision : 2.1 (pvr 8024 1021)
bogomips : 62.50

processor : 2
cpu : e5500
clock : 1200.000000MHz
revision : 2.1 (pvr 8024 1021)
bogomips : 62.50

processor : 3
cpu : e5500
clock : 1200.000000MHz
revision : 2.1 (pvr 8024 1021)
bogomips : 62.50

total bogomips : 250.00
timebase : 31250000
platform : CoreNet Calix
model : fsl,T1040RDB
Memory : 4087 MB

Iperf test result:

iperf3 -s -V

[ 5] local 192.168.6.18 port 5201 connected to 192.168.6.17 port 33294
Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 2.29 MBytes 19.2 Mbits/sec 0.013 ms 0/1656 (0%)
[ 5] 1.00-2.00 sec 2.38 MBytes 20.0 Mbits/sec 0.008 ms 0/1727 (0%)
[ 5] 2.00-3.00 sec 2.38 MBytes 20.0 Mbits/sec 0.024 ms 0/1726 (0%)
[ 5] 3.00-4.00 sec 2.38 MBytes 20.0 Mbits/sec 0.009 ms 0/1727 (0%)
[ 5] 4.00-5.00 sec 2.38 MBytes 20.0 Mbits/sec 0.008 ms 0/1726 (0%)
[ 5] 5.00-6.00 sec 2.38 MBytes 20.0 Mbits/sec 0.010 ms 0/1727 (0%)
[ 5] 6.00-7.00 sec 2.38 MBytes 20.0 Mbits/sec 0.009 ms 0/1727 (0%)
[ 5] 7.00-8.00 sec 2.38 MBytes 20.0 Mbits/sec 0.016 ms 0/1726 (0%)
[ 5] 8.00-9.00 sec 2.38 MBytes 20.0 Mbits/sec 0.008 ms 0/1727 (0%)
[ 5] 9.00-10.00 sec 2.38 MBytes 20.0 Mbits/sec 0.015 ms 0/1726 (0%)
[ 5] 10.00-10.04 sec 97.6 KBytes 19.9 Mbits/sec 0.009 ms 0/69 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] (sender statistics not available)
[SUM] 0.0-10.0 sec 134 datagrams received out-of-order
[ 5] 0.00-10.04 sec 23.8 MBytes 19.9 Mbits/sec 0.009 ms 0/17264 (0%) receiver
CPU Utilization: local/receiver 0.7% (0.0%u/0.7%s), remote/sender 3.1% (0.1%u/2.9%s)
iperf3: the client has unexpectedly closed the connection
iperf 3.2
Linux eolus 4.14.47+gb09b730 #1 SMP Thu Jun 3 01:31:06 UTC 2021 ppc
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

root@eolus:~# iperf3 -c 192.168.6.18 -u -b 20M
Connecting to host 192.168.6.18, port 5201
[ 5] local 192.168.6.17 port 33294 connected to 192.168.6.18 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 2.38 MBytes 20.0 Mbits/sec 1725
[ 5] 1.00-2.00 sec 2.38 MBytes 20.0 Mbits/sec 1727
[ 5] 2.00-3.00 sec 2.38 MBytes 20.0 Mbits/sec 1727
[ 5] 3.00-4.00 sec 2.38 MBytes 20.0 Mbits/sec 1726
[ 5] 4.00-5.00 sec 2.38 MBytes 20.0 Mbits/sec 1727
[ 5] 5.00-6.00 sec 2.38 MBytes 20.0 Mbits/sec 1726
[ 5] 6.00-7.00 sec 2.38 MBytes 20.0 Mbits/sec 1727
[ 5] 7.00-8.00 sec 2.38 MBytes 20.0 Mbits/sec 1726
[ 5] 8.00-9.00 sec 2.38 MBytes 20.0 Mbits/sec 1727
[ 5] 9.00-10.00 sec 2.38 MBytes 20.0 Mbits/sec 1726
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 23.8 MBytes 20.0 Mbits/sec 0.000 ms 0/17264 (0%) sender
[ 5] 0.00-10.04 sec 23.8 MBytes 19.9 Mbits/sec 0.009 ms 0/17264 (0%) receiver

Thanks

[wengangzhu]

Hi yiping,

  Since only one of policy_ipv4.xml and policy_ipv6.xml can be used on the system, if we need support dual stacks (ipv4 and ipv6), how can we do?

[yipingwang]

You need to create your own policy file including ipv4 and ipv6.

[wengangzhu]

Hi yiping,
      If one of 4 CPUs is isolated from the general SMP balancing and scheduler algorithms with isolcpus parameter, is this isolated cpu core still used by core-affined queue for implementing order preservation.

Thanks

-sky

[yipingwang]

Unlike TCP, UDP is an unreliable high level protocol, we also encountered out-of-order issue in iperf UDP testing on T1040RDB.

You could use core-affined queue to implement order preservation, please use one of the following default FMC policy files.
/etc/fmc/config/private/t1040d4rdb/RR_P_66/policy_ipv4.xml
/etc/fmc/config/private/t1040d4rdb/RR_P_66/policy_ipv6.xml

Please execute FMC commands as the following before all networking related operation.
# fmc -c /etc/fmc/config/private/t1040d4rdb/RR_P_66/config.xml -p /etc/fmc/config/private/t1040d4rdb/RR_P_66/policy_ipv4.xml -a

This is because in DPAA1, each interface used by default one pool channel across all software portals and also the dedicated channel of each CPU. In Linux Kernel, PCD Frame queues in use dedicated channels. You could refer to
the section "5. Dedicated and Pool Channels Usage in Linux Kernel" in Using QMAN Dedicated and Pool Channels in USDPAA and Linux Kernel .

You need to use multiple flows, after executing FMC policy, one flow will bind to one core, so all 4 cores will be used by multiple flows. In the real scenario, one user application uses one flow.

[wengangzhu]

Hi yiping,

     Thank for your kindly reply.

Another question is that where  I can download those FMC policy files and the FMC code, so that I port them on my product ?

Thanks

[wengangzhu]

BTW:

Below is the network devices on our system.

root@eolus:~# ifconfig -a
es0 Link encap:Ethernet HWaddr 00:02:86:ff:00:fd
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

eth0 Link encap:Ethernet HWaddr 00:02:86:ff:00:f0
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Memory:fe4e6000-fe4e6fff

eth2 Link encap:Ethernet HWaddr 00:22:44:66:88:52
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Memory:fe4e2000-fe4e2fff

eth5 Link encap:Ethernet HWaddr 00:22:44:66:88:50
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Memory:fe4e4000-fe4e4fff

fm1-gb0 Link encap:Ethernet HWaddr 00:22:44:66:88:51
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Memory:fe4e0000-fe4e0fff

lo Link encap:Local Loopback
LOOPBACK MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

sit0 Link encap:UNSPEC HWaddr 00-00-00-00-49-B8-10-11-00-00-00-00-00-00-00-00
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

teql0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

tunl0 Link encap:IPIP Tunnel HWaddr
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

[yipingwang]

Please only execute fmc command once after boot up Linux OS.

Would you please provide your Linux Kernel uImage and dtb file? I will do verification on my T1040RDB demo board.

[wengangzhu]

Hi yiping,

     The hw on our board has a big different with the demo board. So I am afraid the uImage can't boot up on the demo card.

Today, I was porting the fmc , fmlib and eth-config from QorIQ-SDK-V2.0-20160527-yocto to our system, then the fmc can be running on our system without error. But after that , I did the iperf test, the out-of-order issue still existed.

Is that any other debug log I can enabled for further debug?

How can I check the compatible between the fmc, policy profiles and our system(fm ucode, kernel dpaa related version, and other configurations) ?

Attachments are the log when I run the fmc and the test reset on iperf server and client.

[wengangzhu]

Hi yiping,

   I deleted the mac 5 on config.xml. It works.

<cfgdata>
<config>
<engine name="fm0">
<port type="MAC" number="1" policy="fm_policy_1"/>
<port type="MAC" number="2" policy="fm_policy_2"/>
<port type="MAC" number="3" policy="fm_policy_3"/>
<port type="MAC" number="4" policy="fm_policy_4"/>
</engine>
</config>
</cfgdata>

Another question is that what the impact to throughput, latency and stability is after using core-affined queue ?

Thanks

[wengangzhu]

Hi yiping,

   On our board , the ucode version is 106.4.10 . And on 'QorIQ-SDK-V2.0-20160527', the ucode version is 106.4.18. 

Because our product has been deployed on customer for long time, so it is impossible to upgrade ucode version to '106.4.18' currently.

If we use the mfc and config policy from 'QorIQ-SDK-V2.0-20160527',  is that compatible for our board?

If not, which QorIQ SDK contains the correct fmc, fmlib, policy profile version compatible with our uc code '106.4.10'?

Is there any method to confirm whether all the configuration applied to DPAA is correct?

Thanks

[wengangzhu]

Hi yiping,

  I compiled the fmc and download the fmc and policy config files to our product board.

root@eolus:~# fmc --version

fmc version: 0.9.34

root@eolus:~# ls /etc/fmc/ -R
/etc/fmc/:
config

/etc/fmc/config:
hxs_pdl_v3.xml private

/etc/fmc/config/private:
t1040d4rdb

/etc/fmc/config/private/t1040d4rdb:
RR_P_66

/etc/fmc/config/private/t1040d4rdb/RR_P_66:
config.xml policy_ipv4.xml policy_ipv6.xml

root@eolus:~# uname -a
Linux E7-2 4.14.47+gb09b730 #1 SMP Mon Jun 7 09:01:49 UTC 2021 ppc GNU/Linux

Is this fmc version and policy files are compatible with our linux kernel & dpaa driver?

===================================================================

The kernel OOPs when I run the below command:

root@eolus:~# fmc -c /etc/fmc/config/private/t1040d4rdb/RR_P_66/config.xml -p /etc/fmc/config/private/t1040d4rdb/RR_P_66/policy_ipv4.xml -a
[ 252.095989] cpu1/8: ! MAJOR FM-PCD Error [CPU01, /usr/src/kernel/drivers/net/ethernet/freescale/sdk_fman/Peripherals/FM/Pcd/fm_kg.c:1061 BuildSchemeRegs]: Invalid State;
[ 252.095995] cpu1/8: Next Engine of the scheme is not Valid
[ 252.111291] cpu1/8:
[ 252.119080] cpu1/8: ! MAJOR FM-PCD Error [CPU01, /usr/src/kernel/drivers/net/ethernet/freescale/sdk_fman/Peripherals/FM/Pcd/fm_kg.c:3020 FM_PCD_KgSchemeSet]: Invalid State;
[ 252.119083] cpu1/8:
[ 252.134596] cpu1/8:
[ 252.139070] cpu1/8: ! MINOR FM Error [CPU01, /usr/src/kernel/drivers/net/ethernet/freescale/sdk_fman/src/wrapper/lnxwrp_ioctls_fm.c:3444 LnxwrpFmPcdIOCTL]: Invalid Value;
[ 252.139073] cpu1/8: IOCTL FM PCD
[ 252.154414] cpu1/8:
ERR : [ 252.160026] cpu1/8: *** ASSERT_COND failed [CPU01, /usr/src/kernel/drivers/net/ethernet/freescale/sdk_fman/Peripherals/FM/Pcd/fm_pcd.c:770 FmPcdGetHcHandle]
Invocation of FM_PCD_KgSchemeSet for fm0/port/MAC/1/dist/udpeth[ 252.174632]
[ 252.174632]
[ 252.174632] FMD: fatal error, driver can't go on!!!
[ 252.174632]
1 failed
[ 252.189516] ------------[ cut here ]------------
[ 252.194904] WARNING: CPU: 1 PID: 4855 at /usr/src/kernel/drivers/net/ethernet/freescale/sdk_fman/src/xx/xx_linux.c:161 XX_Exit+0x24/0x40
[ 252.207186] Modules linked in: ramoops reed_solomon pramfs i2c_dev caam_jr error(P) m25p80 spidev linux_kernel_bde(O) enet_sim caam fta(O) adt7470 ltc4261 sn65lvcp114(O) lm90 pmbus rtc_pcf8583 ti_adc081c industrialio_triggered_buffer kfifo_buf industrialio zl6100 pmbus_core sc16is7xx at24 lm75 hwmon si5345(O) spi_fsl_espi
[ 252.235804] CPU: 1 PID: 4855 Comm: fmc Tainted: P O 4.14.47+gb09b730 #1
[ 252.243560] task: e91ca100 task.stack: e8732000
[ 252.248095] **bleep**: c06a6ae4 LR: c06a6ae4 CTR: 00000000
[ 252.253151] REGS: e8733c40 TRAP: 0700 Tainted: P O (4.14.47+gb09b730)
[ 252.260993] MSR: 00029002 <CE,EE,ME> CR: 24000422 XER: 00000000
[ 252.267190]
[ 252.267190] GPR00: c06a6ae4 e8733cf0 e91ca100 0000002a 00021002 0000026f c0d40000 00021002
[ 252.267190] GPR08: 2ceee000 c0cbe9a8 c0cbe9a8 ffffffff 24000824 10178358 bfe2628c 144dc780
[ 252.267190] GPR16: bfe26188 144dcb40 144dcb00 bfe26450 bfe26558 12304768 1016b3a8 123a4768
[ 252.267190] GPR24: 123aa460 000149b9 000526e4 10123da0 10123da0 10123da0 e957bab0 00000000
[ 252.302156] **bleep** [c06a6ae4] XX_Exit+0x24/0x40
[ 252.306430] LR [c06a6ae4] XX_Exit+0x24/0x40
[ 252.310616] Call Trace:
[ 252.313062] [e8733cf0] [c06a6ae4] XX_Exit+0x24/0x40 (unreliable)
[ 252.319084] [e8733d00] [c062722c] FmPcdGetHcHandle+0x6c/0x90
[ 252.324753] [e8733d10] [c0647494] DetachPCD+0x74/0x1e0
[ 252.329904] [e8733d30] [c0659c38] FM_PORT_DeletePCD+0x158/0x560
[ 252.335839] [e8733d50] [c0689848] LnxwrpFmPortIOCTL+0xa8/0x1ec0
[ 252.341770] [e8733ea0] [c068b6d8] fm_ioctl+0x78/0x1b0
[ 252.346831] [e8733ec0] [c024b988] do_vfs_ioctl+0xc8/0x940
[ 252.352240] [e8733f20] [c024c25c] SyS_ioctl+0x5c/0x110
[ 252.357390] [e8733f40] [c0013278] ret_from_syscall+0x0/0x3c
[ 252.362973] --- interrupt: c01 at 0xfee26bc
[ 252.362973] LR = 0xfee25c8
[ 252.370206] Instruction dump:
[ 252.373175] 4bb7533c 60000000 60000000 7c0802a6 90010004 60000000 9421fff0 7c0802a6
[ 252.380940] 90010014 3c60c0bb 38635344 4ba09edd <0fe00000> 80010014 38210010 7c0803a6
[ 252.388880] ---[ end trace 91ebd89f9939f643 ]---
[ 252.393537] Unable to handle kernel paging request for data at address 0x00004b90
[ 252.401041] Faulting instruction address: 0xc0627230
[ 252.406025] Oops: Kernel access of bad area, sig: 11 [#1]
[ 252.411430] BE SMP NR_CPUS=8 CoreNet Calix
[ 252.415531] Modules linked in: ramoops reed_solomon pramfs i2c_dev caam_jr error(P) m25p80 spidev linux_kernel_bde(O) enet_sim caam fta(O) adt7470 ltc4261 sn65lvcp114(O) lm90 pmbus rtc_pcf8583 ti_adc081c industrialio_triggered_buffer kfifo_buf industrialio zl6100 pmbus_core sc16is7xx at24 lm75 hwmon si5345(O) spi_fsl_espi
[ 252.444135] CPU: 1 PID: 4855 Comm: fmc Tainted: P W O 4.14.47+gb09b730 #1
[ 252.451889] task: e91ca100 task.stack: e8732000
[ 252.456422] **bleep**: c0627230 LR: c062722c CTR: 00000000
[ 252.461479] REGS: e8733c50 TRAP: 0300 Tainted: P W O (4.14.47+gb09b730)
[ 252.469320] MSR: 00029002 <CE,EE,ME> CR: 24000422 XER: 00000000
[ 252.475516] DEAR: 00004b90 ESR: 00000000
[ 252.475516] GPR00: c0647494 e8733d00 e91ca100 0000002a 00021002 0000026f c0d40000 00021002
[ 252.475516] GPR08: 2ceee000 c0cbe9a8 c0cbe9a8 ffffffff 24000824 10178358 bfe2628c 144dc780
[ 252.475516] GPR16: bfe26188 144dcb40 144dcb00 bfe26450 bfe26558 12304768 1016b3a8 123a4768
[ 252.475516] GPR24: 123aa460 000149b9 000526e4 10123da0 10123da0 10123da0 e957bab0 00000000
[ 252.513014] **bleep** [c0627230] FmPcdGetHcHandle+0x70/0x90
[ 252.518071] LR [c062722c] FmPcdGetHcHandle+0x6c/0x90
[ 252.523039] Call Trace:
[ 252.525486] [e8733d00] [c062722c] FmPcdGetHcHandle+0x6c/0x90 (unreliable)
[ 252.532286] [e8733d10] [c0647494] DetachPCD+0x74/0x1e0
[ 252.537434] [e8733d30] [c0659c38] FM_PORT_DeletePCD+0x158/0x560
[ 252.543365] [e8733d50] [c0689848] LnxwrpFmPortIOCTL+0xa8/0x1ec0
[ 252.549296] [e8733ea0] [c068b6d8] fm_ioctl+0x78/0x1b0
[ 252.554355] [e8733ec0] [c024b988] do_vfs_ioctl+0xc8/0x940
[ 252.559763] [e8733f20] [c024c25c] SyS_ioctl+0x5c/0x110
[ 252.564911] [e8733f40] [c0013278] ret_from_syscall+0x0/0x3c
[ 252.570492] --- interrupt: c01 at 0xfee26bc
[ 252.570492] LR = 0xfee25c8
[ 252.577724] Instruction dump:
[ 252.580692] 54290024 3ca0c0bb 80890004 3c60c0bb 38e70200 38a5c924 38c00302 3863875c
[ 252.588457] 4807f8e1 38600001 4807f899 80010014 <807f4b90> 7c0803a6 83e1000c 38210010
[ 252.596399] ---[ end trace 91ebd89f9939f644 ]---
[ 252.608200]
[ 253.601726] Kernel panic - not syncing: Fatal exception
[ 253.614200] Rebooting in 2 seconds..
[ 255.601799] ------------[ cut here ]------------
[ 255.606429] WARNING: CPU: 1 PID: 4855 at /usr/src/kernel/kernel/smp.c:416 smp_call_function_many+0x120/0x3d0
[ 255.616273] Modules linked in: ramoops reed_solomon pramfs i2c_dev caam_jr error(P) m25p80 spidev linux_kernel_bde(O) enet_sim caam fta(O) adt7470 ltc4261 sn65lvcp114(O) lm90 pmbus rtc_pcf8583 ti_adc081c industrialio_triggered_buffer kfifo_buf industrialio zl6100 pmbus_core sc16is7xx at24 lm75 hwmon si5345(O) spi_fsl_espi
[ 255.644881] CPU: 1 PID: 4855 Comm: fmc Tainted: P D W O 4.14.47+gb09b730 #1
[ 255.652637] task: e91ca100 task.stack: e8732000
[ 255.657172] **bleep**: c00f2cc0 LR: c00f2fa8 CTR: c0049780
[ 255.662228] REGS: e8733aa0 TRAP: 0700 Tainted: P D W O (4.14.47+gb09b730)
[ 255.670069] MSR: 00021002 <CE,ME> CR: 24002422 XER: 00000000
[ 255.676002]
[ 255.676002] GPR00: c00f2fa8 e8733b50 e91ca100 c0d471c8 c0014190 00000000 00000000 00021002
[ 255.676002] GPR08: 2ceee000 00000000 00000001 ffffffff 24002422 10178358 bfe2628c 144dc780
[ 255.676002] GPR16: bfe26188 144dcb40 144dcb00 bfe26450 bfe26558 12304768 00000000 c0d40000
[ 255.676002] GPR24: 123aa460 00000000 c0014190 00000000 c0d471c8 c0d471c8 c0d6080c 00000001
[ 255.710972] **bleep** [c00f2cc0] smp_call_function_many+0x120/0x3d0
[ 255.716726] LR [c00f2fa8] smp_call_function+0x38/0x50
[ 255.721782] Call Trace:
[ 255.724228] [e8733b50] [12304768] 0x12304768 (unreliable)
[ 255.729637] [e8733b90] [c00f2fa8] smp_call_function+0x38/0x50
[ 255.735393] [e8733ba0] [c000fc9c] machine_restart+0x5c/0x90
[ 255.740975] [e8733bc0] [c004a18c] panic+0x228/0x28c
[ 255.745865] [e8733c20] [c000d114] oops_end+0x194/0x1a0
[ 255.751014] [e8733c40] [c0013794] handle_page_fault+0x7c/0x80
[ 255.756774] --- interrupt: 300 at FmPcdGetHcHandle+0x70/0x90
[ 255.756774] LR = FmPcdGetHcHandle+0x6c/0x90
[ 255.766970] [e8733d10] [c0647494] DetachPCD+0x74/0x1e0
[ 255.772120] [e8733d30] [c0659c38] FM_PORT_DeletePCD+0x158/0x560
[ 255.778056] [e8733d50] [c0689848] LnxwrpFmPortIOCTL+0xa8/0x1ec0
[ 255.783987] [e8733ea0] [c068b6d8] fm_ioctl+0x78/0x1b0
[ 255.789047] [e8733ec0] [c024b988] do_vfs_ioctl+0xc8/0x940
[ 255.794456] [e8733f20] [c024c25c] SyS_ioctl+0x5c/0x110
[ 255.799603] [e8733f40] [c0013278] ret_from_syscall+0x0/0x3c
[ 255.805185] --- interrupt: c01 at 0xfee26bc
[ 255.805185] LR = 0xfee25c8
[ 255.812417] Instruction dump:
[ 255.815385] 83c10038 83e1003c 38210040 4e800020 3d20c0d6 8129c1d0 2f890000 40deff88
[ 255.823150] 3d20c0d4 89297084 2f890000 40deff78 <0fe00000> 3860ffff 7fa5eb78 7f84e378
[ 255.831090] ---[ end trace 91ebd89f9939f645 ]---

=============================================================

[wengangzhu]

Hi yiping,

   Any update for the oops. I also attached the oops log with '-l dbg3'.

Thanks