Does Linux kernel 3.0.4 support use of PCIe AER driver with P2020 or any powerpc based architecture?

[rkj]

I am having trouble getting the Linux kernel PCIe AER driver working with a P2020 based system.  A summary of my issues and observations can be found in a recent post to the linux-pci mail list but so far I have not received any responses:

http://thread.gmane.org/gmane.linux.kernel.pci/33564

I would think info for any powerpc based architecture would be useful.  Any information and/or suggestions would be greatly appreciated.

[yipingwang]

Hello Johnson,

PCIe AER feature has already been included since SDK 1.3(Kernel 3.0.48).

The driver source file exists in drivers/pci/pcie/aer/*.c.

Please pay attention to the following when boot up the system.

1. Please make sure CONFIG_PCIEAER and CONFIG_PCIEAER_INJECT are enabled in the Kernel configuration file.

2. In u-boot, please add the following in bootargs.

=> setenv othbootargs pcie_ports=native

3.Check if the aer injector device is created.

# ls /dev/aer_inject

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[rkj]

Do do you think that the Linux AER driver may not have been fully functional for P2020 (powerpc) in 3.0.4 kernel but may have been functional in later kernels like 3.0.48?

I checked my 3.0.4 kernel config and both params were already defined:

CONFIG_PCIEAER=y

CONFIG_PCIEAER_INJECT=y

And the aer_inject device was present in my system:

# ls -l /dev/a*

crw------- 1 root root 10, 63 Jul 17 15:48 /dev/aer_inject

Here are some things I have already tried from U-boot for the linux cmdline:

pcie_ports=native aerdriver.forceload=y

The posting to the linux-pci mail list I mentioned previously shows all of the other indications I see that things are partially working but not fully.

[yipingwang]

Hello Johnson,

Verifying PCIe AER function in actual environment is quite difficult, because it is hard to trigger real hardware errors. The testing team use a software tool to fake various kinds of PCIe errors.

Please refer to the following procedure.

1. Download aer-inject test utility.

Write a test config file

e.g. $ vi aer-cfg

      AER

      DOMAIN 0001

      BUS 1

      DEV 0

      FN 0

      COR_STATUS BAD_TLP

      HEADER_LOG 0 1 2 3

NOTE:

error type can be ["COR_STATUS", "UNCOR_STATUS"]

Corrected error can be:

["BAD_TLP", "RCVR", "BAD_DLLP", "REP_ROLL", "REP_TIMER"]

Uncorrected non-fatal error can be:

["POISON_TLP", "COMP_TIME", "COMP_ABORT", "UNX_COMP", "ECRC", "UNSUP"]

Uncorrected fatal error can be:

["TRAIN", "DLP", "FCP", "RX_OVER", "MALF_TLP"]

2. # lspci

00:00.0 Class 0604: 1957:0410

01:00.0 Class 0200: 8086:10d3

e.g. here device "01:00.0" is the PCIe NIC e1000 network card in the test scenario.

3. Download aer-inject and aer-cfg from host to test-board

$ scp aer-inject aer-cfg root@test-board-ip:~

4. ensure the pcie device domain-number/bus-number/device-number/function-number in aer-cfg

is accordant to those in the output of lspci

5. Run aer-inject, corresponding error information will be reported as below and AER will

recover PCIE device according to the type of errors.

# ./aer-inject aer-cfg

example of error report as below:

pcieport 0000:00:00.0: AER: Corrected error received: id=0100

e1000e 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer,

id=0100(Receiver ID)

e1000e 0000:01:00.0: device [8086:10d3] error status/mask=00000040/00002000

e1000e 0000:01:00.0: [ 6] Bad TLP

root@p1010rdb:~#

6. The pcie device(e1000e PCIE NIC) should still work after AER error recovery.

# ping 192.168.1.1 -c 2 -s 64

PING 192.168.1.1 (192.168.1.1): 64 data bytes

72 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.272 ms

72 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.210 ms

--- 192.168.1.1 ping statistics ---

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max/stddev = 0.210/0.241/0.272/0.031 ms

Additionally, I have attached SDK 1.3 AER driver code as the above, you could compare it with yours.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[rkj]

I am not able to get aer-inject to work on my system.

I found an early version of aer-inject (0.1), but it does not support things like "DOMAIN" key word.  This link to the kernel site is empty:

https://www.kernel.org/pub/linux/utils/pci/aer-inject/

Is there somewhere else I can download a more recent version of aer-inject?

I tried testing with a GbE PCIe card (see lspci output down below) but I get error messages at the command line.:

# ./aer-inject test/aer2

Error: Failed to write, Invalid argument

Here is the contents of aer2:

# more test/aer2

AER

BUS 0x13 DEV 0 FN 0

COR_STATUS BAD_TLP

HEADER_LOG 0 1 2 3

The GbE PCIe card is functional in the system.

The right params are configured in the kernel:

$ grep AER .config

CONFIG_PCIEAER=y

CONFIG_PCIEAER_INJECT=y

The Linux cmdline shows pcie_ports=native set properly:

# cat /proc/cmdline 

console=ttyS1,115200 root=/dev/nfs rw nfsroot=192.168.8.120:/opt/xes/XPedite5570/sn07112008 ip=192.168.8.97:192.168.8.120:192.168.8.254:255.255.255.0:mm:bp0:off pcie_ports=native

lspci output for GbE device:

13:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)

        Subsystem: Device 7470:3468

        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 32 bytes

        Interrupt: pin A routed to IRQ 62

        Region 0: I/O ports at 1000 [size=256]

        Region 2: Memory at f80500000 (64-bit, non-prefetchable) [size=4K]

        Region 4: Memory at f80504000 (64-bit, prefetchable) [size=16K]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)

Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+

Address: 00000000fff41740  Data: 0014

Capabilities: [70] Express (v2) Endpoint, MSI 01

DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-

DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-

MaxPayload 128 bytes, MaxReadReq 4096 bytes

DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-

LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <64us

ClockPM+ Surprise- LLActRep- BwNot-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Not Supported, TimeoutDis+

DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-

LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB

Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

Compliance De-emphasis: -6dB

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

Capabilities: [b0] MSI-X: Enable- Count=4 Masked-

Vector table: BAR=4 offset=00000000

PBA: BAR=4 offset=00000800

Capabilities: [d0] Vital Product Data

Unknown small resource type 05, will not decode more.

Capabilities: [100 v1] Advanced Error Reporting

UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-

CESta:  RxErr- BadTLP+ BadDLLP- Rollover- Timeout- NonFatalErr-

CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

Capabilities: [140 v1] Virtual Channel

Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1

Arb:    Fixed- WRR32- WRR64- WRR128-

Ctrl:   ArbSelect=Fixed

Status: InProgress-

VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-

Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-

Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff

Status: NegoPending- InProgress-

Capabilities: [160 v1] Device Serial Number 00-00-00-00-68-4c-e0-00

        Kernel driver in use: r8168

[yipingwang]

Hello Johnson,

I have verified aer-inject 0.1, there is on problem on my target.

Please wait my result to verify on a P2020 target board,because on some legacy platforms with legacy PCI conroller(e.g. some non-DPAA platforms), hardware doesn't support Fatal error type for AER, just support Non-Fatal error.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[rkj]

It would first be interesting to know if PCIe AER is fully functional on P2020 using SDK 1.3 since that is closest to the current board support package for my SBC.  If not, then it would be interesting to know if it works better using SDK 1.6.

[rkj]

I was looking at both the Freescale SDK 1.3 and 1.3.2 source ISO's and they both appear to be based on Linux kernel 3.2.0.  I thought you said SDK 1.3 used Linux kernel 3.0.48?

[yipingwang]

Hello Johnson,

Linux Kernel in SDK 1.3 is 3.0.48, I am sure.

I have verified PCIe AER function on P2020RDB-PC with SDK 1.3 images, and it executed successfully.

I used aer-inject 0.1 and your configuration file.

I attached Kernel log and aer-inject binary file for you, hope they are helpful.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[rkj]

OK, I guess I can try to compare the SDK 1.3 kernel to my 3.0.4 kernel to see what the differences are.  I can also try to get the SDK 1.3 kernel to run on my target to see if at least the aer-inject test works.

As I mentioned, I downloaded both the SDK 1.3 source ISO and the 1.3.2 source ISO, they both have a Linux 3.2.0 kernel.  For instance, the only kernel source I see is named "linux-3.2.tar.bz2", the top level Makefile inside this file starts with:

VERSION = 3

PATCHLEVEL = 2

SUBLEVEL = 0

EXTRAVERSION =

NAME = Saber-toothed Squirrel

which indicates kernel version 3.2.0.  Is there another older version of SDK 1.3 that is based on Linux kernel 3.0.48?  Is it available to download?

[yipingwang]

Hello Johnson,

In SDK 1.3, the tar ball linux-3.2.tar.bz2 is not Freescale Linux Kernel source tar ball, actually git2_git.freescale.com.ppc.sdk.linux.git.tar.gz is Kernel source tar ball,  the Kernel source is packaged from Freescale git tree.

You could use the following method to extract Kernel source code.

After set up the host build environment, run the command

>>bitbake virtual/kernel -c patch -f

Get the source in build_p2020rdb_release/tmp/work/p2020rdb-fsl-linux-gnuspe/linux-qoriq-sdk-3.0.48-r5/git.

>>bitbake virtual/kernel -c configure -f

( this command is used to get the default Kernel configuration file)

Then go to Kernel source folder, and use "make ARCH=powerpc menuconfig" to configure Kernel.

Rebuild Kernel with

>>bitbake virtual/kernel

SDK 1.3 could be downloaded from Freescale Linux SDK for QorIQ Processors, v1.|Freescale, please remember to download and install both SOURCE and E500V2 Cache Binary ISOs, the SOURCE ISO contains source tar balls, recipes and documents, the Cache Binary ISO contains binaries to accelerate the building.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[rkj]

I did not get a chance to try anything with the SDK 1.3 kernel source.  I am not familiar with Yocto and it looked like it may take some effort to properly configure a kernel.

However, the single board computer vendor produced a 3.14.15 kernel and device tree that I was able to use for testing.  The AER driver appeared to be functional:

# dmesg|grep aer

aer 0000:00:00.0:pcie02: service driver aer loaded

dmesg output after using aer-inject test:

pcieport 0000:00:00.0: AER: Corrected error received: id=0000

pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0000(Receiver ID)

pcieport 0000:00:00.0:   device [1957:0070] error status/mask=00000040/00000000

pcieport 0000:00:00.0:    [ 6] Bad TLP

3.14.X is currently the latest long term maintenance Linux kernel release.  We will likely migrate to this kernel version.

[yipingwang]

Hello Johnson,

It is OK that you want to use Linux 3.14, and it is newer than SDK 1.6 release(Linux 3.12).

Previously you said you were using Linux 3.0.4, so I recommended you SDK 1.3(3.0.48) the closest version to yours. I think probably you had finished most of your porting work on Linux 3.0.4, so I didn't suggest you to migrate to the latest SDK.

Anyway if you need the assistance to use Freescale Linux SDK, please feel free to let me know.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[rkj]

I did have a few questions about the Freescale Linux SDK.

Which Linux kernel will SDK 1.7 use?

For the case of the P2020, does SDK 1.6 include patches to the standard Linux kernel or is it mainly have to do with properly configuring the standard Linux kernel?  In other words, does Freescale find it necessary to modify the Linux kernel source in order to get things to work well on the P2020?

[yipingwang]

Hello Johnson,

Linux SDK 1.6 is just released in June, it integrates Linux 3.12.

Freescale Linux SDK is based on standard Linux Kernel, and integrates Freescale specific patches for Freescale target boards to make them work well in functionality and performance.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[rkj]

I was trying to explore SDK 1.6 for the p2020rdb machine.  I would like to see the list of patches that are applied to the standard Linux kernel.  I have no experience with Yocto, but from looking at the documentation, I thought it might be helpful to use the yocto-kernel commands "config list", "patch list", and "feature list", however they do not appear to work as I get an error message:

user@XGbE-00:~/linux/fsl/QorIQ-SDK-V1.6-20140619-yocto/build_p2020rdb_release$ yocto-kernel patch list p2020rdb

Unable to find the BSP layer for machine p2020rdb.

Please make sure it is listed in bblayers.conf

What is the proper format of yocto commands to look at kernel config, patch, and feature lists for SDK 1.6?  Are there other commands that would be useful to use in order to determine how the Linux kernel has  been modified, configured, etc.?

[yipingwang]

Hello Johnson,

Please use the command "bitbake virtual/kernel -c patch -f" to extract the source in

build_p2020rdb_release/tmp/work/p2020rdb-fsl-linux-gnuspe/linux-qoriq-sdk/3.12-r0/git.

This is a git tree folder, you could use "git log" to show the patch list.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[scottwood]

P2020RDB is supported in upstream Linux; unless there is some specific SDK feature you need, I would not bother with the SDK kernel for that target.

SDK 1.6 has many patches on top of the 3.12 kernel, most of which are not relevant to p2020, or are special-purpose optimizations, etc.  The SDK does not have a separate kernel for each target.

[lunminliang]

Have a compare of this part drivers/pci/pcie/aer/*.c, between kernel 3.0.4 and kernel in SDK1.3, they are same...