PCIe PHY error when nvme card plug-in on RDB3 with BSP44 release

[hittzt]

Hi,

Recently, I tested RDB3 board with SDK BSP44 release, and the kernel version is v6.6.

When I tested nvme card status, there are following errors:

    [ 1.839960] pcieport 0000:00:00.0: AER: enabled with IRQ 52
    [ 1.845986] nvme nvme0: pci function 0000:01:00.0
    [ 1.850782] nvme 0000:01:00.0: enabling device (0000 -> 0002)
    [ 1.859216] nvme nvme0: missing or invalid SUBNQN field.
    [ 1.867590] nvme nvme0: 8/0/0 default/read/poll queues
    [ 1.878452] s32cc-pcie 40400000.pcie: PHY error detected after Hot-Plug event

And I found that, if the CONFIG_PCIEAER setting to 'y', then the above error will show, and if 'n', the board can boot normally without above error. And the log is attached.

But I am not sure why this happened, would you please help to check the issue?

Thanks,

Zhantao

[chenyin_h]

Hello, @hittzt 

Sorry for late response.

We had purchased a "Kingston SA2000M8/250 250G SSD" to do the test, and reproduced your issue, the reason is that currently the AER feature is not supported correctly.

BR

Chenyin

[chenyin_h]

Hello, @hittzt 

Thanks for the information.

Actually, with another Toshiba SSD, the issue could not be reproduced, the issue may possibly only be triggered by certain SSDs 

I have discussed it with my colleague with your specific issue, and is still investigating on it.

If any findings, I will update it for your reference.

BR

Chenyin

[chenyin_h]

Hello, @hittzt 

Thanks for your post.

May I know which kind of serdes settings you are working with? I assume that the NVME is used via the M.2 interface on serdes0lan1?

May I know if similar issue is found from the BSP43?

BR

Chenyin 

[hittzt]

Hi @chenyin_h,

Yes, I used the M.2 Mkey slot for NVMe card, and with default hwconfig:

=> pri hwconfig
hwconfig=serdes0:mode=pcie,clock=ext;pcie0:mode=rc;serdes1:mode=xpcs0&xpcs1,clock=ext,fmhz=125;xpcs1_0:speed=2G5;xpcs1_1:speed=2G5

I will try to test BSP43 image to see if similar issue or not.

Thanks,

Zhantao

[chenyin_h]

Hello, @hittzt 

Thanks for your reply.

I did a similar test on my RDB3, but do not find such errors, but I only tested one kind of SSD since only one in hand.

May I know if it is possible for you to change another SSD for a trial?

BR

Chenyin

[hittzt]

Hi @chenyin_h,

Did you used the M.2 M-key slot for the test, I have only one M.2 NVMe card too, and I will try to find another NVMe SSD to test.

And did you enable the PCIEAER configure?

Thanks,

Zhantao

[chenyin_h]

Hello, @hittzt 

Thanks for your reply.

1. Yes, I enabled it, CONFIG_PCIEAER=y

2. I tested it again and found that the NVME could not well recognized by the system if enabled the configuration you mentioned, although the device could be detected as a PCIe device, but I cannot find the error log like your "PHY error detected after Hot-Plug event"

Whatever, seems there is issue when enabling it, let me check it further, would you mind providing the detailed information of the SSD you used? only CONFIG_PCIEAER=y is added upon the default kernel configuration provided by the BSP44?

BR

Chenyin

[hittzt]

Hi @chenyin_h,

Thanks for your quick check.

The NVMe card I used is Kingston SA2000M8/250 250G SSD card.

Yes, just setting the CONFIG_PCIEAER=y should trigger the issue.

Thanks,

Zhantao

[chenyin_h]

Hi, @hittzt 

Thanks for your confirmation.

One more question, once the warning/error log found while enabled the configuration from your side, could the NMVE be well recognized by the system? say it could be found and operated with fdisk, cp related commands?

BR

Chenyin

[hittzt]

Hi @chenyin_h,

As the previously attached log shows, the NVMe card can be recognized, but the phy error will be printed in a forever loop, which causes the normal login failure, so I dont think the NVMe card is workable in that status.

Thanks,

Zhantao

[chenyin_h]

Hello, @hittzt 

Thanks for your confirmation.

Would you mind also sharing your full logs, with also prints of /proc/interrupts?

BR

Chenyin

[hittzt]

Hi @chenyin_h,

Because of the "phy error" messages, the board can not be logged in and accessed, so I am not sure the interrupt status.

Thanks,

Zhantao

[hittzt]

Hi @chenyin_h,

I tried to disable the PCIEAER config, and got the following interrupt status:

root@nxp-s32g:~# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
9: 0 0 0 0 0 0 0 0 GICv3 25 Level vgic
11: 105842 102465 103481 104614 107152 109777 107475 106602 GICv3 30 Level arch_timer
12: 0 0 0 0 0 0 0 0 GICv3 27 Level kvm guest vtimer
15: 0 0 0 0 0 0 0 0 GICv3 242 Level 4009d700.siul2-gpio
18: 0 0 0 0 0 0 0 0 GICv3 332 Edge arm-scmi.0.auto
21: 0 0 0 0 0 0 0 0 GICv3 102 Level 401f8000.adc
22: 0 0 0 0 0 0 0 0 GICv3 103 Level 402e8000.adc
23: 0 0 0 0 0 0 0 0 GICv3 23 Level arm-pmu
24: 666 0 0 0 0 0 0 0 GICv3 40 Level eDMA tx0_15
25: 0 0 0 0 0 0 0 0 GICv3 41 Level eDMA tx16_31
26: 0 0 0 0 0 0 0 0 GICv3 42 Level eDMA err
27: 0 0 0 0 0 0 0 0 GICv3 43 Level eDMA tx0_15
28: 0 0 0 0 0 0 0 0 GICv3 44 Level eDMA tx16_31
29: 0 0 0 0 0 0 0 0 GICv3 45 Level eDMA err
30: 248 0 0 0 0 0 0 0 GICv3 124 Level 401e4000.i2c
31: 0 0 0 0 0 0 0 0 GICv3 126 Level 401ec000.i2c
32: 17 0 0 0 0 0 0 0 GICv3 128 Level 402dc000.i2c
34: 0 0 0 0 0 0 0 0 GICv3 160 Level PCIe PME
35: 22 0 0 0 0 0 0 0 MBI-PCI-MSIX-0000:01:00.0 0 Edge nvme0q0
36: 8 0 0 0 0 0 0 0 MBI-PCI-MSIX-0000:01:00.0 1 Edge nvme0q1
37: 0 1 0 0 0 0 0 0 MBI-PCI-MSIX-0000:01:00.0 2 Edge nvme0q2
38: 0 0 18 0 0 0 0 0 MBI-PCI-MSIX-0000:01:00.0 3 Edge nvme0q3
39: 0 0 0 10 0 0 0 0 MBI-PCI-MSIX-0000:01:00.0 4 Edge nvme0q4
40: 0 0 0 0 14 0 0 0 MBI-PCI-MSIX-0000:01:00.0 5 Edge nvme0q5
41: 0 0 0 0 0 12 0 0 MBI-PCI-MSIX-0000:01:00.0 6 Edge nvme0q6
42: 0 0 0 0 0 0 1 0 MBI-PCI-MSIX-0000:01:00.0 7 Edge nvme0q7
43: 0 0 0 0 0 0 0 7 MBI-PCI-MSIX-0000:01:00.0 8 Edge nvme0q8
44: 0 0 0 0 0 0 0 0 GICv3 156 Level s32cc-pcie-hot-unplug
45: 0 0 0 0 0 0 0 0 GICv3 164 Level s32cc-pcie-hotplug
48: 0 0 0 0 0 0 0 0 GICv3 132 Level fccu_alarm
49: 0 0 0 0 0 0 0 0 GICv3 133 Level fccu_misc
50: 0 0 0 0 0 0 0 0 GICv3 118 Level 401d8000.spi
51: 0 0 0 0 0 0 0 0 GICv3 122 Level 402d0000.spi
52: 0 0 0 0 0 0 0 0 GICv3 64 Level 0.spi
61: 0 0 0 0 0 0 0 0 GICv3 89 Level end0:mac
62: 1499 0 0 0 0 0 0 0 GICv3 91 Level end0:rx-0
63: 0 0 0 0 0 0 0 0 GICv3 93 Level end0:rx-1
64: 0 0 0 0 0 0 0 0 GICv3 95 Level end0:rx-2
65: 0 0 0 0 0 0 0 0 GICv3 97 Level end0:rx-3
66: 0 0 0 0 0 0 0 0 GICv3 99 Level end0:rx-4
67: 0 0 0 0 0 0 0 0 GICv3 90 Level end0:tx-0
68: 0 0 0 0 0 0 0 0 GICv3 92 Level end0:tx-1
69: 1 0 0 0 0 0 0 0 GICv3 94 Level end0:tx-2
70: 0 0 0 0 0 0 0 0 GICv3 96 Level end0:tx-3
71: 0 0 0 0 0 0 0 0 GICv3 98 Level end0:tx-4
72: 0 0 0 0 0 0 0 0 GICv3 243 Level 44064000.usb
73: 0 0 0 0 0 0 0 0 GICv3 153 Level rtc
74: 0 0 0 0 0 0 0 0 GICv3 152 Level 400a8000.tmu
75: 7170 0 0 0 0 0 0 0 GICv3 68 Level mmc0
76: 0 0 0 0 0 0 0 0 GICv3 147 Level s32cc_ddr_perf_pmu
77: 1 0 0 0 0 0 0 0 GICv3 226 Edge PFE BMU IRQ
80: 18 0 0 0 0 0 0 0 GICv3 222 Edge pfe-hif-0:shared
81: 1409 0 0 0 0 0 0 0 GICv3 223 Edge pfe-hif-1:excl
82: 0 0 0 0 0 0 0 0 GICv3 224 Edge pfe-hif-2:excl
83: 0 0 0 0 0 0 0 0 GICv3 206 Level rxin_fifo_0_7
84: 0 0 0 0 0 0 0 0 GICv3 207 Level rxin_fifo_8_15
90: 0 0 0 0 0 0 0 0 GICv3 204 Level linflex_irq
91: 0 0 0 0 0 0 0 0 GICv3 203 Level lpspi_irq
IPI0: 54768 52807 51048 54161 57014 57239 60500 55862 Rescheduling interrupts
IPI1: 399446 405813 368876 278209 358055 344725 336507 394790 Function call interrupts
IPI2: 0 0 0 0 0 0 0 0 CPU stop interrupts
IPI3: 0 0 0 0 0 0 0 0 CPU stop NMIs
IPI4: 0 0 0 0 0 0 0 0 Timer broadcast interrupts
IPI5: 43932 34401 32135 33969 37469 40678 40230 34483 IRQ work interrupts
IPI6: 0 0 0 0 0 0 0 0 CPU backtrace interrupts
IPI7: 396 2811 3105 3106 3098 3086 3034 3106 KGDB roundup interrupts
Err: 0

Hope it can help.

Thanks,

Zhantao