MSI Still Broken on NXP BSP L4.14.98_2.3.1_patch?

[marcelziswiler]

Since a while, we carry the pci=nomsi kernel boot command-line argument in our defaults. After our recent re-base on-top-of NXP BSP L4.14.98_2.3.1_patch we did some more PCIe tests but it looks like enabling MSI (e.g. leaving that pci=nomsi away) together with the PCIe switch on our Apalis Evaluation board still fails on the i.MX 8QuadMax B0 silicon chips:

[ 7.634640] imx6q-pcie 5f000000.pcie: 5f000000.pcie supply epdev_on not found, using dummy regulator
[ 7.653008] OF: PCI: host bridge /pcie@0x5f000000 ranges:
[ 7.660167] OF: PCI: IO 0x6ff80000..0x6ff8ffff -> 0x00000000
[ 7.667846] OF: PCI: MEM 0x60000000..0x6fefffff -> 0x60000000
[ 7.782667] imx6q-pcie 5f000000.pcie: pcie phy pll is locked.
[ 7.924853] imx6q-pcie 5f000000.pcie: Link: Gen2 disabled
[ 7.933635] imx6q-pcie 5f000000.pcie: Link up, Gen2
[ 7.943544] imx6q-pcie 5f000000.pcie: PCI host bridge to bus 0000:00
[ 7.953249] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 7.962069] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 7.971528] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fefffff]
[ 8.064416] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 8.080064] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 8.091303] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 8.102556] pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 8.114902] pci 0000:04:00.0: enabling Extended Tags
[ 8.132732] pci 0000:00:00.0: bridge configuration invalid ([bus 00-ff]), reconfiguring
[ 8.143227] pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x60ffffff 64bit]
[ 8.152326] pci 0000:00:00.0: BAR 6: assigned [mem 0x61000000-0x61ffffff pref]
[ 8.161305] pci 0000:00:00.0: BAR 14: assigned [mem 0x62000000-0x623fffff]
[ 8.169922] pci 0000:00:00.0: BAR 15: assigned [mem 0x62400000-0x625fffff 64bit pref]
[ 8.179511] pci 0000:00:00.0: BAR 13: assigned [io 0x1000-0x1fff]
[ 8.187445] pci 0000:01:00.0: BAR 14: assigned [mem 0x62000000-0x622fffff]
[ 8.196068] pci 0000:01:00.0: BAR 15: assigned [mem 0x62400000-0x625fffff 64bit pref]
[ 8.205636] pci 0000:01:00.0: BAR 0: assigned [mem 0x62300000-0x62303fff]
[ 8.214155] pci 0000:01:00.0: BAR 13: assigned [io 0x1000-0x1fff]
[ 8.222057] pci 0000:02:01.0: BAR 14: assigned [mem 0x62000000-0x621fffff]
[ 8.230643] pci 0000:02:01.0: BAR 15: assigned [mem 0x62400000-0x625fffff 64bit pref]
[ 8.240174] pci 0000:02:02.0: BAR 14: assigned [mem 0x62200000-0x622fffff]
[ 8.248717] pci 0000:02:01.0: BAR 13: assigned [io 0x1000-0x1fff]
[ 8.256558] pci 0000:02:01.0: PCI bridge to [bus 03]
[ 8.263189] pci 0000:02:01.0: bridge window [io 0x1000-0x1fff]
[ 8.270974] pci 0000:02:01.0: bridge window [mem 0x62000000-0x621fffff]
[ 8.279456] pci 0000:02:01.0: bridge window [mem 0x62400000-0x625fffff 64bit pref]
[ 8.288904] pci 0000:04:00.0: BAR 0: assigned [mem 0x62200000-0x62203fff 64bit]
[ 8.297930] pci 0000:02:02.0: PCI bridge to [bus 04]
[ 8.304572] pci 0000:02:02.0: bridge window [mem 0x62200000-0x622fffff]
[ 8.313070] pci 0000:02:03.0: PCI bridge to [bus 05]
[ 8.319748] pci 0000:01:00.0: PCI bridge to [bus 02-05]
[ 8.326611] pci 0000:01:00.0: bridge window [io 0x1000-0x1fff]
[ 8.334369] pci 0000:01:00.0: bridge window [mem 0x62000000-0x622fffff]
[ 8.342780] pci 0000:01:00.0: bridge window [mem 0x62400000-0x625fffff 64bit pref]
[ 8.352185] pci 0000:00:00.0: PCI bridge to [bus 01-05]
[ 8.358998] pci 0000:00:00.0: bridge window [io 0x1000-0x1fff]
[ 8.366649] pci 0000:00:00.0: bridge window [mem 0x62000000-0x623fffff]
[ 8.374954] pci 0000:00:00.0: bridge window [mem 0x62400000-0x625fffff 64bit pref]
[ 8.385018] pcieport 0000:00:00.0: enabling device (0000 -> 0003)
[ 8.392965] pcieport 0000:00:00.0: Signaling PME with IRQ 454
[ 8.400450] pcieport 0000:00:00.0: AER enabled with IRQ 455
[ 8.408135] pcieport 0000:01:00.0: enabling device (0000 -> 0003)
[ 8.417018] pcieport 0000:02:01.0: enabling device (0000 -> 0003)
[ 8.432548] pciehp 0000:02:01.0:pcie204: Slot #1 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise- Interlock- NoCompl- LLActRep+
[ 8.446701] Unable to handle kernel NULL pointer dereference at virtual address 0000003c
[ 8.456496] Mem abort info:
[ 8.460976] Exception class = DABT (current EL), IL = 32 bits
[ 8.468595] SET = 0, FnV = 0
[ 8.473353] EA = 0, S1PTW = 0
[ 8.478202] Data abort info:
[ 8.482811] ISV = 0, ISS = 0x00000004
[ 8.488352] CM = 0, WnR = 0
[ 8.492992] [000000000000003c] user address but active_mm is swapper
[ 8.501053] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 8.508361] Modules linked in:
[ 8.513128] Process kworker/1:3 (pid: 1974, stack limit = 0xffff000012668000)
[ 8.522062] CPU: 1 PID: 1974 Comm: kworker/1:3 Not tainted 4.14.170-07228-g3a6cd4e88715 #18
[ 8.532249] Hardware name: Toradex Apalis iMX8QM/QP on Apalis Evaluation Board (DT)
[ 8.541797] Workqueue: events deferred_probe_work_func
[ 8.548828] task: ffff8008794b3600 task.stack: ffff000012668000
[ 8.556653] PC is at msi_set_mask_bit.isra.8+0x10/0xb8
[ 8.563700] LR is at pci_msi_unmask_irq+0x1c/0x28
[ 8.570307] pc : [<ffff0000084efb08>] lr : [<ffff0000084efbf4>] pstate: 400000c5
[ 8.579675] sp : ffff00001266b450
[ 8.584950] x29: ffff00001266b450 x28: ffff800874180800
[ 8.592251] x27: 0000000000000040 x26: ffff0000092d08d8
[ 8.599540] x25: ffff80087412caa4 x24: ffff80087412cb58
[ 8.606821] x23: 0000000000000000 x22: 0000000000000001
[ 8.614113] x21: ffff80087412ca28 x20: ffff80087412ca00
[ 8.621417] x19: 0000000000000000 x18: 00000000fffffffb
[ 8.628724] x17: 00000000139eb36a x16: 0000000000000000
[ 8.636005] x15: 0000000000000000 x14: ffffffffffffffff
[ 8.643274] x13: ffff80087417c91c x12: 0000000000000000
[ 8.650536] x11: 0000000000000000 x10: 0000000000000040
[ 8.657808] x9 : 0000000000000000 x8 : ffff800878cc3100
[ 8.665069] x7 : 0000000000000000 x6 : 000000000000003f
[ 8.672328] x5 : 0000000000000040 x4 : 0000000000000000
[ 8.679578] x3 : 0000000000000040 x2 : 0000000000000000
[ 8.686786] x1 : ffff80087412ca00 x0 : 00000000000001ca
[ 8.693989] Call trace:
[ 8.698304] Exception stack(0xffff00001266b310 to 0xffff00001266b450)
[ 8.706690] b300: 00000000000001ca ffff80087412ca00
[ 8.716483] b320: 0000000000000000 0000000000000040 0000000000000000 0000000000000040
[ 8.726301] b340: 000000000000003f 0000000000000000 ffff800878cc3100 0000000000000000
[ 8.736112] b360: 0000000000000040 0000000000000000 0000000000000000 ffff80087417c91c
[ 8.745919] b380: ffffffffffffffff 0000000000000000 0000000000000000 00000000139eb36a
[ 8.755715] b3a0: 00000000fffffffb 0000000000000000 ffff80087412ca00 ffff80087412ca28
[ 8.765499] b3c0: 0000000000000001 0000000000000000 ffff80087412cb58 ffff80087412caa4
[ 8.775203] b3e0: ffff0000092d08d8 0000000000000040 ffff800874180800 ffff00001266b450
[ 8.784837] b400: ffff0000084efbf4 ffff00001266b450 ffff0000084efb08 00000000400000c5
[ 8.794448] b420: ffff00000907e680 0000000000000000 ffffffffffffffff 0000000000000000
[ 8.804051] b440: ffff00001266b450 ffff0000084efb08
[ 8.810675] [<ffff0000084efb08>] msi_set_mask_bit.isra.8+0x10/0xb8
[ 8.818651] [<ffff0000084efbf4>] pci_msi_unmask_irq+0x1c/0x28
[ 8.826200] [<ffff00000811d408>] irq_enable+0x38/0x70
[ 8.833050] [<ffff00000811d4b4>] __irq_startup+0x74/0x98
[ 8.840160] [<ffff00000811d53c>] irq_startup+0x64/0x120
[ 8.847167] [<ffff00000811b744>] __setup_irq+0x734/0x778
[ 8.854229] [<ffff00000811b908>] request_threaded_irq+0xd8/0x188
[ 8.861988] [<ffff0000084ec380>] pcie_init_notification+0x48/0xd8
[ 8.869813] [<ffff0000084ea0a0>] pciehp_probe+0x170/0x338
[ 8.876902] [<ffff0000084e5d04>] pcie_port_probe_service+0x34/0x60
[ 8.884745] [<ffff0000086dc6bc>] driver_probe_device+0x20c/0x2c8
[ 8.892388] [<ffff0000086dc8f0>] __device_attach_driver+0xb8/0xe8
[ 8.900111] [<ffff0000086da9c8>] bus_for_each_drv+0x68/0xa8
[ 8.907294] [<ffff0000086dc390>] __device_attach+0xc0/0x130
[ 8.914477] [<ffff0000086dc970>] device_initial_probe+0x10/0x18
[ 8.921978] [<ffff0000086db8a0>] bus_probe_device+0x90/0x98
[ 8.929112] [<ffff0000086d98d0>] device_add+0x328/0x5b0
[ 8.935893] [<ffff0000086d9b74>] device_register+0x1c/0x28
[ 8.942939] [<ffff0000084e6000>] pcie_port_device_register+0x228/0x490
[ 8.951040] [<ffff0000084e6504>] pcie_portdrv_probe+0x34/0xc0
[ 8.958351] [<ffff0000084d77bc>] local_pci_probe+0x3c/0xb0
[ 8.965402] [<ffff0000084d84bc>] pci_device_probe+0xe4/0x188
[ 8.972617] [<ffff0000086dc6bc>] driver_probe_device+0x20c/0x2c8
[ 8.980201] [<ffff0000086dc8f0>] __device_attach_driver+0xb8/0xe8
[ 8.987881] [<ffff0000086da9c8>] bus_for_each_drv+0x68/0xa8
[ 8.995031] [<ffff0000086dc390>] __device_attach+0xc0/0x130
[ 9.002188] [<ffff0000086dc410>] device_attach+0x10/0x18
[ 9.009095] [<ffff0000084cccac>] pci_bus_add_device+0x4c/0xa8
[ 9.016452] [<ffff0000084ccd40>] pci_bus_add_devices+0x38/0x88
[ 9.023903] [<ffff0000084ccd70>] pci_bus_add_devices+0x68/0x88
[ 9.031350] [<ffff0000084ccd70>] pci_bus_add_devices+0x68/0x88
[ 9.038780] [<ffff0000084fc63c>] dw_pcie_host_init+0x2c4/0x4f0
[ 9.046207] [<ffff0000084fe2b4>] imx_pcie_probe+0x60c/0xa08
[ 9.053354] [<ffff0000086de268>] platform_drv_probe+0x58/0xb8
[ 9.060641] [<ffff0000086dc6bc>] driver_probe_device+0x20c/0x2c8
[ 9.068180] [<ffff0000086dc8f0>] __device_attach_driver+0xb8/0xe8
[ 9.075818] [<ffff0000086da9c8>] bus_for_each_drv+0x68/0xa8
[ 9.082932] [<ffff0000086dc390>] __device_attach+0xc0/0x130
[ 9.090043] [<ffff0000086dc970>] device_initial_probe+0x10/0x18
[ 9.097496] [<ffff0000086db8a0>] bus_probe_device+0x90/0x98
[ 9.104610] [<ffff0000086dc09c>] deferred_probe_work_func+0xa4/0x148
[ 9.112523] [<ffff0000080e69b0>] process_one_work+0x1c8/0x328
[ 9.119816] [<ffff0000080e6d50>] worker_thread+0x240/0x450
[ 9.126844] [<ffff0000080ecae0>] kthread+0x128/0x130
[ 9.133319] [<ffff000008084e98>] ret_from_fork+0x10/0x18
[ 9.140128] Code: a9be7bfd 910003fd f9000bf3 f9400833 (3940f261)
[ 9.147734] ---[ end trace fd7940b8b69853f2 ]---
[ 9.153866] note: kworker/1:3[1974] exited with preempt_count 1

Could NXP please confirm whether or not MSI indeed should work? Have you guys ever tried this with a PCIe switch setup similar to what we have on our EvalBoard?

For reference, I attached full log files both on our EvalBoard with and without MSI as well as on Ixora (which is another carrier board of ours however without any PCIe switch) with MSI.

[workkkk]

Hello.

We are facing the same problem. Have you ever solved it? Excuse me, but if it has been resolved, can you tell me how it was resolved?

thanks.

[marcelziswiler]

So nobody cares that MSI is broken?

[gusarambula]

Hello Marcel Ziswiler,

My apologies. The i.MX Linux BSPs do support the Support Legacy Interrupts (INTx) and MSI. However, I don’t have information on testing in NXP boards.

The Apalis board is manufactured and supported by Toradex so we cannot reproduce on this board. Have you tested on an NXP board? Since this processor’s technical information is not publicly available (like the Reference Manual). I would recommend contacting your NXP FAE or Distributor for more information on this behavior (please include the Silicon revision you are using on your inquiery).

My apologies for the inconvenience.

Regards,

[marcelziswiler]

> My apologies. The i.MX Linux BSPs do support the Support Legacy Interrupts (INTx) and MSI. However, I don’t have information on testing in NXP boards.

I am not exactly sure what you mean. It is supported but has never been tested or what?

> The Apalis board is manufactured and supported by Toradex so we cannot reproduce on this board.

Yes, I am the team lead at Toradex responsible for the BSP bring-up and we closely work on this with NXP since more than 3 years!

> Have you tested on an NXP board?

No. Unfortunately, I do neither have such NXP board, nor any further equipment which may be required, available to me right now. That said, we should have a i.MX 8QuadMax Multisensory Enablement Kit (MEK) available at Toradex somewhere. Then, of course, the next question would be how exactly to go about testing such a use case involving a PCIe switch...

> Since this processor’s technical information is not publicly available (like the Reference Manual).

As early access partners, we at Toradex of course have and always had access to all this.

> I would recommend contacting your NXP FAE or Distributor for more information on this behavior (please include the Silicon revision you are using on your inquiery).

I initially already asked on the early access forum where I got told to ask here now as the chip has been officially launched. Now, you tell me to contact our NXP FAE or distributor. Are you kidding me?

[gusarambula]

Hello Marcel Ziswiler,

My apologies for the delay. I made some research with our experts and they couldn’t reproduce the issue on the i.MX8QM-MEK with connect switch.  They also confirmed that our BSP have always supported MSI.

00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01) (prog-if 00 [Normal decode])

Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+

Address: 0000000000000000 Data: 0000

01:00.0 PCI bridge: Pericom Semiconductor Device b304 (rev 01) (prog-if 00 [Normal decode])

Capabilities: [4c] MSI: Enable- Count=1/4 Maskable+ 64bit+

root@imx8qmmek:~# lspci

00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01)

01:00.0 PCI bridge: Pericom Semiconductor Device b304 (rev 01)

02:01.0 PCI bridge: Pericom Semiconductor Device b304 (rev ff)

02:02.0 PCI bridge: Pericom Semiconductor Device b304 (rev ff)

This confirms that MSI should be working with the BSP Release and the i.MXQM. Is this problem still present in your board and how may we help?

Regards,