mask MSI interrupt doesn't work on IMX8

qunz · ‎05-10-2020

The current used platform: IMX8 EVK/QXP board

PCIe wlan chip: Qualcomm qca6696

Linux kernel version: L4.14.78_1.0.0_MX8QXP (REV L4.14.78_1.0.0 ) L4.14.78_1.0.0_ga

Image version: 2019-01-03 17:47:00 L4.14.78_1.0.0_MX8QXP

Hi Sir,

I am integrating Qualcomm WLAN chip to IMX8 platform, and found the disable_irq_nosync for MSI interrupt doesn’t work as expected.

The background is:

Our WLAN chip driver have multiple MSI interrupts through DMA copy engine
In WLAN RX data path, when receive RX data interrupt, wlan host driver will disable the MSI interrupt firstly, then schedule kernel NAPI softirq to process RX data. When NAPI schedule processing finished, will re-enable the MSI interrupt again. Plz find below call stack:

[ 104.930753] [<ffff0000084db030>] msi_set_mask_bit.isra.7+0xa8/0xd0

[ 104.930758] [<ffff0000084db070>] pci_msi_mask_irq+0x18/0x20

[ 104.930763] [<ffff000008120434>] __irq_disable+0x94/0xa8

[ 104.930768] [<ffff000008120fe4>] irq_disable+0x14/0x20

[ 104.930775] [<ffff00000811d204>] __disable_irq_nosync+0x5c/0x68

[ 104.930781] [<ffff00000811d21c>] disable_irq_nosync+0xc/0x18 -> disable MSI interrupt

[ 104.933638] [<ffff0000010cc440>] pld_srng_disable_irq+0x38/0x80 [wlan_cnss0]

[ 104.936499] [<ffff000001017014>] hif_ce_srng_msi_irq_disable+0x1c/0x28 [wlan_cnss0]

[ 104.939352] [<ffff0000010134f0>] hif_irq_disable+0x10/0x18 [wlan_cnss0]

[ 104.942218] [<ffff000001021920>] ce_dispatch_interrupt+0x40/0x278 [wlan_cnss0]

[ 104.945068] [<ffff000001016ad0>] hif_ce_interrupt_handler+0x10/0x18 [wlan_cnss0] -> wlan receive interrupt

3. I found the disabled MSI action doesn’t take effect as expected, the RX interrupt still continue to send to kernel, so the NAPI softirq processing is always preempted by RX interrupt and others tasks don’t have chances to be scheduled, kernel side always have RCU stall report. Plz find below RCU stall info:

[ 850.313246] INFO: rcu_preempt self-detected stall on CPU

[ 850.313260] 0-...: (5053 ticks this GP) idle=66e/140000000000001/0 softirq=4987/4987 fqs=2133

[ 850.313261] (t=5250 jiffies g=1405 c=1404 q=1269)

[ 850.313268] Task dump for CPU 0:

[ 850.313272] ksoftirqd/0 R running task 0 7 2 0x00000022

[ 850.313280] Call trace:

The issue can reproduce on I.MX8 EVK/QXP two boards with QCOM WIFI chip QCA6696. I didn’t verify on i.MX6 series, but I think it should be same.

Actually we also porting other QCOM WIFI chip to i.MX6/i.MX8 before, all can work up, this is because these chips use legacy PCIe interrupt or single MSI interrupt. The QCA6696 chip has more then ten MSI interrupts and use NAPI softirq for RX data, so need disable related MSI interrupt in each processing.

In IMX8 kernel 4.14, CONFIG_PCI_MSI_IRQ_DOMAIN Is defined, there should be a hierarchy layer for MSI irq, just like below, all WLAN MSI interrupts are registered to mx6-pcie-msi layer, so there are two layers interrupt for PCIe MSi, GIC and PCIe. In QCOM kernel, we have private processing for this part.

root@imx8qxpmek:~# cat /proc/interrupts

108: 825 0 0 0 GICv3 134 Level mx6-pcie-msi

445: 3 0 0 0 PCI-MSI 34 Edge WLAN_CE_0

446: 2 0 0 0 PCI-MSI 35 Edge WLAN_CE_1

447: 100 0 0 0 PCI-MSI 36 Edge WLAN_CE_2

448: 93 0 0 0 PCI-MSI 37 Edge WLAN_CE_3

450: 1 0 0 0 PCI-MSI 39 Edge WLAN_CE_5

451: 0 0 0 0 PCI-MSI 40 Edge WLAN_CE_6

453: 0 0 0 0 PCI-MSI 42 Edge WLAN_CE_8

457: 0 0 0 0 PCI-MSI 46 Edge wlan_EXT_GRP

458: 0 0 0 0 PCI-MSI 47 Edge wlan_EXT_GRP

459: 0 0 0 0 PCI-MSI 48 Edge wlan_EXT_GRP

460: 0 0 0 0 PCI-MSI 49 Edge wlan_EXT_GRP

461: 0 0 0 0 PCI-MSI 50 Edge wlan_EXT_GRP

462: 0 0 0 0 PCI-MSI 51 Edge wlan_EXT_GRP

463: 0 0 0 0 PCI-MSI 52 Edge wlan_EXT_GRP

In IMX8 current kernel 4.14, disable_irq_nosync will call __pci_msi_desc_mask_irq finally:

/*

* PCI 2.3 does not specify mask bits for each MSI interrupt. Attempting to

* mask all MSI interrupts by clearing the MSI enable bit does not work

* reliably as devices without an INTx disable bit will then generate a

* level IRQ which will never be cleared.

*/

u32 __pci_msi_desc_mask_irq(struct msi_desc *desc, u32 mask, u32 flag)

{

u32 mask_bits = desc->masked;

if (pci_msi_ignore_mask || !desc->msi_attrib.maskbit)

return 0;

mask_bits &= ~mask;

mask_bits |= flag;

pci_write_config_dword(msi_desc_to_pci_dev(desc), desc->mask_pos,

mask_bits);

return mask_bits;

}

From the highlight comments ,seems the bit mask may not work as expected, so I think IMX8 PCIe MSI controller should mask/unmask the source IRQ whenever PCIe device enable/disable their MSI.

So could you please help to confirm that: when calling disable IRQ, does it actually mask the IRQ at the interrupt controller on NXP platform or does it just go over PCIe link and mask MSI of QC WLAN chip? I think your side should do the latter, so wlan chip RX interrupt will continue report to kernel side. If it’s the latter, could you please raise a fix for it.

Thank you very much!

haidong_zheng · ‎05-11-2020

Hi Qun:

4.14 Kernel seem not implement MSI bottom interrupt mask/unmask feature.

Could you try Kernel 4.19.35 which has supported this feature ?

qunz · ‎05-11-2020

Hi Tom,

From 4.19.35 kernel link:

https://source.codeaurora.org/external/imx/linux-imx/log/?h=imx_4.19.35_1.1.0

Seems the drivers/pci/dwc folder was removed, right? if so, it should have a big difference with kernel 4.14.

So where is IMX8 PCIe controller driver located in kernel 4.19.35? Thanks!

Regards,

Qun

haidong_zheng · ‎05-11-2020

Hi Qun:

PCI dwc controller directory moved to drivers/pci/controller/dwc.

Bio_TICFSL · ‎05-11-2020

Good!

qunz · ‎05-11-2020

Hi Tom,

Thanks very much for your help!

Could you point to me which change enabled the feature on kernel 4.19.35? I can take a quick verification.

Regards,

Qun