The current used platform: IMX8 EVK/QXP board
PCIe wlan chip: Qualcomm qca6696
Linux kernel version: L4.14.78_1.0.0_MX8QXP (REV L4.14.78_1.0.0 ) L4.14.78_1.0.0_ga
Image version: 2019-01-03 17:47:00 L4.14.78_1.0.0_MX8QXP
Hi Sir,
I am integrating Qualcomm WLAN chip to IMX8 platform, and found the disable_irq_nosync for MSI interrupt doesn’t work as expected.
The background is:
[ 104.930753] [<ffff0000084db030>] msi_set_mask_bit.isra.7+0xa8/0xd0
[ 104.930758] [<ffff0000084db070>] pci_msi_mask_irq+0x18/0x20
[ 104.930763] [<ffff000008120434>] __irq_disable+0x94/0xa8
[ 104.930768] [<ffff000008120fe4>] irq_disable+0x14/0x20
[ 104.930775] [<ffff00000811d204>] __disable_irq_nosync+0x5c/0x68
[ 104.930781] [<ffff00000811d21c>] disable_irq_nosync+0xc/0x18 -> disable MSI interrupt
[ 104.933638] [<ffff0000010cc440>] pld_srng_disable_irq+0x38/0x80 [wlan_cnss0]
[ 104.936499] [<ffff000001017014>] hif_ce_srng_msi_irq_disable+0x1c/0x28 [wlan_cnss0]
[ 104.939352] [<ffff0000010134f0>] hif_irq_disable+0x10/0x18 [wlan_cnss0]
[ 104.942218] [<ffff000001021920>] ce_dispatch_interrupt+0x40/0x278 [wlan_cnss0]
[ 104.945068] [<ffff000001016ad0>] hif_ce_interrupt_handler+0x10/0x18 [wlan_cnss0] -> wlan receive interrupt
3. I found the disabled MSI action doesn’t take effect as expected, the RX interrupt still continue to send to kernel, so the NAPI softirq processing is always preempted by RX interrupt and others tasks don’t have chances to be scheduled, kernel side always have RCU stall report. Plz find below RCU stall info:
[ 850.313246] INFO: rcu_preempt self-detected stall on CPU
[ 850.313260] 0-...: (5053 ticks this GP) idle=66e/140000000000001/0 softirq=4987/4987 fqs=2133
[ 850.313261] (t=5250 jiffies g=1405 c=1404 q=1269)
[ 850.313268] Task dump for CPU 0:
[ 850.313272] ksoftirqd/0 R running task 0 7 2 0x00000022
[ 850.313280] Call trace:
The issue can reproduce on I.MX8 EVK/QXP two boards with QCOM WIFI chip QCA6696. I didn’t verify on i.MX6 series, but I think it should be same.
Actually we also porting other QCOM WIFI chip to i.MX6/i.MX8 before, all can work up, this is because these chips use legacy PCIe interrupt or single MSI interrupt. The QCA6696 chip has more then ten MSI interrupts and use NAPI softirq for RX data, so need disable related MSI interrupt in each processing.
In IMX8 kernel 4.14, CONFIG_PCI_MSI_IRQ_DOMAIN Is defined, there should be a hierarchy layer for MSI irq, just like below, all WLAN MSI interrupts are registered to mx6-pcie-msi layer, so there are two layers interrupt for PCIe MSi, GIC and PCIe. In QCOM kernel, we have private processing for this part.
root@imx8qxpmek:~# cat /proc/interrupts
108: 825 0 0 0 GICv3 134 Level mx6-pcie-msi
445: 3 0 0 0 PCI-MSI 34 Edge WLAN_CE_0
446: 2 0 0 0 PCI-MSI 35 Edge WLAN_CE_1
447: 100 0 0 0 PCI-MSI 36 Edge WLAN_CE_2
448: 93 0 0 0 PCI-MSI 37 Edge WLAN_CE_3
450: 1 0 0 0 PCI-MSI 39 Edge WLAN_CE_5
451: 0 0 0 0 PCI-MSI 40 Edge WLAN_CE_6
453: 0 0 0 0 PCI-MSI 42 Edge WLAN_CE_8
457: 0 0 0 0 PCI-MSI 46 Edge wlan_EXT_GRP
458: 0 0 0 0 PCI-MSI 47 Edge wlan_EXT_GRP
459: 0 0 0 0 PCI-MSI 48 Edge wlan_EXT_GRP
460: 0 0 0 0 PCI-MSI 49 Edge wlan_EXT_GRP
461: 0 0 0 0 PCI-MSI 50 Edge wlan_EXT_GRP
462: 0 0 0 0 PCI-MSI 51 Edge wlan_EXT_GRP
463: 0 0 0 0 PCI-MSI 52 Edge wlan_EXT_GRP
In IMX8 current kernel 4.14, disable_irq_nosync will call __pci_msi_desc_mask_irq finally:
/*
* PCI 2.3 does not specify mask bits for each MSI interrupt. Attempting to
* mask all MSI interrupts by clearing the MSI enable bit does not work
* reliably as devices without an INTx disable bit will then generate a
* level IRQ which will never be cleared.
*/
u32 __pci_msi_desc_mask_irq(struct msi_desc *desc, u32 mask, u32 flag)
{
u32 mask_bits = desc->masked;
if (pci_msi_ignore_mask || !desc->msi_attrib.maskbit)
return 0;
mask_bits &= ~mask;
mask_bits |= flag;
pci_write_config_dword(msi_desc_to_pci_dev(desc), desc->mask_pos,
mask_bits);
return mask_bits;
}
From the highlight comments ,seems the bit mask may not work as expected, so I think IMX8 PCIe MSI controller should mask/unmask the source IRQ whenever PCIe device enable/disable their MSI.
So could you please help to confirm that: when calling disable IRQ, does it actually mask the IRQ at the interrupt controller on NXP platform or does it just go over PCIe link and mask MSI of QC WLAN chip? I think your side should do the latter, so wlan chip RX interrupt will continue report to kernel side. If it’s the latter, could you please raise a fix for it.
Thank you very much!
Hi Qun:
4.14 Kernel seem not implement MSI bottom interrupt mask/unmask feature.
Could you try Kernel 4.19.35 which has supported this feature ?
Hi Tom,
From 4.19.35 kernel link:
https://source.codeaurora.org/external/imx/linux-imx/log/?h=imx_4.19.35_1.1.0
Seems the drivers/pci/dwc folder was removed, right? if so, it should have a big difference with kernel 4.14.
So where is IMX8 PCIe controller driver located in kernel 4.19.35? Thanks!
Regards,
Qun
Hi Qun:
PCI dwc controller directory moved to drivers/pci/controller/dwc.