mask MSI interrupt doesn't work on IMX8

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

mask MSI interrupt doesn't work on IMX8

2,267 Views
qunz
Contributor I

The current used platform: IMX8 EVK/QXP board

PCIe wlan chip: Qualcomm qca6696

Linux kernel version: L4.14.78_1.0.0_MX8QXP (REV L4.14.78_1.0.0 )   L4.14.78_1.0.0_ga

Image version: 2019-01-03 17:47:00 L4.14.78_1.0.0_MX8QXP

Hi Sir,

I am integrating Qualcomm WLAN chip to IMX8 platform, and found the disable_irq_nosync for MSI interrupt doesn’t work as expected.

The background is:

  1. Our WLAN chip driver have multiple MSI interrupts through DMA copy engine
  2. In WLAN RX data path, when receive RX data interrupt, wlan host driver will disable the MSI interrupt firstly, then schedule kernel NAPI softirq to process RX data. When NAPI schedule processing finished, will re-enable the MSI interrupt again. Plz find below call stack:

[  104.930753] [<ffff0000084db030>] msi_set_mask_bit.isra.7+0xa8/0xd0

[  104.930758] [<ffff0000084db070>] pci_msi_mask_irq+0x18/0x20

[  104.930763] [<ffff000008120434>] __irq_disable+0x94/0xa8

[  104.930768] [<ffff000008120fe4>] irq_disable+0x14/0x20

[  104.930775] [<ffff00000811d204>] __disable_irq_nosync+0x5c/0x68

[  104.930781] [<ffff00000811d21c>] disable_irq_nosync+0xc/0x18                                                      ->  disable MSI interrupt

[  104.933638] [<ffff0000010cc440>] pld_srng_disable_irq+0x38/0x80 [wlan_cnss0]

[  104.936499] [<ffff000001017014>] hif_ce_srng_msi_irq_disable+0x1c/0x28 [wlan_cnss0]

[  104.939352] [<ffff0000010134f0>] hif_irq_disable+0x10/0x18 [wlan_cnss0]

[  104.942218] [<ffff000001021920>] ce_dispatch_interrupt+0x40/0x278 [wlan_cnss0]

[  104.945068] [<ffff000001016ad0>] hif_ce_interrupt_handler+0x10/0x18 [wlan_cnss0]                 -> wlan receive interrupt

3. I found the disabled MSI action doesn’t take effect as expected,  the RX interrupt still continue to send to kernel, so the NAPI softirq processing is always preempted by RX interrupt and others tasks don’t have chances to be scheduled,  kernel side always have RCU stall report. Plz find below RCU stall info:

[  850.313246] INFO: rcu_preempt self-detected stall on CPU

[  850.313260]  0-...: (5053 ticks this GP) idle=66e/140000000000001/0 softirq=4987/4987 fqs=2133

[  850.313261]   (t=5250 jiffies g=1405 c=1404 q=1269)

[  850.313268] Task dump for CPU 0:

[  850.313272] ksoftirqd/0     R  running task        0     7      2 0x00000022

[  850.313280] Call trace:

The issue can reproduce on I.MX8 EVK/QXP two boards with QCOM WIFI chip QCA6696. I didn’t verify on i.MX6 series, but I think it should be same.

Actually  we also porting other QCOM WIFI chip to i.MX6/i.MX8 before,  all can work up, this is because these chips use legacy PCIe interrupt or single MSI interrupt. The QCA6696 chip has more then ten MSI interrupts and use NAPI softirq for RX data, so need disable related MSI interrupt in each processing.

In IMX8 kernel 4.14, CONFIG_PCI_MSI_IRQ_DOMAIN Is defined, there should be a hierarchy layer for MSI irq, just like below, all WLAN MSI interrupts are registered to mx6-pcie-msi layer, so there are two layers interrupt for PCIe MSi, GIC and PCIe. In QCOM kernel, we have private processing for this part.

root@imx8qxpmek:~# cat /proc/interrupts

108:        825          0          0          0     GICv3 134 Level     mx6-pcie-msi

445:          3          0          0          0   PCI-MSI  34 Edge      WLAN_CE_0

446:          2          0          0          0   PCI-MSI  35 Edge      WLAN_CE_1

447:        100          0          0          0   PCI-MSI  36 Edge      WLAN_CE_2

448:         93          0          0          0   PCI-MSI  37 Edge      WLAN_CE_3

450:          1          0          0          0   PCI-MSI  39 Edge      WLAN_CE_5

451:          0          0          0          0   PCI-MSI  40 Edge      WLAN_CE_6

453:          0          0          0          0   PCI-MSI  42 Edge      WLAN_CE_8

457:          0          0          0          0   PCI-MSI  46 Edge      wlan_EXT_GRP

458:          0          0          0          0   PCI-MSI  47 Edge      wlan_EXT_GRP

459:          0          0          0          0   PCI-MSI  48 Edge      wlan_EXT_GRP

460:          0          0          0          0   PCI-MSI  49 Edge      wlan_EXT_GRP

461:          0          0          0          0   PCI-MSI  50 Edge      wlan_EXT_GRP

462:          0          0          0          0   PCI-MSI  51 Edge      wlan_EXT_GRP

463:          0          0          0          0   PCI-MSI  52 Edge      wlan_EXT_GRP

 

In IMX8 current kernel 4.14,  disable_irq_nosync will call __pci_msi_desc_mask_irq finally:

 

/*

* PCI 2.3 does not specify mask bits for each MSI interrupt.  Attempting to

* mask all MSI interrupts by clearing the MSI enable bit does not work

* reliably as devices without an INTx disable bit will then generate a

* level IRQ which will never be cleared.

*/

u32 __pci_msi_desc_mask_irq(struct msi_desc *desc, u32 mask, u32 flag)

{

                u32 mask_bits = desc->masked;

 

                if (pci_msi_ignore_mask || !desc->msi_attrib.maskbit)

                                return 0;

 

                mask_bits &= ~mask;

                mask_bits |= flag;

                pci_write_config_dword(msi_desc_to_pci_dev(desc), desc->mask_pos,

                                                       mask_bits);

 

                return mask_bits;

}

From the highlight comments ,seems the bit mask may not work as expected,  so  I think IMX8 PCIe MSI controller should  mask/unmask the source IRQ whenever PCIe device enable/disable their MSI.

So could you please help to confirm that:  when calling disable IRQ, does it actually mask the IRQ at the interrupt controller on NXP platform or does it just go over PCIe link and mask MSI of QC WLAN chip?  I think your side should do the latter, so  wlan chip RX interrupt will continue report to kernel side. If it’s the latter, could you please raise a fix for it.

Thank you very much!

0 Kudos
5 Replies

2,154 Views
haidong_zheng
NXP Employee
NXP Employee

Hi Qun:

4.14 Kernel seem not implement MSI bottom interrupt mask/unmask feature.  

Could you try Kernel 4.19.35 which has supported this feature ? 

0 Kudos

2,154 Views
qunz
Contributor I

Hi Tom,

From 4.19.35 kernel link:

https://source.codeaurora.org/external/imx/linux-imx/log/?h=imx_4.19.35_1.1.0

Seems the drivers/pci/dwc folder was removed, right?  if so, it should have a big difference with kernel 4.14.

So where is IMX8 PCIe controller driver located in kernel 4.19.35? Thanks!

Regards,

Qun

0 Kudos

2,154 Views
haidong_zheng
NXP Employee
NXP Employee

Hi Qun:

PCI dwc controller directory moved to drivers/pci/controller/dwc. 

0 Kudos

2,154 Views
Bio_TICFSL
NXP TechSupport
NXP TechSupport

Good!

0 Kudos

2,154 Views
qunz
Contributor I

Hi Tom,

Thanks very much for your help!

Could you point to me which change enabled the feature on kernel 4.19.35?  I can take a quick verification.

Regards,

Qun

0 Kudos