topic Re: i.MX6 PCIe interrupt mapping changes during runtime in i.MX Processors

i.MX6 PCIe interrupt mapping changes during runtime

andreasgeißler — Wed, 22 Aug 2018 13:39:04 GMT

Hello,

I am using an i.MX6qdl processor on a customer platform. An FPGA is connected to the PCIe interface to communicate with the internal IP's, using legacy interrupts. We figured out, that when running long-term tests sometimes the interrupt mapping seems to be changed for some reason. The interrupt of the FPGA is mapped to irq 300 after booting. After starting the long-term tests do perform traffic on the PCIe bus, within a few days (It does not occurs very often), it happens that the interrupt of the FPGA is shown up on irq 299 where the handler is not registered and nobody cares about the interrupt:

72: 0 0 0 0 GPC 114 Level mmdc_1

73: 0 0 0 0 GPC 8 Level 2800000.ipu

74: 0 0 0 0 GPC 7 Level 2800000.ipu

241: 0 0 0 0 gpio-mxc 6 Edge ad7606

299: 100000 0 0 0 GPC 123 Level PCIe PME

300: 544352 0 0 0 GPC 121 Level 16Z087

301: 16 0 0 0 GIC-0 137 Level 2101000.jr0

[171521.126091] irq 299: nobody cared (try booting with the "irqpoll" option)

…

[171521.135735] Disabling IRQ #299

We already know that we have to disable MSI for legacy interrupts otherwise nothing would work. After the first received wrong interrupt the IRQ gets disabled and the communication with the FPGA stops working. We check the interrupt behavior also manually after the error case and it really shows that the interrupt of the FPGA is shown on a different bit of the GIC:

Reading CPU irq status register of CPU before error:

root@men-cc10:~# memrw 0x020dc024 l
Value at 0x20dc024 (0x76ff7024): 0x82000000

Reading CPU irq status register of CPU after error:

root@men-cc10:~# memrw 0x020dc024 l
Value at 0x20dc024 (0x76f16024): 0x84000000

Disable irq within FPGA and read status register of CPU again:

root@men-cc10:~# memrw 0x020dc024 l
Value at 0x20dc024 (0x76f0b024): 0x80000000

Reenable irq within FPGA and read status register of CPU again:

root@men-cc10:~# memrw 0x020dc024 l
Value at 0x20dc024 (0x76ff4024): 0x84000000

- Did we missed something when disabling the MSI?

- Or is this an already known issue with an existing workaround?

- What can be the reason for this behavior?

Regards,

Andreas

Re: i.MX6 PCIe interrupt mapping changes during runtime

Yuri — Tue, 28 Aug 2018 09:47:33 GMT

Hello,

What Linux release is used in the case?

Regards,

Yuri.

Re: i.MX6 PCIe interrupt mapping changes during runtime

andreasgeißler — Tue, 28 Aug 2018 09:52:10 GMT

Hello Yuri,

Thanks for fast reply.

We are using the latest version of Yocto (sumo branch). With the corresponding meta-freescale layer. The used kernel version is v4.9.88 with the recipe from meta-freescale.

Regards,

Andreas

B. Eng.

Andreas Geißler

FPGA Engineer

MEN Mikro Elektronik GmbH

Neuwieder Straße 3-7

90411 Nürnberg, Germany

Phone +49 911 99 33 5 - 230

Fax +49 911 / 99 33 5 - 901

Andreas.Geissler@men.de<mailto:Andreas.Geissler@men.de>

www.men.de<http://www.men.de>;

Subscribe to our newsletter<https://www.men.de/news-media/newsletter/>

MEN Mikro Elektronik GmbH - Bernd Härtlein (CEO) - Yilmaz Kocak (CFO) - Handelsregister/Trade Register AG Nürnberg HRB 5540

Please consider the environment before printing this e-mail

Re: i.MX6 PCIe interrupt mapping changes during runtime

gfine — Tue, 28 Aug 2018 15:09:03 GMT

Hi andreasgeißler‌,

We can not proceed any further without the version and source of the Linux OS being used.

BR,

Glen

Re: i.MX6 PCIe interrupt mapping changes during runtime

andreasgeißler — Tue, 28 Aug 2018 15:14:36 GMT

Hi Yuri and Glen,

Sorry, i have replied on the email and it seems that there is a problem with our signature. Here is what i have sent:

Thanks for fast reply.

We are using the latest version of Yocto (sumo branch). With the corresponding meta-freescale layer. The used kernel version is v4.9.88 with the recipe from meta-freescale.

Regards,

Andreas

Re: i.MX6 PCIe interrupt mapping changes during runtime

gfine — Tue, 28 Aug 2018 16:33:26 GMT

Hi andreasgeißler,

What did you use as your source? The one below?

https://source.codeaurora.org/external/imx/imx-manifest/tree/?h=imx-linux-rocko

I was not aware we have a 'sumo' build that we support at this time. I think 'sumo' is reserved for a future release.

Can you try using the 'rocko' release with 4.9.88 and get back to me as ASAP

BR,

Glen

Re: i.MX6 PCIe interrupt mapping changes during runtime

andreasgeißler — Wed, 29 Aug 2018 11:27:29 GMT

Hi Glen,

We have used the sumo branch version from:

http://git.yoctoproject.org/cgit.cgi/meta-freescale/?h=sumo

to build the Yocto project.

I will try to reproduce the issue with the version from:

https://github.com/Freescale/fsl-community-bsp-platform

but it seems that this rocko branch version only supports kernel version v4.9.11.

And it will take some time to get back with new results because i have to adapted our layers and recipes. In addion the issue does not happen very often it occurs once within 3 days running the iperf tests.

Regards,

Andreas

Re: i.MX6 PCIe interrupt mapping changes during runtime

gfine — Thu, 30 Aug 2018 15:44:56 GMT

Hi andreasgeißler,

As mentioned we do not support anything in the sumo release.

Our 4.9.88 is available on

https://source.codeaurora.org/external/imx/imx-manifest/tree/README?h=imx-linux-rocko

The fsl-community (which you have linked to in your reply) is supported by Octavio Salvador and we, NXP, have no control on its content. Hence we do not support it.

The content we do support is on the Code Aurora link above. If you can reproduce the problem with the NXP supported BSP we can work on it.

Anyhow, a drifting interrupt indicates there is something amiss in that kernel's interrupt handler. You may want to make Octavio aware of what you found.

BR,

Glen

Re: i.MX6 PCIe interrupt mapping changes during runtime

andreasgeißler — Mon, 03 Sep 2018 15:05:23 GMT

Hi Glen,

I have now started our test on the NXP supported BSP. I took a while to get everything working again. I will keep you informed about any results of the test. Let's see if it also occurs on this BSP.

Find attached the dmesg output with the NXP supported BSP.

Regards,

Andreas

Re: i.MX6 PCIe interrupt mapping changes during runtime

gfine — Tue, 04 Sep 2018 15:38:40 GMT

Hi Andreas,

I can understand the process (and pains) of rebuilding the OS.

But now I can support you if you recreate the same problem.

Cheers,

Glen

Re: i.MX6 PCIe interrupt mapping changes during runtime

andreasgeißler — Wed, 05 Sep 2018 08:41:50 GMT

Hi Glen,

I could reproduce the problem also with the Code Aurora BSP. The interrupt of the FPGA was disabled after 21h, while running the long term load tests. It seems to be a little bit different this time but I guess the source of the problem is the same:

The IRQ becomes disabled because no handler cared about:

[75800.350797] irq 300: nobody cared (try booting with the "irqpoll" option)
[75800.356296] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 4.9.88-imx_4.9.88_2.0.0_ga #2
[75800.363865] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[75800.369119] [<801108b4>] (unwind_backtrace) from [<8010c1b0>] (show_stack+0x10/0x14)
[75800.375573] [<8010c1b0>] (show_stack) from [<803ccb2c>] (dump_stack+0x78/0x8c)
...
[75800.585058] [<8010cb8c>] (__irq_svc) from [<8076ca28>] (cpuidle_enter_state+0x13c/0x2cc)
[75800.591855] [<8076ca28>] (cpuidle_enter_state) from [<8016a2a0>] (cpu_startup_entry+0x168/0x228)
[75800.599351] [<8016a2a0>] (cpu_startup_entry) from [<80f00c58>] (start_kernel+0x374/0x380)
[75800.606225] handlers:
[75800.607209] [<7f0604c0>] z77_irq [men_lx_z77]
[75800.610288] [<7f0604c0>] z77_irq [men_lx_z77]
[75800.613365] [<7f057930>] men_z135_intr [men_lx_z135]
[75800.617045] [<7f057930>] men_z135_intr [men_lx_z135]
[75800.620721] Disabling IRQ #300

This time it seems that an other interrupt which is not the FPGA occurred on the same interrupt vector:

241:          0          0          0          0 gpio-mxc   6 Edge      ad7606
299:          0          0          0          0       GPC 123 Level     PCIe PME
300: 1283785018          0          0          0       GPC 121 Level     16Z087, 16Z087, men_z135_intr, men_z135_intr
301:         16          0          0          0     GIC-0 137 Level     2101000.jr0
302:          0          0          0          0     GIC-0 138 Level     2102000.jr1

What is interesting is that a second interrupt is still active in the error case:

root@imx6qsabresd:~/log# memrw 0x020dc014 l
Value at 0x20dc014 (0x76efd014): 0xF738F7FF
root@imx6qsabresd:~/log# memrw 0x020dc024 l
Value at 0x20dc024 (0x76fbd024): 0x83000000

When disabling the interrupt of the FPGA the CPU register looks like followed (means FPGA IRQ is 0x01000000):

root@imx6qsabresd:~/log# memrw 0x020dc024 l
Value at 0x20dc024 (0x76fae024): 0x82000000

After reboot the interrupt of the FPGA appears on an other interrupt as at the error case (FPGA IRQ is 0x02000000):

Value at 0x20dc024 (0x76fd5024): 0x82000000

Unfortunately, I have forgotten to check on with interrupt the FPGA is before the error appeared. So I started yesterday a second test, do ensure that the interrupt of the FPGA is really changing. No issue so far on this iteration.

Please find attached the full dmesg log and the interrupt behavior.

Please let me know if you need other information.

Regards,

Andreas

Re: i.MX6 PCIe interrupt mapping changes during runtime

gfine — Thu, 06 Sep 2018 00:00:11 GMT

Hi andreasgeißler,

I have created a defect ticket for this (MLK-19463). I know the developers are probably going to ask how to recreate, and ask for details about how the FPGA and PCIe are being mapped or if a custom driver is involved.

BR,

Glen

Re: i.MX6 PCIe interrupt mapping changes during runtime

gfine — Fri, 07 Sep 2018 20:35:18 GMT

Hi andreasgeißler,

Our developer asked :

Regarding to the log, the irq123 (299) that nobody cared about, and it is used by the PCIe PME driver. Can the customer build-out the PCIe PME driver, then do the tests again? Thus, it would be helpful to see the result of the debug this issue.

I think, in a nutshell, he is asking if you can rebuild the PCIe PME driver separately, compile with DEBUG turned on, and report your findings.

Is this possible?

BR,

Glen

Re: i.MX6 PCIe interrupt mapping changes during runtime

andreasgeißler — Tue, 11 Sep 2018 07:41:49 GMT

Hello Glen,

I will give it a try. With separately you mean as a loadable kernel module?
And by the way I tried to reproduce the problem again. and this time it took 5 days. The occurence can be also very rare.

It will take sometime to come back with the result of the DEBUG message. I will keep you informed

Thanks,

Andreas

Re: i.MX6 PCIe interrupt mapping changes during runtime

gfine — Tue, 11 Sep 2018 15:39:38 GMT

Hi andreasgeißler‌,

Yes, as a LKM with DEBUG turned on.

Let me know as soon as the error recurs.

Cheers,

Glen

Re: i.MX6 PCIe interrupt mapping changes during runtime

gfine — Thu, 20 Sep 2018 18:41:05 GMT

Hi andreasgeißler‌,

It has been a week. Any luck in recreating the problem?

BR,

Glen

Re: i.MX6 PCIe interrupt mapping changes during runtime

andreasgeißler — Mon, 01 Oct 2018 14:53:12 GMT

Hi glen,

I have implemented the changes today and started the test again. But i am not sure if this is realy enough to get some new information. The driver has only a few debug messages. And i could not build the driver as a kernel module, because this is not provided by the kernel config. it seems to be a static library?

Could I do anything else to get more detailed debug messages for the issue?

Regards,

Andreas

Re: i.MX6 PCIe interrupt mapping changes during runtime

andreasgeißler — Fri, 12 Oct 2018 08:34:27 GMT

Hello Glen,

I have added the DEBUG to the PCI PME driver and could reproduce the problem again. I have attached the dmesg log, but could not anything new or something else what could help to find the reason for the problem

Regards,

Andreas

Re: i.MX6 PCIe interrupt mapping changes during runtime

gfine — Fri, 12 Oct 2018 15:26:51 GMT

HI andreasgeißler‌,

R&D replied on the query about building PCIe into the kernel.

What I meant was not to compile the PCIe driver to a KLM.
Just don't build the PCIe PME into kernel.
config PCIE_PME
def_bool y
depends on PCIEPORTBUS && PM

BR,

Glen

Re: i.MX6 PCIe interrupt mapping changes during runtime

gfine — Fri, 12 Oct 2018 15:44:44 GMT

Hi Andreas,

Pushed the dmesg to R&D. Should have a reply in a few days.

I can see the IRQ assignment (and reassignments) starting at timestamp [ 0.597072].

BR,

Glen