Legacy Interrupts in PCI have become a thing of the past in the PCI world, now that we have more efficient MSI interrupts in PCI Express, being used extensively across applications. However they are still used primarily for backward compatibility and boot-up support.
Legacy Interrupts[INTx] used to be physical signals in PCI, however PCIe devices do not have physical INTx pins. So to support these legacy interrupts, PCIe emulates them logically. Almost like a virtual INTx mechanism. We shall see how these interrupts are realized in PCI Express, in such a way that developers/enthusiasts can get a hands-on following this guide. This blog has the following agendas:-
1. Why do we need to care about Legacy INTx style interrupts in PCIe?
- System Overview
- How are INTx legacy interrupts realized in PCIe?
- Enabling and testing INTx emulation on iMX8MM
Why do we need to care about Legacy INTx style interrupts in PCIe?
Here are some of the reasons why even today it is important to understand Legacy INTx interrupts: -
- Many operating systems, drivers, and applications were originally designed for PCI and expect INTx-style interrupts. They may not support MSI/MSIX
- When a PCIe device is used in a system that expects legacy PCI behavior, INTx emulation ensures the device can still signal interrupts in the old way. This avoids breaking older software stacks that do not support MSI/MSI-X.
- As a fallback mechanism, If MSI/MSI-X cannot be allocated (due to resource constraints or configuration issues), INTx emulation proves to be helpful.
System Overview
The system under test has the following components: -
Hardware Components: -
iMX8MM EVK board - i.MX 8M Mini Evaluation Kit | NXP Semiconductors
iMX95 19x19 EVK - i.MX 95 Applications Processors Family | NXP Semiconductors
Software Components: -
GitHub - nxp-imx/linux-imx: i.MX Linux kernel
Linux BSP on iMX95 RC - 6.6.3
Linux BSP on iMX8MM EP - 6.1.55
As evident in Figure 1, the system comprises of iMX8MM EVK and iMX95 19x19 EVK connected back-to-back via a M.2 Key E PCIe bridge.
iMX8MM EVK will be the PCIe Endpoint, while iMX95 EVK will act as a PCIe root complex.
iMX95 <------> iMX8MM
[RC] [EP]
To set it up in the above configuration, please follow this blog -
Enabling PCIe End-point framework on iMX95 torradex board and iMX8MM EVK - NXP Community
After this, you would successfully be able to trigger pcie transactions between iMX8MM and iMX95 EVKs using the pci endpoint test framework.
With this setup, we will demonstrate how a PCIe EP[iMX8MM EVK] sends legacy INTx interrupt to the PCIe RC[iMX95 EVK]. We will also trace the origin of these interrupts in the linux driver on linux-imx source code.
How are INTx legacy interrupts realized in PCIe?
PCI Express supports PCI Interrupts as defined in the PCI Local Bus Specification. It includes the Interrupt Pin and Interrupt Line registers of PCI Configuration Space. PCIe devices support these registers for backward compatibility. So the actual interrupt signaling uses an in-band messages rather than using physical pins.
In PCIe two types of messages are defined as per the specification:-
Assert_INTx and Deassert_INTx for emulating PCI INTx signaling
x -> A/B/C/D
these messages act as "virtual wires" for signaling interrupts across a link. These "virtual wires" in form of messages reach 'Root Complex' which then gets mapped to the system interrupt controller. This means that when RC receives these in-band "virtual wire" messages, it translates it into appropriate hardware interrupt going to a CPU.
Note:- Mapping of PCI Express INTx emulation to system interrupts is purely implementation specific. Similar to physical interrupt signals, the INTx emulation mechanism may potentially cause spurious interrupts which must be handled by the system software.
Enabling and testing INTx emulation on iMX8MM
On the official linux factory kernel, the legacy interrupts are not yet supported. There is a zip file attached with this blog that contains the patches to enable the legacy interrupts in PCIe. It has 3 files: -
1. legacy_intx_tested_imx8mm_patch_1
- legacy_intx_tested_imx8mm_patch_2
- README
The instructions on how to apply the patches are given in the README file. Please feel free to DM me if you encounter any issues.
So long story short, steps to enable PCIe INTx emulation in linux:-
1. Flash the default iMX8MM 6.1.55 wic image on the board and make sure it boots to Linux.
2. Fetch linux-imx repo [checkout branch lf-6.1.y]
3. Apply the patch on the Linux source code.
4. Build the kernel and scp 'arch/arm64/boot/Image' to the board. Boot the board with this 'Image' and 'imx8mm-evk-pcie-ep.dtb'
Feel free to refer this blog in case you observe any issues - Enabling PCIe End-point framework on iMX95 torradex board and iMX8MM EVK
At this point, the kernel running on PCIe EP iMX8MM is capable of sending INTx interrupts to the PCIe RC[iMX95]. Here's how you test it:-
1. Make sure iMX95 RC is connected via M.2 with iMX8MM EP. Both of them are booted up and iMX8MM is visible as a pcie endpoint on iMX95. Execute 'lspci' on iMX95 and you should see an entry with 'Unassigned class'.
2. On iMX95EVK RC, execute the following 2 commands to test the INTx emulation: -
a. pcitest -i 0
b. pcitest -l
Console output: -
root@imx95evk:~# pcitest -i 0
SET IRQ TYPE TO LEGACY: OKAY
root@imx95evk:~# pcitest -l
LEGACY IRQ: OKAY
'pcitest -i 0' configures the RC to handle Legacy INTx interrupts. As a result of this, an ioctl call will be made 'PCITEST_SET_IRQTYPE':-
Handled as follows in drivers/misc/pci_endpoint_test.c
arg is 0 for IRQ_TYPE_LEGACY
Have a look at the below snippet which shows where it all starts: -
Inside pci_endpoint_test_release_irq
test->num_irqs=16 in our case, because on end-point :-
echo 16 > functions/pci_epf_test/func1/msi_interrupts
pci_irq_vector returns the linux IRQ number corresponding to the MSI Interrupt 0 to 16
Then, it frees those irqs using devm_free_irq so that they can be allocated again.
free_irq is used inside this function which removes an interrupt handler
and if the interrupt line is no longer in use by any driver it is disabled.
On a shared IRQ the caller must ensure the interrupt is disabled on the card it drives before calling this function.
Inside pci_endpoint_test_free_irq_vectors
It calls pci_free_irq_vectors [drivers/pci/msi/api.c] to :-
Undo the interrupt vector allocations and possible device MSI/MSI-X enablement earlier done through pci_alloc_irq_vectors_affinity() or pci_alloc_irq_vectors().
Note :- One thing to understand is that in the pci endpoint test driver, before allocating irq vectors and requesting for legacy intx interrupts, we are freeing the already allocated MSI interrupts and de-allocating the vectors associated with it. In the process, we are disabling the MSI interrupts. This is because MSI and INTx are mutually exclusive as per PCIe specification. It will be an interesting exercise to check what happens if we try to enable both MSI and INTx. Let me know in the comments if you happen to test this.
Then, inside pci_endpoint_test_alloc_irq_vectors
pci_alloc_irq_vectors allocates interrupt vectors for legacy intx interrupts. Deep inside, this creates a cpu affinity mask for cpu core 0 and then pci_intx is called to enable the legacy interrupt by reading and writing to pcie config space:-
Inside pci_endpoint_test_request_irq
pci_irq_vector allocates a linux IRQ number which is passed to the
devm_request_irq -> devm_request_threaded_irq
devm_request_irq in the Linux kernel is part of the device-managed resource allocation API. Its primary role is to simplify interrupt request handling for device drivers by automatically managing the lifecycle of the IRQ resource. It simply registers an interrupt handler for an IRQ number. It ties the IRQ resource to the device's lifecycle. When the device is detached or the driver is unloaded, the kernel automatically frees the IRQ
It gets passed a handler 'pci_endpoint_test_irqhandler' with the IRQF_SHARED flag. This handler will be called when there's an incoming INTx interrupt.
IRQF_SHARED -- enables sharing of an interrupt line with several devices
Note- Discussing Linux IRQ subsystem is outside the scope of this blog, so do not worry if you feel overwhelmed by some of the kernel APIs. We will be covering it in one of our upcoming blogs.
Now we have got some idea of what goes on behind the scenes when we enable INTx in PCIe.
'pcitest -l' triggers the legacy IRQ test which returns OKAY if everything goes fine. If not, it returns NOT OKAY
When you execute the above, PCIe RC sends a command 'COMMAND_RAISE_LEGACY_IRQ' to iMX8MM EP on the pcie link. EP on getting this command as part of its handler 'pci_epf_test_cmd_handler' calls 'pci_epf_test_raise_irq'
pci_epc_raise_irq, ultimately calls __dw_pcie_ep_raise_intx_irq of the patch.
This sends PCI_CODE_ASSERT_INTA message followed by a delay and then PCI_CODE_DEASSERT_INTA. Both these messages are of routing type PCI_MSG_ROUTING_LOCAL, meaning that the message is routed to a specific function within the same device or to a switch. Unlike memory transactions that need a target address, INTx messages do not have a destination address.
In easy words, it says that "Route this to my local Root complex".
It then reaches the Root Complex iMX95. From there it reaches the interrupt controller [GICv3] using Shared Peripheral Wired Interrupt. The GIC informs one of the cpu cores to handle the interrupt. In our case CPU 0 will be handling it since we have set its affinity to CPU 0. At last pci_endpoint_test_irqhandler will be called in the RC's endpoint test framework.
Following is the sequence once the interrupt reaches GIC
If you connect a JTAG to the iMX95EVK board, you can actually trace the calls when legacy INTx IRQs are handled in linux:-
The interrupts routed to the GIC on imx95 can be observed by executing
'cat /proc/interrupts'
every time you trigger a legacy interrupt test using 'pcitest -l'
you will see the following in the output of 'cat /proc/interrupts' in case of INTA
The number 3 against CPU0[which handled the interrupt], you will see this will increment every time you trigger a legacy interrupt test. It denotes how many interrupts were generated for the IRQ number on CPU core
Level - type of interrupt
224 is the Interrupt request number
//Testing INTA,INTB,INTC and INTD
Single Function devices support only 1 INTx Interrupt[Emulated in case of PCIe]
To use INTB, INTC or INTD
- In drivers/pci/controller/dwc/pcie-designware-ep.c
Figure. 16
The 3rd argument of the fn __dw_pcie_ep_raise_intx_irq
0 = INTA
1 = INTB
2 = INTC
3 = INTD
Modify it according to above mapping
- In drivers/pci/endpoint/functions/pci-epf-test.c
Figure. 17
You can change the interrupt_pin member of the test_header
PCI_INTERRUPT_INTA = INTA
PCI_INTERRUPT_INTB = INTB
PCI_INTERRUPT_INTC = INTC
PCI_INTERRUPT_INTD = INTD
-- Monitoring Interrupt for INTA
-- Monitoring Interrupt for INTB
-- Monitoring Interrupt for INTC
-- Monitoring Interrupt for INTD
From the above snippets, you must have observed that only core 0 is handling the INTx IRQ. This is because the cpu affinity mask is set to 1 which means cpu core 0.
in kernel/irq/affinity.c
The iRQ mask for legacy interrupts is set to 1 - meaning core 0
That's a wrap for this blog. It was a considerably long blog covering a lot of details. However do not fret if you get overwhelmed by any of the details. Eventually when you start working on PCIE and IRQ linux subsystem, you will gather a good understanding. Please let me know if you have any queries on the subject. Feel free to DM or email. See you next time.
