LS1043A handle MSI interrupt with msi-controller

draven — Sat, 31 Dec 2022 12:06:22 GMT

Hi forum,

There is an LS1043A and a fpga on my setup. I can communicate between ls1043 and fpga over pcie. LS1043A is root complex and fpga is end-point at my setup.

lspci output on my ls1043a :

0000:01:00.0 Memory controller: Xilinx Corporation Device 7011 Subsystem: Xilinx Corporation Device 0007 Flags: bus master, fast devsel, latency 0 Memory at 4040000000 (32-bit, non-prefetchable) [size=64K] Memory at 4040010000 (32-bit, non-prefetchable) [size=64K] Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00

lspci on my fpga output :

00:00.0 Memory controller: Xilinx Corporation Device 7011 Subsystem: Xilinx Corporation Device 0007 Flags: bus master, fast devsel, latency 0 Memory at <ignored> (32-bit, non-prefetchable) Memory at <ignored> (32-bit, non-prefetchable) Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00 Capabilities: [128] Vendor Specific Information: ID=0001 Rev=0 Len=038 <?>

I can communicate between ls1043a and fpga bidirectional with devmem on 0x4040000000 address.

LS1043a : $devmem 0x4040000000 32 , output : 0x0000000 LS1043A: $devmem 0x4040000000 32 0xDEAD LS1043a : $devmem 0x4040000000 32 , output : 0xDEAD FPGA : $devmem 0x40000000 32 , output : 0xDEAD

I want to send MSI interrupt from ls1043a to fpga and also vice-versa. I know that there is an driver to handle msi interrupt on ls1043 as "irq-ls-scfg-msi.c".

My devicetree related with msi :

msi1: msi-controller1@1571000 { compatible = "fsl,ls1043a-msi"; reg = <0x0 0x1571000 0x0 0x8>; msi-controller; interrupts = <0 116 0x4>; }; msi2: msi-controller2@1572000 { compatible = "fsl,ls1043a-msi"; reg = <0x0 0x1572000 0x0 0x8>; msi-controller; interrupts = <0 126 0x4>; }; msi3: msi-controller3@1573000 { compatible = "fsl,ls1043a-msi"; reg = <0x0 0x1573000 0x0 0x8>; msi-controller; interrupts = <0 160 0x4>; };

And also I added printk to "irq-ls-scfg-msi.c" driver to check if probe and setup_irq methods calling and working. So driver is probed and setup all msi interrupts.

My "irq-ls-scfg-msi.c" driver's debug outputs are on dmesg : (all output's last string function name and last number is file number index)

root@tqmls1043a-mbls10xxa:~# dmesg | grep MSI [ 2.242754] 003: MSI DEBUG: msir->gic_irq : 51, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.254258] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.261211] 003: MSI DEBUG: msir->gic_irq : 52, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.272709] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.279663] 003: MSI DEBUG: msir->gic_irq : 53, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.291156] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.298102] 003: MSI DEBUG: msir->gic_irq : 54, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.309588] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.316542] 003: MSI DEBUG: probe success ls_scfg_msi_probe 406 [ 2.322712] 003: MSI DEBUG: msir->gic_irq : 55, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.334200] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.341147] 003: MSI DEBUG: msir->gic_irq : 56, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.352632] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.359578] 003: MSI DEBUG: msir->gic_irq : 57, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.371063] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.378008] 003: MSI DEBUG: msir->gic_irq : 58, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.389493] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.396445] 003: MSI DEBUG: probe success ls_scfg_msi_probe 406 [ 2.402607] 003: MSI DEBUG: msir->gic_irq : 59, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.414095] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.421042] 003: MSI DEBUG: msir->gic_irq : 60, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.432529] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.439475] 003: MSI DEBUG: msir->gic_irq : 61, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.450959] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.457905] 003: MSI DEBUG: msir->gic_irq : 62, msi_data->msir_num = 4, msi_data->irqs_num = 128, ls_scfg_msi_setup_hwirq 271 [ 2.469391] 003: MSI DEBUG: setup irq success ls_scfg_msi_setup_hwirq 286 [ 2.476337] 003: MSI DEBUG: probe success ls_scfg_msi_probe 406

But I cannot find msi interrupts on linux userspace. /proc/interrupt's output is :

root@tqmls1043a-mbls10xxa:~# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 1: 0 0 0 0 GICv2 25 Level vgic 3: 18291 29847 30490 47937 GICv2 30 Level arch_timer 4: 0 0 0 0 GICv2 27 Level kvm guest vtimer 6: 0 0 0 0 GICv2 138 Level arm-pmu 7: 0 0 0 0 GICv2 139 Level arm-pmu 8: 0 0 0 0 GICv2 127 Level arm-pmu 9: 0 0 0 0 GICv2 129 Level arm-pmu 11: 5 0 0 0 GICv2 131 Level 1550000.spi 12: 2007 0 0 0 GICv2 94 Level mmc0 13: 0 0 0 0 GICv2 176 Level [EDAC] MC err 15: 0 0 0 0 GICv2 77 Level bman-err, qman-err, fman-err 16: 0 0 0 0 GICv2 205 Level BMan portal 0 17: 0 0 0 0 GICv2 207 Level BMan portal 1 18: 0 0 0 0 GICv2 209 Level BMan portal 2 19: 0 0 0 0 GICv2 211 Level BMan portal 3 26: 0 0 0 0 GICv2 204 Level QMan portal 0 27: 0 0 0 0 GICv2 206 Level QMan portal 1 28: 0 0 0 0 GICv2 208 Level QMan portal 2 29: 0 0 0 0 GICv2 210 Level QMan portal 3 36: 0 0 0 0 GICv2 96 Level 2100000.spi 37: 224 0 0 0 GICv2 88 Level 2180000.i2c 38: 0 0 0 0 GICv2 91 Level 21b0000.i2c 39: 5243 0 0 0 GICv2 86 Level ttyS1 45: 0 0 0 0 GICv2 115 Level 2ad0000.wdog 47: 0 0 0 0 GICv2 92 Level xhci-hcd:usb1 48: 7 0 0 0 GICv2 93 Level dwc3-otg, xhci-hcd:usb5 49: 52 0 0 0 GICv2 95 Level xhci-hcd:usb3 50: 0 0 0 0 GICv2 101 Level ahci-qoriq[3200000.sata] 69: 0 0 0 0 GICv2 185 Level qDMA error 70: 0 0 0 0 GICv2 71 Level qDMA queue 74: 0 0 0 0 GICv2 76 Level fman, ptp_qoriq 75: 0 0 0 0 GICv2 163 Edge 76: 0 0 0 0 GICv2 164 Edge 77: 0 0 0 0 GICv2 165 Edge 78: 0 0 0 0 GICv2 174 Edge IPI0: 8139 9691 8545 5158 Rescheduling interrupts IPI1: 50 258 198 72 Function call interrupts IPI2: 0 0 0 0 CPU stop interrupts IPI3: 0 0 0 0 CPU stop (for crash dump) interrupts IPI4: 0 0 0 0 Timer broadcast interrupts IPI5: 0 13 119 0 IRQ work interrupts IPI6: 0 0 0 0 CPU wake-up interrupts Err: 0

Also ls of /proc/irq is :

root@tqmls1043a-mbls10xxa:~# ls /proc/irq/ 1 12 15 17 19 26 28 3 37 39 45 48 5 6 7 74 76 78 9 11 13 16 18 2 27 29 36 38 4 47 49 50 69 70 75 77 8 default_smp_affinity

So how can I receive MSI interrupt which sent from fpga and how can I send a MSI interrupt to FPGA?

I cannot anything about MSI interrupt on linux. Frankly I want to bind to msi-interrupt from c application. Also I want to check if MSI interrupt come from fpga with command line (ex. cat /proc/interrupts)

Also if I have to write a driver, how can I do, do you have an example? What kind of driver I have to write?

LayerscapeのトピックLS1043A handle MSI interrupt with msi-controller

LS1043A handle MSI interrupt with msi-controller