This article describes a Full-Bridge PCIe reference design that connects an NXP i.MX95 SoC (acting as PCIe Root Complex) to a Lattice CertusPro-NX / Certus-NX FPGA endpoint, together with a Linux kernel driver. Updating the FPGA bitstream over PCIe is demonstrated as the reference use case, built on top of a general-purpose PCIe data-transfer, register-access and MSI-completion interface.
Hardware
- Host / Root Complex: NXP i.MX95 SoC running Linux (DesignWare PCIe controller)
- FPGA target / endpoint: Lattice CertusPro-NX PCIe Bridge board (Device ID
0x9C25) or Certus-NX (Device ID0x9C1D) — Lattice Vendor ID0x1204 - PCIe link: 1-lane (x1), single BAR (BAR1) design
- Interrupt: 1 MSI vector, EP→Host completion signalling
Board ConnectionBoard ConnectionArchitecture
The host application talks to the FPGA through the kernel driver. The driver exposes BAR1 as a character device and manages two host-side DMA-coherent buffers. Because the design is a full bridge, both sides can initiate PCIe transactions: the host issues MWr/MRd to BAR1, and the FPGA (as bus master) issues its own MWr/MRd back into host memory.
Driver ArchitectureDriver Architecture| Block | Role |
|---|---|
| Application (userspace) | Issues IOCTLs; supplies payload and DMA target addresses |
Driver lattice_nxp_pcie_ep.c |
BAR1 MMIO window, SRC/DST DMA buffers, MSI completion, char device /dev/lattice_nxp_pcie_bar1 |
| FPGA (CertusPro-NX / Certus-NX) | PCIe endpoint; executes command, performs MRd/MWr, raises MSI on completion |
| Host memory | DMA-coherent SRC and DST buffers |
What the driver covers
The kernel module is a general-purpose PCIe endpoint transport, not a bitstream-specific driver. It contains no bitstream or image parsing — bitstream update is simply the reference application layered on top of these primitives. It provides three planes:
- FPGA→Host DMA path (EP-initiated bulk transfer): two host DMA-coherent buffers (SRC/DST, default 4 KB, tunable up to 1 MB). The host fills SRC, programs the FPGA SRC/DST mailbox registers and rings a doorbell; the FPGA (bus master) then moves data via its own MRd/MWr and writes results into the host DST buffer. Usable for any payload.
- Host↔FPGA BAR1 MMIO window: the char-device
read()/write()path exposes the whole BAR1 aperture as an 8-byte-granular, byte-addressable window (readq/writeq, or pairedreadl/writelon 32-bit) for register/RAM access and MWr+MRd readback tests. - Control + completion plane: arbitrary 32-bit BAR1 register read/write by offset (the "command" and "doorbell" registers are conventions the application chooses), plus MSI-based completion notification (atomic counter + wait/poll IOCTLs).
In short, it is a generic data-transfer + register-access + interrupt-notification interface. Bitstream loading is one application built on it; any host↔FPGA bulk transfer or register-control workflow uses the same primitives.
Driver overview —
Probe & init: enable the PCIe device and set bus master → configure the DMA mask (32-bit default, 64-bit fallback) → map BAR1 MMIO → allocate SRC + DST coherent DMA buffers → allocate 1 MSI vector → register /dev/lattice_nxp_pcie_bar1.
IOCTL interface (9 commands):
| Command | Di r | Description |
|---|---|---|
IOCTL_GET_INFO |
R | Return BAR1 size, SRC/DST DMA handles, MSI IRQ number |
IOCTL_FILL_SRC |
W | Copy userspace buffer into the SRC DMA buffer |
IOCTL_CLEAR_DMA |
– | Zero the DST DMA buffer before a transfer |
IOCTL_READ_DMA |
R | Read back bytes from the DST DMA buffer after the FPGA copy |
IOCTL_WAIT_MSI |
W | Block until the next MSI fires (with timeout_ms) |
IOCTL_WAIT_MSI_SINCE |
W | Block until msi_count exceeds a given baseline (race-free) |
IOCTL_GET_MSI_COUNT |
R | Read the current MSI interrupt counter (atomic u64) |
IOCTL_WRITE_REG |
W | Write one 32-bit MMIO register in BAR1 by offset |
IOCTL_READ_REG |
R | Read one 32-bit MMIO register from BAR1 by offset |
Driver internals
- BAR1 read/write paths:
pcie_read()/pcie_write()expose BAR1 as a char device. Accesses must be 8-byte aligned; 64-bit usesreadq/writeq, 32-bit uses pairedreadl/writel. - MSI ISR:
pcie_isr()atomically incrementsmsi_countand wakes the wait-queue, unblockingWAIT_MSI/WAIT_MSI_SINCEcallers. Verbosity level 2 logs each interrupt with its running count. - iATU resync workaround (
FORCE_SYNC_IATUon probe, the driver clears and then restores the root-port Memory Base/Limit registers to force the DesignWare PCIe block to rebuild its inbound iATU translation table. This is required after an FPGA reconfiguration + PCIe remove + rescan cycle. The macro can be commented out if the workaround is not needed. Note that this alters the upstream port and can affect other PCIe devices on the same bus.
Vendor ID: 0x1204 | Device IDs: 0x9C25 (CertusPro-NX), 0x9C1D (Certus-NX).
Module parameters
| Parameter | Type / default / perm | Description |
|---|---|---|
verbose |
int / 1 / 0644 | 0 = minimal (critical errors only); 1 = info (probe steps, DMA sizes, MSI events); 2 = chatty (all IOCTL calls, read/write rejections, MSI interrupts) |
dma_bits |
int / 32 / 0644 | DMA address width for coherent-buffer allocation. 32 = default (most platforms); 64 = use if probe fails with -ENOMEM (-12) on a 32-bit mask |
dma_buf_bytes |
uint / 4096 / 0644 | Size (bytes) of each DMA coherent buffer (SRC and DST). Min 256, default 4096 |
All three parameters are adjustable at load time; verbose and dma_bits can also be changed at runtime via sysfs.
Known limitations — unsupported PCIe transaction types
- Configuration Read (CfgRd0 / CfgRd1) — PCIe TLP Type 0/1 config reads are not supported
- Configuration Write (CfgWr0 / CfgWr1) — PCIe TLP Type 0/1 config writes are not supported
- Legacy I/O Read (IORd) — x86-style I/O reads are not supported; no I/O BARs are mapped
- Legacy I/O Write (IOWr) — x86-style I/O writes are not supported; all access goes through BAR1 MMIO
Supported: Memory Read (MRd), Memory Write (MWr), and MSI interrupt signalling via the BAR1 mailbox.
Reference
lattice_nxp_pcie_ep.c— Linux kernel PCIe EP driver