LS1046A intermittent Linux 6.6.52 NVMe probe failure with invalid Admin CQE Hi NXP team, We are seeing an intermittent NVMe probe failure on our LS1046A-based boards. SoC: LS1046A
Kernel: Linux 6.6.52
U-Boot: 2022.04
NVMe: ATP I-Temp. NVMe M.2 2242 SSD
NVMe FW: W1207A1
Connection: LS1046A PCIe3, pcie@3600000
Link: PCIe Gen3 x2
Linux BDF: 0002:01:00.0 U-Boot can consistently detect the NVMe, list its filesystem, and load the kernel, initramfs, and FDT. Linux occasionally fails during NVMe probe: [ 5.419777] could not locate request for tag 0x0
[ 5.424399] nvme nvme0: invalid id 0 completed on queue 0
[ 66.527417] nvme nvme0: I/O 17 QID 0 timeout, disable controller
[ 66.529448] nvme nvme0: Device not ready; aborting shutdown, CSTS=0x1
[ 66.542612] nvme: probe of 0002:01:00.0 failed with error -4 The timed-out request tag is not constant. We have seen both I/O 17 QID 0 and I/O 21 QID 0, but the invalid completion always reports CID/tag 0 on Admin Queue 0. ASPM and APST are disabled: nvme_core.default_ps_max_latency_us=0
nvme_core.apst=0
pcie_aspm=off Any guidance would be appreciated. Re: LS1046A intermittent Linux 6.6.52 NVMe probe failure with invalid Admin CQE Hello,
Fix Options
Option 1 — Preferred: Patch U-Boot to set PCIe snoop bits in SCFG (correct fix)
In your U-Boot board init code (e.g. board/freescale/ls1046ardb/ls1046ardb.c or equivalent), ensure the SCFG SNPCNFGCR register enables PCIe read/write snooping. The LS1046A Reference Manual defines these bits; the relevant additions are:
#include
/* In board_init() or equivalent early init: */
struct ccsr_scfg *scfg = (struct ccsr_scfg *)CONFIG_SYS_FSL_SCFG_ADDR;
/* Enable PCIe, SATA, USB, eDMA snooping */
setbits_be32(&scfg->snpcnfgcr,
SCFG_SNPCNFGCR_SECRDSNP | SCFG_SNPCNFGCR_SECWRSNP |
SCFG_SNPCNFGCR_SATARDSNP | SCFG_SNPCNFGCR_SATAWRSNP |
SCFG_SNPCNFGCR_USB1RDSNP | SCFG_SNPCNFGCR_USB1WRSNP |
SCFG_SNPCNFGCR_USB2RDSNP | SCFG_SNPCNFGCR_USB2WRSNP |
SCFG_SNPCNFGCR_USB3RDSNP | SCFG_SNPCNFGCR_USB3WRSNP |
SCFG_SNPCNFGCR_EDMASNP);
The LS1046A CCI-400 interconnect requires these bits to be set so that PCIe master reads/writes participate in cache coherency. Once set, the NVMe DMA controller will snoop the ARM caches, and Linux's dma-coherent mappings will be correct.
Strongly consider upgrading U-Boot to 2024.01 or later — NXP Layerscape U-Boot has accumulated several PCIe/coherency fixes since 2022.04.
Option 2 — Fallback: Override DTS to mark PCIe3 as non-coherent
If you cannot change U-Boot right now, override the PCIe3 node in your board DTS to explicitly mark it non-coherent. This reverts to the pre-6.1 behaviour with software cache maintenance:
/* In your board .dts (e.g. fsl-ls1046a-yourboard.dts) */
&pcie3 { /* pcie@3600000 */
/delete-property/ dma-coherent;
};
This forces Linux to use non-cacheable DMA mappings and perform explicit cache flushes, at the cost of some throughput. It is a valid workaround while you fix U-Boot.
The Layerscape PCIe DT binding documentation explicitly notes that dma-coherent requires the hardware IP to guarantee coherency — it should not be set if the SoC interconnect snooping bits are not programmed.
Option 3 — Secondary: Ensure proper NVMe shutdown in U-Boot before handoff
U-Boot 2022.04's NVMe driver may not issue a graceful NVMe shutdown ( CC.SHN = 01b , wait for CSTS.SHST = 10b ) before jumping to Linux. This leaves the controller in CSTS.RDY=1 with potentially stale Admin Queue state. Linux resets the controller (CC.EN = 0 → 1), but if the DMA coherency issue is also present, the reset may not drain the stale completion reliably.
After fixing coherency, if intermittent failures persist, add an explicit NVMe shutdown to your U-Boot boot script or board code before booti / bootm . Alternatively, the Linux NVMe driver already handles a controller that is pre-enabled (it resets it via nvme_disable_ctrl() ), so this is lower priority than Option 1.
Regards Re: LS1046A intermittent Linux 6.6.52 NVMe probe failure with invalid Admin CQE thanks for option 1, the proposed SNPCNFGCR mask contains SEC, SATA, USB and eDMA snoop bits, but no PCIe1/2/3 snoop bits. Could you please identify the exact bit(s) in SNPCNFGCR that control PCIe3 snooping on LS1046A? According to an earlier NXP Community response, PCIe snooping is controlled by the PCIe No Snoop TLP attribute rather than SNPCNFGCR. Could you clarify how the proposed mask makes NVMe DMA through pcie@3600000 coherent? see the earlier NXP link: https://community.nxp.com/t5/Layerscape/Does-LS1046A-support-DMA-and-Cache-coherency/m-p/1416557 https://community.nxp.com/t5/QorIQ/PCIe-and-data-cache-on-LS1046A/m-p/1053625 Re: LS1046A intermittent Linux 6.6.52 NVMe probe failure with invalid Admin CQE thanks a lot for the detailed explaination. below is the PCI info at our boards root@E7-2:~# lspci 0000:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 81c0 (rev 10) 0000:01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries Device 6866 (rev a0) 0001:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 81c0 (rev 10) 0002:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 81c0 (rev 10) 0002:01:00.0 Non-Volatile memory controller: ATP ELECTRONICS INC Device 0d0b (rev 03) and the NVME endpoint PCIe Device Control register root@E7-2:~# setpci -s 0002:01:00.0 78.w 201f root@E7-2:~# setpci -s 0002:01:00.0 CAP_EXP+8.w 201f it looks the no-snoop field is cleared for the endpoint. Anything else needs to be checked? very appreciated for the help. Regards Re: LS1046A intermittent Linux 6.6.52 NVMe probe failure with invalid Admin CQE Hello,
SNPCNFGCR has no bit(s) that control PCIe3 snooping on LS1046A. PCIe coherency is determined by the PCIe No Snoop TLP attribute, not this register.
If the proposed mask is the commonly referenced:
0xc0f3e000
it sets only:
Bits
Function
31:30
SEC read/write snoop
23:22
SATA read/write snoop
21:20
USB1 read/write snoop
17
eDMA snoop
16:15
USB2 read/write snoop
14:13
USB3 read/write snoop
The EDMASNP bit is for eDMA, not PCIe DMA. Thus, this mask cannot by itself make NVMe DMA through pcie@3600000 coherent.
For the NVMe path:
NVMe is the PCIe endpoint and initiates inbound Memory Write TLPs into LS1046A DDR.
Those TLPs must have No Snoop = 0 for hardware snooping/coherency.
Check the NVMe endpoint’s PCIe Device Control register, not merely the LS1046A root port. Its Enable No Snoop field is at CAP_EXP+0x08 , bit 11 ( 0x0800 ). Clearing it prevents the endpoint from issuing No-Snoop transactions.
The LS1046A’s PCIe inbound/CCI/SMMU configuration must also route the transaction through the coherent path. NXP states that inbound PCIe coherency is handled by the host coherency setup/SMMU rather than SNPCNFGCR .
Therefore, dma-coherent in the device tree is a software declaration, not the hardware switch that enables PCIe snooping. It is valid only when the PCIe/CCI/SMMU and endpoint No-Snoop settings have already been configured consistently.
Regards
Regards Re: LS1046A intermittent Linux 6.6.52 NVMe probe failure with invalid Admin CQE We checked the complete board DTS. The board DTS "odin-maroon.dts" includes the base fsl-ls1046a.dtsi which defines dma-coherent on the soc node, see below. However, the odin-maroon.dts explicitly removes it, see below: &soc {
/delete-property/ dma-coherent;
pcie@3400000 {
/delete-property/ iommu-map;
};
pcie@3500000 {
/delete-property/ iommu-map;
};
pcie@3600000 {
/delete-property/ iommu-map;
};
/delete-node/ iommu@9000000;
}; The board DTS then adds dma-coherent back to many selected devices, but not to pcie1, pcie2, or pcie3. Therefore, PCIe3/NVMe is currently expected to operate as a non-coherent DMA device, without SMMU translation. We also confirmed that the ATP NVMe endpoint reports NoSnoop-, so its Device Control No Snoop Enable bit is clear. root@E7-2:~# lspci -s 0002:01:00.0
0002:01:00.0 Non-Volatile memory controller: ATP ELECTRONICS INC Device 0d0b (rev 03)
root@E7-2:~# lspci -vvv -s 0002:01:00.0|grep -i snoop
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
Max snoop latency: 0ns
Max no snoop latency: 0ns
root@E7-2:~# setpci -s 0002:01:00.0 CAP_EXP+8.w
201f
root@E7-2:~# setpci -s 0002:01:00.0 78.w
201f Could you please confirm: 1. On LS1046A, can PCIe3 inbound Memory Write TLPs with No Snoop=0 remain hardware coherent when iommu-map and the SMMU node are removed? 2. Should we add dma-coherent specifically to pcie3: &pcie3 { dma-coherent; }; while keeping iommu-map and SMMU removed? 3. Or does PCIe3 hardware coherency require restoring the SMMU node and iommu-map? Does PCIe3 require SMMU to reach the coherent path, or can the inbound window route directly to coherent DDR? 4. What registers can we dump to verify whether PCIe3 inbound DMA transactions are using the coherent path? thanks Re: LS1046A intermittent Linux 6.6.52 NVMe probe failure with invalid Admin CQE Hi , We completed an additional A/B validation for the intermittent NVMe probe failure. We added `dma-coherent` specifically to the PCIe3 node connected to the NVMe SSD: pcie@3600000 {
dma-coherent;
}; The runtime DTB confirms the property is present: /sys/firmware/devicetree/base/soc/pcie@3600000/dma-coherent Linux now reports `dev_is_dma_coherent() = 1` for the NVMe endpoint. The endpoint PCIe Device Control register remains `0x201f`, so Enable No Snoop (bit 11) is clear. root@E7-2:~# setpci -s 0002:01:00.0 CAP_EXP+8.w
201f After this change, we completed more than 600 consecutive reboot cycles without reproducing the NVMe probe failure. Before the change, Linux reported `dev_is_dma_coherent() = 0`, and the failure could be reproduced intermittently during NVMe Admin Queue initialization. This result strongly suggests that the PCIe3/NVMe DMA coherency configuration was involved. Regards
View full article