Hello,
We are working with a QorIQ T1022 design and have completed DDR bring-up and validation using the NXP DDR Validation Suite.
The results are encouraging:
- All DDR validation tests pass successfully.
- We have achieved good timing margins across the tested conditions.
- No errors are reported by the validation suite during stress testing.
However, when we load and run our main application, we begin to observe what appear to be memory-related issues/corruption. These issues are not reproducible using the DDR validation tests alone.
This has raised the question of whether there are additional mechanisms or test methodologies we should be using to detect issues that may only appear under real application workloads.
We are interested in understanding:
- What types of DDR or memory subsystem issues can escape the standard DDR Validation Suite tests on the T1022?
- Are there known application-level scenarios that can expose problems not detected during DDR training and validation?
- Are there additional stress tests, performance monitors, error counters, or debugging techniques available on the T1022 that could help identify the root cause?
- Has anyone encountered a situation where DDR validation showed excellent margins, yet memory corruption or instability was later observed in a production application?
Any guidance on additional diagnostics, hardware checks, or software debugging approaches would be greatly appreciated.
Thank you.
Hello,
The QCVS DDRv tool tests DDR timing margins (write leveling, read/write centering, clock adjustment) using sequential, deterministic access patterns from a single core via JTAG. It does not exercise:
- Multi-master / multi-core concurrent access — The T1022 has two e5500 cores plus the DPAA (Data Path Acceleration Architecture) with frame managers, queue managers, and DMA engines all competing for the DDR bus simultaneously. Contention-induced timing violations only appear under real traffic.
- Cache coherency stress — Application workloads involving cache flushes, invalidations, and coherent DMA transfers create access patterns the validation suite never generates.
- Thermal and power-supply variation — DDR margins measured at room temperature under light load can degrade significantly when the SoC is at full utilization and the AVDD_DDR supply droops under load.
- DQ mapping errors — If the DQ_MAPn registers are incorrect, the controller may still pass validation (which uses known training patterns) but corrupt data under real application traffic. This is a documented T1022-specific issue.
- Marginal single-bit ECC errors — The validation suite may not accumulate enough transactions to trigger SBE threshold reporting, while a real application running for hours will accumulate them silently.
DQ Mapping Misconfiguration
- The DQ_MAPn registers provide the mapping of DRAM DQ signals to the controller. If these are wrong, the controller cannot correctly interpret training patterns, and data corruption occurs under real workloads. NXP has confirmed this on T1022 designs: clearing all DQn_MAP registers (setting to 0 for 1:1 mapping) is a recommended diagnostic step.
- Memory Ordering / Pipeline Effects (PowerPC e5500)
- The e5500 core has well-documented memory ordering subtleties. Writes may not be fully committed to DDR before subsequent reads, especially without explicit msync/isync barriers. NXP's apps team has confirmed: "The write may not be fully committed to memory before the error injection is disabled. The subsequent read may hit a cache or pipeline, not triggering the ECC logic immediately."In application code, missing barriers around DMA setup or shared-memory structures can cause apparent corruption that is actually a coherency ordering issue.
- ECC Single-Bit Error Accumulation
- The T1022 DDR controller supports ECC. Single-bit errors (SBEs) are silently corrected by the hardware but counted in ERR_SBE[SBEC]. If the SBE counter crosses the threshold ERR_SBE[SBET], a critical interrupt is generated. Under light validation traffic, this threshold is never reached. Under a real application, accumulated SBEs can eventually become uncorrectable multi-bit errors (MBEs), which are fatal — data cannot be recovered.
- Multi-Bit ECC Errors Manifesting as Application Crashes
- NXP has documented cases on QorIQ platforms (P2020, T1042) where application crashes (e.g., a lwz instruction faulting on a valid-looking address) were traced to MBE events in DDR. The crash is not a software bug — it is the e5500 core receiving corrupted data from the DDR controller and raising an IVOR1 Machine Check Exception. Importantly, the MCSR register shows 0xA000 (uncorrectable L1 cache/tag error) even when the memory region is cache-inhibited, because the DDR controller asserts a corrupted-data signal that the core registers as an L1 error
Cross-check your board design against:
- AN3940 — Hardware and Layout Design Considerations for DDR3 SDRAM Memory Interfaces
- AN5097 — Hardware and Layout Design Considerations for DDR4 SDRAM Memory Interfaces
- AN4039 — PowerQUICC and QorIQ DDR3 SDRAM Controller Register Setting Considerations
Pay particular attention to: AVDD_DDR power supply noise and decoupling, DDR reset signal routing (HRESET_B to DRAM RESET), and termination resistor values.
Regards
Hi,
Analysis of the Two Results
Figure 1 (QCVS Tool
Figure 2 (Your own measurements
Possible Causes for the Discrepancy
1. WRLVL_START Values Are Not Being Recalculated When CLK_ADJ Changes
This is the most likely root cause. The QCVS tool correctly recalculates WRLVL register values for each CLK_ADJ value it tests. When you sweep CLK_ADJ in your own test while keeping the WRLVL_CNTL register value fixed at 0x8655F606U (WRLVL_START = 3/4), you are testing an invalid combination for most CLK_ADJ settings — the write leveling start delay must be appropriate for the specific CLK_ADJ phase chosen.
In the QCVS tool, after "Centering the clock" completes, if you click on any other passing CLK_ADJ cell, it generates updated WRLVL register values in the "Updated Configuration Registers" window corresponding to that specific CLK_ADJ. This is the correct way to evaluate margin at alternative CLK_ADJ settings.
2. QCVS Uses Write-Read-Compare; Your Test May Use BIST or a Different Algorithm
The QCVS "Centering the clock" scenario uses the Write-Read-Compare (WRC) algorithm, which performs a proper margin sweep and optimization of CLK_ADJ and WRLVL simultaneously. If your own testing uses BIST-based patterns, it is important to note that BIST tests do NOT retrain or tune any PHY timing — they only validate functionality using existing settings, and do not measure margin.
3. Signal Integrity / Board-Level Factors
The QCVS tool operates under its own controlled test sequence and does not account for board-specific factors such as PCB trace length skew, temperature variation, or supply voltage margins. Board-level measurements can reveal a narrower effective timing window, which is expected for a custom board versus the reference design assumptions built into QCVS.
Regarding the Blue-Highlighted Original CLK_ADJ Setting
Your original CLK_ADJ setting (highlighted in blue, at 9/16) falls at or beyond the edge of the passing window in your board-level tests. This indicates it is operating with insufficient margin. The QCVS tool selected 1/8 as the optimum — a significantly different value — which implies the WRLVL_START register values used with your original 9/16 setting may not be appropriately matched to that CLK_ADJ, further reducing the effective margin.
Recommended Margin Requirements
The QCVS tool considers the bright green cell as the optimal setting and the passing window (green cells) around it as the margin indicator. The generally accepted NXP recommendation is:
- A minimum of 2–3 passing green cells on each side of the selected operating point is considered adequate margin.
- Only 1 passing cell on either side is considered insufficient/marginal and should not be used for production.
- The operating point should be centered within the passing window — not at an edge.
Recommended Next Steps
- Use the QCVS tool's optimal CLK_ADJ of 1/8 and extract the corresponding WRLVL_CNTL register values from the "Updated Configuration Registers" window. Do not manually sweep CLK_ADJ while keeping WRLVL fixed.
- Re-run the full "Centering the clock" validation using the Write-Read-Compare test (not BIST) to ensure both CLK_ADJ and WRLVL_START are co-optimized.
- Verify the CLK-to-DQS skew input in QCVS is correctly set based on your PCB trace length measurements (CLK length minus DQS length from your EDA tool), as this directly seeds the initial WRLVL_START values.
- If you wish to evaluate the margin at your preferred CLK_ADJ, click that cell in the QCVS tool after the "Centering the clock" run so the tool recalculates the correct WRLVL values, then run the WRLVL margin scenario with those updated registers
Regards
Hi,
Thank you for the detailed response.
Since then, we have carried out some additional investigation. Figure 1 shows the data extracted from the QorIQ validation tool. Based on these results, it appeared that we had sufficient margin available, and when we reviewed our register settings, they were within the values reported by the tool.
Because of this, we initially did not make any changes to the register settings, as the validation results did not indicate that there was an issue.
Figure 2 shows the results from our own testing. As you can see, these results do not appear to correlate with what the validation tool is reporting. From our measurements, the available margin seems to be significantly lower than that indicated by the tool.
Is there anything we may be missing, or are there any additional factors that could explain the discrepancy between the validation tool results and our measurements?
Please also note that the value highlighted in blue was our original CLK_ADJ setting. Could you also clarify what would be considered a sufficient margin in this case? The tool appears to indicate that having one green value on either side of the optimum setting is acceptable, but we would appreciate confirmation of the recommended margin requirements.
figure 1) Are testing carried out on QorIQ DDR Validation tool.
figure 2) Actual results when adjusting the CLK ADJ values with our own software.
