LS1046A DDR Issues The LS1046A and BCM88270 (switch chip) are connected via PCIe.3 x1 Gen2 (SD2_T/RX2_P/N). The LS1046A uses four K4AAG165WC-BCWE chips (without ECC). The current situation encountered: 1. When the BCM88270 is not sending Ethernet packets to the LS1046A via PCIe, memory tests using stress ng and memtester show no problems. 2. When the BCM88270 sends Ethernet packets to the LS1046A via PCIe, `stress ng` and `memtester` will encounter errors. 3. The LS1046A repeatedly reads and writes to the BCM88270 registers or entries via PCIe (through DMA) without any problems. Furthermore, stress testing at the same time also showed no issues. Board Design Communication & Control(I3C | I2C | SPI | FlexCAN | Ethernet | FlexIO) Re: LS1046A DDR问题 Since individual tests of stress ng and memtester were normal, it proves that the basic functions of DDR are normal. The only error occurred when BCM88270 sent packets, indicating that the error was triggered by PCIe traffic. Based on the phenomena you described, the probabilities of the possible causes are ranked as follows: 40% BCM88270 RX DMA address out of bounds/descriptor error 25% PCIe RX direction SI issue (exposed during packet reception) 15% PCIe Cache Coherent Configuration Error 10% DDR SI issue (triggered by high bandwidth) 10% power integrity issues First possibility: The PCIe DMA corrupted the DDR memory (highest probability). The phenomenon most closely matches: BCM88270 packet sending PCIe DMA write to LS1046A memory > DMA address error or descriptor error -> Overwriting memory used by Memtester or Stress-ng -> Memory error detected Inspection method: Check the DMA buffer address to confirm whether RX Descriptor, RX Buffer, SKB, and DMA Pool are out of bounds. On Linux, check the return address of dma_alloc_coherent() to check if start, size, and end overlap. Avoid DMA regions using memtester, for example, memtester 2M. Observe whether the DMA is located in low memory. If no more errors are reported after avoiding the DMA region, the DMA overwriting is basically locked. The second possibility is a PCIe Cache Coherent configuration error. The LS1046A uses a DPAA architecture. PCIe DMA involves the CPU cache, CCI-400, PCIe controller, and DDR. If the BCM88270 uses DMA to write to DDR: However, errors in the driver's `dma_sync_single_for_cpu()` and `dma_sync_single_for_device()` functions can cause the CPU to see old data, the DMA to write new data, and then `memcmp` and `memory test` to fail. Check the device tree to see if the PCIe node pcie@340000 has dma-coherent; if the configuration is incorrect, it will cause random memory errors. The third possibility: PCIe receive direction SI problem This deserves special attention. You mentioned that for PCIe.3 x1 Gen2, SD2_TX/RX2_P/N, the error only occurs when sending packets from BCM88270 to LS1046A. However, the DMA access table entries are normal when LS1046A -> BCM88270. This suggests that the PCIe RX direction is more suspicious. PCIe register access: traffic is very small, and it is not easily exposed even if the BER is high. Ethernet packet reception: Continuous PCIe DMA TLP, traffic volume is several orders of magnitude higher. At this time: CRC retransmission, replay, and NAK (Network Address Translation) increase significantly. Although PCIe theoretically has LCRC protection, if the link becomes marginalized, it may lead to: DMA timeout, descriptor corruption, driver anomalies, and ultimately, memory test failures. To check the PCIe error counters: `lspci -vv`, pay attention to: CESta: Correctable Error; UESta: Uncorrectable Error; check if BadTLP, BadDLLP, ReplayNumRollover, and ReceiverError are increasing. The fourth possibility: DDR SI/PI edge problem Although the stress level is normal on its own, it cannot be completely ruled out. Reason: When BCM88270 sends packets, it will increase the following: 1) PCIe SerDes power consumption Added: 1V, 1.8V, AVDD_SERDES noise. 2) DDR access volume surged Normal test: CPU <-> DDR Now, the CPU, PCIe DMA, and DDR controller work simultaneously, significantly increasing bandwidth. If DDR capacity is insufficient, errors will begin to occur. Verification method: Reduce the DDR frequency, for example: 1600MT/s → 1333MT/s. If the problem disappears, it is basically a DDR SI/PI issue. To check DDR ECC statistics: Although there is no ECC, you can use md.l in uboot to read the DDR controller status register and check if DDR_ERR_DETECT is abnormal. Fifth possibility: Power integrity issue The four K4AAG165WC-BCWE chips have a relatively large capacity. When high-speed PCIe packet reception, CPU stress, and high DDR bandwidth occur simultaneously, the following may occur on the board: VDD_DDR, VDD_SOC, and VDD_CORE drops. Key measurements: Use an oscilloscope to check if the ripple and transient voltage drop of VDD_DDR and VDD_SOC exceed the limits during a fault. Pay special attention to the moment when the BCM88270 starts sending large amounts of packets. Sixth possibility: PCIe and DDR trace crosstalk On the LS1046A: PCIe3 and DDR4 are both high-speed interfaces. If the layout is tight: PCIe RX, DDR DQ, and DDR DQS have parallel long-distance traces. High-volume PCIe traffic may trigger issues. This phenomenon closely matches the pattern: no packets sent => normal operation, high packet volume => DDR error. Suggested order of investigation: Step 1: Capture PCIe errors: use lspci -vv to check if Receiver Error, Bad TLP, Replay, and CRC Error are continuously increasing. Step 2: Disable network driver DMA packet reception. Only keep the PCIe read/write registers enabled and verify if Stress is still working correctly. If it is, the problem lies in the RX DMA path. Step 3: Add protection zones before and after the RX DMA Buffer, for example: 0x5A5A5A5A, and continuously check if they are overwritten. Confirm whether the DMA has exceeded the limit. Step 4: Reduce the PCIe speed and force Gen2 -> Gen1. If the fault disappears, check the PCIe SI first. Step 5: Reduce DDR frequency, 1600MT/s -> 1333MT/s. If the problem disappears, first check DDR SI/PI. Re: LS1046A DDR问题 The problem has been resolved; it was a BCM driver issue. Thank you for your support!
記事全体を表示
