i.MX8MM LPDDR4 failed

[guerinyoann]

Hi,

We produce a custom board based on a i.MX8MM.

We did 20 boards of pre-serial. 16 boards boot successfully, 4 boards do not boot, because of LPDDR4.

We copied the board routing of the eval board i.MX 8M Mini EVK on our board.

We ran the mscale_ddr_tool_v2.10 on boards which boots successfully to get the lpddr4_timing.c File of SPL u-boot.

We validated the components mounting with X-ray.

We validated the PCB in cutting off sections of the PCB.

So, if the problem is not during fabrication, we are looking for software reasons, so we activated logs in SPL u-boot to target the LPDDR4 problem.

Could you analyze the text files joined to help us?

Regards

[guerinyoann]

I use MX8M_Mini_LPDDR4_RPA_v15.xlsx

[igorpadykov]

You wrote: "But, I have the same file with the i.MX 8M Mini EVK."

So which settings are you using?

[guerinyoann]

mscale_ddr_tool_v310 produce the same file lpddr4_timing.c with i.MX 8M Mini EVK or our custom board

[igorpadykov]

recommended to use latest MX8M_Mini_LPDDR4_RPA_v15.xlsx

aslo, one can try to tweak drive strength in RPA tool, search  word "strength" in xls file.

Best regards
igor

[guerinyoann]

I repeat that already use MX8M_Mini_LPDDR4_RPA_v15.xlsx.

Could it be possible to have relevant answers?

[igorpadykov]

please try to tweak drive strength in RPA tool, search word "strength" in xls file.

Also may be useful to try latest nxp linux from source.codeaurora.org/external/imx/linux-imx repository
https://source.codeaurora.org/external/imx/linux-imx/tree/?h=imx_5.4.24_2.1.0

If this will not help, seems this is hardware issue - one can try to resolder chips.

Best regards
igor

[guerinyoann]

I changed the value "LPDDR4 MR3: PDDS (Pull-Down Drive Strength)" from 6 to 1.

I have the same results. A good card stays OK. A bad card stays KO (the calibration does not pass).

The latest Kernel will not change anything because the initialization of the DDR is done in u-boot SPL.

[igorpadykov]

If this did not help, seems this is hardware issue - one can try to resolder chips.

Best regards
igor

[guerinyoann]

We observed the same problem with 2 different LPDDR4s (Nanya and Micron).
I repeat :
We validated the components mounting with X-ray.
We validated the PCB in cutting off sections of the PCB.

Could you ask an LPDDR4 expert to analyze the logs provided in the first post?

[igorpadykov]

I asked internally below expert answer [EA].:

----------------

according to the logs the CA training failed. The failures point to channel A, no suitable delay was found for the following signals:

KO_SN00008: CA2_A, CA3_A, CA4_A, CA5_A

KO_SN00020: CA0_A, CA1_A, CA2_A, CA4_A, CA5_A

-----------------

In general it may be configuration issue.

Expert can check it, for that please provide (one can use service request for that):

1. Datasheets of all the used memory devices and comment which of them fail.

2. RPAs for all the used memory devices

3. Schematic of the DDR section

Best regards
igor

[guerinyoann]

SN00020 board has a Nanya LPDDR4 (NT6AN512T32AV).

SN00008 board has a Micron LPDDR4 (MT53D512M32D2DS-053 WT:D).

The RPA is the same that the imx8mm evk board.

We checked signals CA0_A to CA5_A. They are connected.

[igorpadykov]

Hi guerinyoann

I asked internally and got below comments:

--------------

per my check there is nothing out of the ordinary in the schematic or the RPA configuration. 

MT53D512M32D2DS-053 WT:D is the part number that is used on our 8MMini EVK and the board also passed validation with this device. Based on this fact and since the customer claims to follow our design, the most probable cause of the failures is a manufacturing defect or a bad device. A few points to check:

1. Did they check if the voltages are fine on the failing boards? (voltage drop, ripple, noise,...)

2. The customer claims that they copied the layout from the EVK. Did they do the same also for the stack-up and PCB dielectric materials? Can they provide also this information?

3. Can they try to swap the processor and/or the memory device to a board proven to be functional to see if the failure follows the part?

--------------

Best regards
igor

[guerinyoann]

1/ Yes, we didn't notice any particular noise, but we have a low bandwidth oscilloscope (100MHz).

2/ See attached file

3/ On a bad board, we changed the memory device, and it didn’t change anything (always bad). We tried to change the processor, but our EMS didn’t succeed to do it, some pads have been broken during the try.

[igorpadykov]

Hi guerinyoann

internally I got below answer on your questions:

------------------

1. Functionality of the DDR is conditioned by clean and stable power supplies that have correct voltage at the processor and memory inputs. Therefore, they should be thoroughly checked to be sure that there is not an issue in terms of ripple, noise and IR drop. Power integrity simulations should be performed.

2. Per the provided document, the stack-up is almost the same as on the NXP EVK board. However, the used dielectric material is different and has slightly different properties. As a result, there are differences in the some of the dielectric thicknesses and also the EM field around the traces will be influenced. Therefore, the design cannot be considered as equivalent to the EVK board. Has the customer performed signal integrity simulations of the layout?

It may be needed to further tune the drive strength and ODT settings for the board. I can see in the thread that the customer already tried to adjust the drive strength of the memory through MR3[PDDS]. This is not complete - there are in total 7 settings for the drive strength and ODT adjustments and MR3[PDDS] will most likely have little influence on CA training since it controls the drive strength of the data signals at the memory side.

Please suggest to customer to adjust the following parameters in the RPA - focus should be mainly on the CA bus settings:

CA bus:

ATxImpedance - the drive strength setting of the CA signals at the i.MX side

MR11[CA ODT] - termination setting of the CA signals at the memory side

Data bus:

TxImepdance - the drive strength setting of the data signals at the i.MX side for the writes

MR11 [DQ ODT] - termination setting of the data signals at the memory side for the writes

MR3[PDDS] - the drive strength setting of the data signals at the memory side for the reads

ODTImpedance - termination setting of the data signals at the i.MX side for the reads. Note that this setting needs to be reflected in MR22[SOC ODT] so the memory device would correctly calibrate the driver to the termination set at the i.MX.

------------------

Best regards
igor

[guerinyoann]

1 – Can you be more explicit: for each parameter (ripple, noise, IR drop), what values can be considered as ok (min/max) ?

2 – No, we don’t have the mean to perform signal integrity. Is it something you can do on our layout ?

3 – For drive strength tuning, we can retest again, but when the CA training fails, we don’t know if the new value is better or worse than before. Do we have any mean to parse the debug return, for us to check what is the better parameter to tune ?

4 - We use the PMIC PF8121. Do you known issues with this PMIC?

[igorpadykov]

--------------

1. The power rails must always stay within the specifications listed in the datasheet. The Hardware Developer's Guide provides further guidance (see sections 3.6, 3.6.2 and Table 13).

2. Unfortunately, we don't perform SI analyzes for customers since we do not have the resources to accommodate the requests.

3. There is currently no automatic parser available. Since there are not that many possible combinations, I suggest to try all of them. Do not set a higher value than 80 Ohm - systems usually don't work with them.

4. No known issues with this PMIC. Just make sure that the correct part number with respect to the used DDR technology is used and the power up sequence is followed. The rest is covered in 1.

--------------

Best Regards

[guerinyoann]

We are checking the stackup, but there is a mismatch on NXP documentations about the same stackup:
- From the latest hardware developer's guide (rev.1 08/2019), table18, copper thicknesses (in oz) are: 1.333, 1, 0.5, 1, 1, 0.5, 1, 1.333, and dielectric (in mils) are: 2.611, 3.94, 3.7, 16.14, 3.805, 3.94, 2.611
- From the EVK SOM board layout (LAY-31399_C), copper thicknesses (in oz) are: 8x1 and dielectric (in mils) are: 2.77, 3.94, 5.04, 12, 5.54, 3.94, 2.77

Both recommandations are rather different, so which one is the best one ? We tried to follow the second one, but we didn't do a non-symetric stackup, was it really important ? What could we improve ?
Our PCB come from China, and they don't have TU768. Do you think using IT180A is a problem ?

Regards.

[igorpadykov]

internally I got below answer:

------------------------

Regarding the discrepancy in the stack-up information between the layout file and the

Hardware Developer's Guide  - internal team confirmed that the stack-up from LAY-31399_C is the correct one.

To comment on the general customer's questions. I don't think that using IT180A would a problem as such. The thing is that simulations need to be run in order to check if everything is ok if the reference design wasn't copied to the letter (and even in such case it is very advisable). Without it, we cannot be sure if there isn't something that needs further optimization or if something hasn't been overlooked.

Any feedback on the previous suggestions (varying the drive strength and ODT settings, power measurements, checking the power-up sequence)?

------------------------

Best regards
igor

[guerinyoann]

Could you answer the question below, please?
Both recommendations are rather different, so which one is the best one ?

Best regards

[guerinyoann]

Hi,

The file lpddr4_timing.c generated by mscale_ddr_tool_v3.10 with RPA version 15 is a little different.

But, I have the same file with the i.MX 8M Mini EVK.

--- a/board/freescale/xxx/lpddr4_timing.c
+++ b/board/freescale/xxx/lpddr4_timing.c
@@ -1,21 +1,23 @@
/*
- * Copyright 2018-2019 NXP
+ * Copyright 2019 NXP
*
* SPDX-License-Identifier: GPL-2.0+
*
* Generated code from MX8M_DDR_tool
+ * Align with uboot version:
+ * imx_v2018.03_4.14.78_1.0.0_ga ~ imx_v2018.04_4.19.35_1.1.0_ga
*/

#include <linux/kernel.h>
#include <asm/arch/imx8m_ddr.h>

struct dram_cfg_param ddr_ddrc_cfg[] = {
- /* Initialize DDRC registers */
+ /** Initialize DDRC registers **/
{ 0x3d400304, 0x1 },
{ 0x3d400030, 0x1 },
{ 0x3d400000, 0xa1080020 },
{ 0x3d400020, 0x223 },
- { 0x3d400024, 0x16e3600 },
+ { 0x3d400024, 0x3a980 },
{ 0x3d400064, 0x5b00d2 },
{ 0x3d4000d0, 0xc00305ba },
{ 0x3d4000d4, 0x940000 },
@@ -45,7 +47,7 @@ struct dram_cfg_param ddr_ddrc_cfg[] = {
{ 0x3d4001a8, 0x80000000 },
{ 0x3d4001b0, 0x11 },
{ 0x3d4001c0, 0x1 },
- { 0x3d4001c4, 0x0 },
+ { 0x3d4001c4, 0x1 },
{ 0x3d4000f4, 0xc99 },
{ 0x3d400108, 0x70e1617 },
{ 0x3d400200, 0x1f },
@@ -54,8 +56,6 @@ struct dram_cfg_param ddr_ddrc_cfg[] = {
{ 0x3d400204, 0x80808 },
{ 0x3d400214, 0x7070707 },
{ 0x3d400218, 0x7070707 },
-
- /* performance setting */
{ 0x3d400250, 0x29001701 },
{ 0x3d400254, 0x2c },
{ 0x3d40025c, 0x4000030 },
@@ -67,10 +67,8 @@ struct dram_cfg_param ddr_ddrc_cfg[] = {
{ 0x3d400498, 0x620096 },
{ 0x3d40049c, 0x1100e07 },
{ 0x3d4004a0, 0xc8012c },
-
- /* P1: 400mts */
{ 0x3d402020, 0x21 },
- { 0x3d402024, 0x30d400 },
+ { 0x3d402024, 0x7d00 },
{ 0x3d402050, 0x20d040 },
{ 0x3d402064, 0xc001c },
{ 0x3d4020dc, 0x840000 },
@@ -93,10 +91,9 @@ struct dram_cfg_param ddr_ddrc_cfg[] = {
{ 0x3d402190, 0x3818200 },
{ 0x3d402194, 0x80303 },
{ 0x3d4021b4, 0x100 },
-
- /* p2: 100mts */
+ { 0x3d4020f4, 0xc99 },
{ 0x3d403020, 0x21 },
- { 0x3d403024, 0xc3500 },
+ { 0x3d403024, 0x1f40 },
{ 0x3d403050, 0x20d040 },
{ 0x3d403064, 0x30007 },
{ 0x3d4030dc, 0x840000 },
@@ -119,8 +116,7 @@ struct dram_cfg_param ddr_ddrc_cfg[] = {
{ 0x3d403190, 0x3818200 },
{ 0x3d403194, 0x80303 },
{ 0x3d4031b4, 0x100 },
-
- /* default boot point */
+ { 0x3d4030f4, 0xc99 },
{ 0x3d400028, 0x0 },
};

@@ -208,8 +204,8 @@ struct dram_cfg_param ddr_ddrphy_cfg[] = {
{ 0x220024, 0x1ab },
{ 0x2003a, 0x0 },
{ 0x20056, 0x3 },
- { 0x120056, 0xa },
- { 0x220056, 0xa },
+ { 0x120056, 0x3 },
+ { 0x220056, 0x3 },
{ 0x1004d, 0xe00 },
{ 0x1014d, 0xe00 },
{ 0x1104d, 0xe00 },
@@ -1060,7 +1056,6 @@ struct dram_cfg_param ddr_fsp0_cfg[] = {
{ 0x54008, 0x131f },
{ 0x54009, 0xc8 },
{ 0x5400b, 0x2 },
- { 0x5400d, 0x100 },
{ 0x54012, 0x110 },
{ 0x54019, 0x2dd4 },
{ 0x5401a, 0x31 },
@@ -1101,7 +1096,6 @@ struct dram_cfg_param ddr_fsp1_cfg[] = {
{ 0x54008, 0x121f },
{ 0x54009, 0xc8 },
{ 0x5400b, 0x2 },
- { 0x5400d, 0x100 },
{ 0x54012, 0x110 },
{ 0x54019, 0x84 },
{ 0x5401a, 0x31 },
@@ -1142,7 +1136,6 @@ struct dram_cfg_param ddr_fsp2_cfg[] = {
{ 0x54008, 0x121f },
{ 0x54009, 0xc8 },
{ 0x5400b, 0x2 },
- { 0x5400d, 0x100 },
{ 0x54012, 0x110 },
{ 0x54019, 0x84 },
{ 0x5401a, 0x31 },
@@ -1182,6 +1175,7 @@ struct dram_cfg_param ddr_fsp0_2d_cfg[] = {
{ 0x54008, 0x61 },
{ 0x54009, 0xc8 },
{ 0x5400b, 0x2 },
+ { 0x5400d, 0x100 },
{ 0x5400f, 0x100 },
{ 0x54010, 0x1f7f },
{ 0x54012, 0x110 },

I passed successfully Stress Test on a board which boot normally.

On a card which does not boot, the Calibration failed during CA training.

Could you analyze the text files to help us?

Downloading file 'bin\lpddr4_train1d_string.bin' ..Done

Downloading file 'bin\lpddr4_train2d_string.bin' ..Done

Downloading file 'bin\lpddr4_pmu_train_1d_imem.bin' ..Done

Downloading file 'bin\lpddr4_pmu_train_1d_dmem.bin' ..Done

Downloading file 'bin\lpddr4_pmu_train_2d_imem.bin' ..Done

Downloading file 'bin\lpddr4_pmu_train_2d_dmem.bin' ..Done

Downloading IVT header...Done
Downloading file 'bin\m845s_ddr_stress_test.bin' ...Done

Download is complete
Waiting for the target board boot...

===================hardware_init=====================
hardware_init exit

*************************************************************************

*************************************************************************

*************************************************************************
MX8 DDR Stress Test V3.10
Built on Feb 5 2020 13:04:09
*************************************************************************

--Set up the MMU and enable I and D cache--
- This is the Cortex-A53 core
- Check if I cache is enabled
- Enabling I cache since it was disabled
- Push base address of TTB to TTBR0_EL3
- Config TCR_EL3
- Config MAIR_EL3
- Enable MMU
- Data Cache has been enabled
- Check system memory register, only for debug

- VMCR Check:
- ttbr0_el3: 0x93d000
- tcr_el3: 0x2051c
- mair_el3: 0x774400
- sctlr_el3: 0xc01815
- id_aa64mmfr0_el1: 0x1122

- MMU and cache setup complete

*************************************************************************
ARM clock(CA53) rate: 1800MHz
DDR Clock: 1500MHz

============================================
DDR configuration
DDR type is LPDDR4
Data width: 32, bank num: 8
Row size: 16, col size: 10
One chip select is used
Number of DDR controllers used on the SoC: 1
Density per chip select: 2048MB
Density per controller is: 2048MB
Total density detected on the board is: 2048MB
============================================

MX8M-mini: Cortex-A53 is found

*************************************************************************

============ Step 1: DDRPHY Training... ============
---DDR 1D-Training @1500Mhz...
[Process] End of CA training
[Process] End of initialization
[Process] End of read enable training
[Process] End of fine write leveling
[Process] End of read DQ deskew training
[Process] End of MPR read delay center optimization
[Process] End of Write Leveling coarse delay
[Process] End of write delay center optimization
[Process] End of read delay center optimization
[Process] End of max read latency training
[Result] PASS
---DDR 1D-Training @200Mhz...
[Process] End of CA training
[Process] End of initialization
[Process] End of read enable training
[Process] End of fine write leveling
[Process] End of MPR read delay center optimization
[Process] End of Write Leveling coarse delay
[Process] End of write delay center optimization
[Process] End of read delay center optimization
[Process] End of max read latency training
[Result] PASS
---DDR 1D-Training @50Mhz...
[Process] End of CA training
[Process] End of initialization
[Process] End of read enable training
[Process] End of fine write leveling
[Process] End of MPR read delay center optimization
[Process] End of Write Leveling coarse delay
[Process] End of write delay center optimization
[Process] End of read delay center optimization
[Process] End of max read latency training
[Result] PASS
---DDR 2D-Training @1500Mhz...
[Process] End of initialization
[Process] End of 2D read delay/voltage center optimization
[Process] End of 2D read delay/voltage center optimization
[Process] End of 2D write delay/voltage center optimization
[Process] End of 2D write delay/voltage center optimization
[Result] PASS

============ Step 2: DDR memory accessing... ============
Verifying DDR frequency point0@1500MHz.......Pass
Verifying DDR frequency point1@200MHz.......Pass
Verifying DDR frequency point2@50MHz.......Pass
[Result] OK

============ Step 3: DDR parameters processing... ============
[Result] Done

Success: DDR Calibration completed!!!
'lpddr4_timing.c' is created!
DDR Stress Test Iteration 1
--------------------------------
--Running DDR test on region 1--
--------------------------------

t0.1: data is addr test
....
t0.2: row hop read test
...

t1: memcpy SSN armv8_x32 test
....
t2: byte-wise SSN armv8_x32 test
..
t3: memcpy pseudo random pattern test
....................................................................
t4: IRAM_to_DDRv1 test
...

t5: IRAM_to_DDRv2 test
--------------------------------
--Running DDR test on frequency point1@200MHz--
--------------------------------

t0.1: data is addr test
....
t0.2: row hop read test
...

t1: memcpy SSN armv8_x32 test
....
t2: byte-wise SSN armv8_x32 test
..
t3: memcpy pseudo random pattern test
....................................................................
t4: IRAM_to_DDRv1 test
...

t5: IRAM_to_DDRv2 test
--------------------------------
--Running DDR test on frequency point2@50MHz--
--------------------------------

t0.1: data is addr test
....
t0.2: row hop read test
...

t1: memcpy SSN armv8_x32 test
....
t2: byte-wise SSN armv8_x32 test
..
t3: memcpy pseudo random pattern test
....................................................................
t4: IRAM_to_DDRv1 test
...

t5: IRAM_to_DDRv2 test

Success: DDR Stress test completed!!!