DDR initialization failed

[jboggs]

We are seeing intermittent failures on our pilot run of boards when running the Operational DDR tests in QCVS tool.  Our initial run of prototype boards ran through the same suite of tests but did not have this failures.  I believe the issue is related to a hardware failure, but I’m having difficulty interpreting the results, and I’m hoping you can shed some light on the output.  The tests we are running are as shown below:

 

The failure can show up in any of the chosen tests but most typically occurs in the DMA Test or the Write-Read-Compare test.  The failure that occurs is a failure to initialize, test not run.  Here is what the error log is for a failure of the walking zeros test, but the log is basically the same regardless of which test actually failed.    

#################### Result for: walking_zeros ###### Run  1 #######################################

Test was not executed!

Err. capture registers:

0xE20, 0x00000000      0xE24, 0x00000000      0xE28, 0x00000000      0xE40, 0x00000080    

0xE44, 0x00000000      0xE48, 0x0000001D      0xE4C, 0x00000000      0xE50, 0x00000000    

0xE54, 0x00000000      0xE58, 0x00010000     

Dump:

0xF00, 0x00000000      0xF04, 0x00001002      0xF08, 0x0000000B      0xF0C, 0x14000C20     

0xF10, 0x00000000      0xF14, 0x00000000      0xF18, 0x00000000      0xF1C, 0x00000000    

0xF20, 0x00000000      0xF24, 0x1E002100      0xF28, 0x1B002200      0xF2C, 0x27002D00    

0xF30, 0x2A002C00      0xF34, 0x25005000      0xF38, 0x00000000      0xF3C, 0x00000000    

0xF40, 0x00000000      0xF44, 0x00000000      0xF48, 0x00000001      0xF4C, 0x94000000    

0xF50, 0x0C000F00      0xF54, 0x0B001200      0xF58, 0x16001C00      0xF5C, 0x18001B00    

0xF60, 0x14000000      0xF64, 0x00000000      0xF68, 0x00000000      0xF6C, 0x00000000    

0xF70, 0x00000000      0xF74, 0x00000000      0xF78, 0x00000000      0xF7C, 0x00000000    

0xF80, 0x00000000      0xF84, 0x00000000      0xF88, 0x00000000      0xF8C, 0x00000000    

0xF90, 0x00000000      0xF94, 0x00000000      0xF98, 0x00000000      0xF9C, 0x13001300    

0xFA0, 0x14001400      0xFA4, 0x12001400      0xFA8, 0x12001400      0xFAC, 0x13000000    

0xFB0, 0x00000003      0xFB4, 0x0F0E0E0F      0xFB8, 0x0F0E0E0D      0xFBC, 0x0E0C0C0F    

0xFC0, 0x0D0D0D0C      0xFC4, 0x0F0D0E10      0xFC8, 0x0E0E0E0D      0xFCC, 0x0E0D0C0F    

0xFD0, 0x0D0D0D0D      0xFD4, 0x0E0C0B0F      0xFD8, 0x0D0C0C0D      0xFDC, 0x0F0E0D10    

0xFE0, 0x0E0F0D0E      0xFE4, 0x0E0D0D03      0xFE8, 0x0D0E0E0B      0xFEC, 0x0F0E0D10    

0xFF0, 0x0E0E0E0E      0xFF4, 0x0F0D0C0E      0xFF8, 0x0E0E0D0D      0xFFC, 0x00000033    

Data:

0x00000000 

--------------------------------------------------------------------

Exception: (<<Error configuring the target! - DDR initialization failed: Invalid value at 0xF04: 0x00001002>>)

--------------------------------------------------------------------

Target system was initialized 0 times and it took 0.000000 seconds.

Target system effective test execution took 0.000000 seconds.

Any help deciphering the 0xF04: 0x00001002 output would be appreciated.

[jboggs]

Yiping:

Thank you for you assistance with this problem. Adjusting the ODT value seems to have resolved the issue.  I believe we have a setting which works for both our prototype and pilot build lots. 

I just have one more question regarding the 0xF04 debug register.  Where can I find the bit definitions of that register, or can you tell me what the meaning of the bits in the register are?  It would be useful to have that information to help debug any future DDR related failures which may occur.

Thank you.        

[yipingwang]

The debug registers are not provided or exposed to customers. The main reason is that it is not really useful information for customer usage.

To debug customer DDR, there are two categories of DDR failures.

1. fail to complete initialization.

2. ECC failures during the operation.

For details on how to tell which categories and how to debug, see the attached FAE training presentation.

Start with a good register settings (you can use QCVS for that).

if we the good setting initialization fails the we have a list of HW issues that customer can examine to see what is causing the failure.

for ECC failures during the operation. use the validation section of QCVS to optimize the setting. if ECC persist with optimized setting then there may be a HW issue (check the AN5097 for the layout guidelines).

[jboggs]

Yiping:

We are running more tests, but I believe we have an impedance problem.  We need to do some more testing to know for sure if we have a fix or not.  I will let you know when we have more information.  

[jboggs]

Hello Yiping:

Thank you for your prompt response.

I did run through the clock centering, and Read and Write ODT tests. I used the results of the Validation process to create a ".regs" file which is imported when the tool is started, before running the tests.

There is a SPD which was read as part of the project setup. The memory is a Smart Modular SH2047SO410872SC 16GB, DDR4, SO-DIMM.

I set the CodeWarrior connection server for verbose logging as you suggested. The results are in the attached files.

ConnectLog.txt, is the initial connection to the board after turning on verbose logging.

TestLog_DMA_Test_Fail.txt, is the log while the tests were running. In this case the DMA test failed. In an effort to minimize the size of the file I canceled the test after the Write-Read-Compare test ran.

DMA_Test_fail_log.txt, is the log report of the DMA test that failed.

[yipingwang]

Hello Johh Boggs,

In your error log, DDR_ERR_DETECT(0x00008E40) is read as 0x80, DDR_ERR_DETECT[ACE] is set.

For DDR_ERR_DETECT[ACE] can be set due to the following reasons:

1. The training sequence that the controller follows to calibrate the read data path was not able to complete. This would probably only happen if there was a hard failure on the memory interface caused by board-level issues or incorrect controller settings.

2. Incorrect termination of MDICx signals.

3. Write leveling calibration was not able to complete. This relates to improper settings of the DDR_WRLVL_CNTL register or board-level issues.

You are using the DDR controller configuration parameters from the SPD, as normal there should be no problem with DDR controller configuration parameters.

I suggest you perform DDR hardware diagnostics.

CodeWarrior provides DDR hardware diagnostics function, please refer to the section "11.3.2 Working with Hardware Diagnostic Action editor" in Freescale\CW_PA_v10.5.1\PA\Help\PDF\Targeting_PA_Processors.pdf.

First please click "C/C++", then create a bareboard CodeWarrior project with "Connect" launch configuration enabled, please modify DDR configuration section in the initialization file in <project_folder>/CFG/T1040RDB_init_core.tcl according to the file ddrCtrl_1.tcl under Generated_Code folder in the QCVS DDRv project. Then connect to the target board to do DDR hardware diagnostics according the user manual.

Thanks,

Yiping

[yipingwang]

Hello Johh Boggs,

Have you completed Centering the clock, Read ODT and driver, write ODT and driver test before Operational DDR tests?

Please check whether there is problem with the DDR controller initial configuration parameters. If there is SPD on the target board, please use reading from SPD to create a DDR project. If no, please create a QCVS DDR project with default parameters, then modify Properties panel according to DDR datasheet.

Would you please capture the low level CCS log to me to do more investigation? Please open CodeWarrior Connection Sever console, type "log v", then connect to the target board again, the low level CCS log will be printed in CCS console, would you please capture the CCS log and attach it here?

Thanks,

Yiping