MIMXRT1166 and W9812G6KB SDRAM errors

[tbonkers]

I have a custom board which uses MIMXRT1166 and W9812G6KB SDRAM at 166 MHz. I've configured the DCD to align the SDRAMCRx settings with the specific values requested by the SDRAM datasheet. DQS pin is not routed on the board.

My application is using Zephyr and SDRAM is utilized for data and application heap/stacks. After a few minutes of running, the application crashes with different error messages:

[00:08:10.517,000] <err> os: ***** USAGE FAULT *****
[00:08:10.525,000] <err> os:   Illegal use of the EPSR

[00:18:33.709,000] <err> os: ***** USAGE FAULT *****
[00:18:33.717,000] <err> os:   Attempt to execute undefined instruction

[00:01:58.470,000] <err> os: ***** USAGE FAULT *****
[00:01:58.478,000] <err> os:   Unaligned memory access

I've tested the SDRAM by running a bare-metal application that writes a pattern, then checks it - runs for hours with no issues. 

I've spent a lot of time tweaking SEMC parameters with no success. From what I can tell, wrong values are popped out of the stack, however the crash happens at a different place every time and I can't find a pattern to it. 

I've spent a lot of time chasing the issue and will really appreciate any pointers. 

[tbonkers]

Hi @Kan_Li ,

The code does not attempt to read or write the EPSR, and the application runs in XIP mode. I don't have access to EVK but I have a previous board revision which uses another SDRAM chip, and the application runs with no problems. This points to SDRAM misconfiguration or hardware problems. 

The exceptions are different most of the time and they occur in different places. Here's an example:

[00:26:09.898,000] <err> os: ***** MPU FAULT *****
[00:26:09.906,000] <err> os:   Instruction Access Violation
[00:26:09.914,000] <err> os: r0/a1:  0x00000001  r1/a2:  0x80101c60  r2/a3:  0x80101c60
[00:26:09.925,000] <err> os: r3/a4:  0x80101c60 r12/ip:  0x8012b4bc r14/lr:  0x300335ad
[00:26:09.936,000] <err> os:  xpsr:  0x20000246
[00:26:09.943,000] <err> os: s[ 0]:  0x80101c60  s[ 1]:  0x800026c4  s[ 2]:  0x00000000  s[ 3]:  0x3005c74c
[00:26:09.955,000] <err> os: s[ 4]:  0x3005c730  s[ 5]:  0x30031fef  s[ 6]:  0x80082460  s[ 7]:  0x00072300
[00:26:09.968,000] <err> os: s[ 8]:  0x00000038  s[ 9]:  0x3000debb  s[10]:  0x8012b4c8  s[11]:  0x3000a52f
[00:26:09.981,000] <err> os: s[12]:  0x00000000  s[13]:  0xfffffffd  s[14]:  0xaaaaaaab  s[15]:  0xaaaaaaab
[00:26:09.993,000] <err> os: fpscr:  0xaabaaaaa
[00:26:10.000,000] <err> os: Faulting instruction address (r15/pc): 0x80101c60
[00:26:10.010,000] <err> os: >>> ZEPHYR FATAL ERROR 20: Unknown error on CPU 0
[00:26:10.020,000] <err> os: Fault during interrupt handling

[00:26:10.029,000] <err> os: Current thread: 0x80101c60 (lvgl_handler_tid)
[00:26:10.039,000] <err> os: Halting system

The PC points to SDRAM, which is odd because the application is not supposed to execute any functions from SDRAM. Some of the registers contain a suspicious value of 0xaaaaaaab, which may mean stack poisoning as Zephyr fills stacks with 0xaaaaaaaa pattern. 

When I vary the SDRAM timing parameters in my DCD file, the errors tend to increase in frequency - again, that makes me think the SDRAM is the issue, however I can't pinpoint the exact reason.

I've tried lowering the SDRAM speed but crashes still appear. Another thing I've noticed is that AXIBUSERR in SEMC_INTR is set - that's not happening when running the bare metal application.

[Kan_Li]

Hi @tbonkers ,

Thanks for the clarification! Is it possible to share part of your schematics for a review? especially regarding the connection between RT and SDRAM.

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[tbonkers]

Hi @Kan_Li ,

Below is the SDRAM schematic:

[Kan_Li]

Hi @tbonkers ,

Thanks for the sharing !  The SDRAM schematics has been double checked by me and the expert, and we could not find any issue within it, is it possible to share your DCD file as well? and did your application code initialize the SEMC as well? Please kindly calrify.

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[tbonkers]

Hi @Kan_Li , thank you for looking into the schematics.

I use DCD for SEMC initialization only, the SEMC is not touched from the application. Please see attached dcd.c file.

[Kan_Li]

Hi @tbonkers ,

Thanks for the info! Your DCD file has been reviewed and updated by the expert, please kindly have it from the attachment.

Hope that helps,

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[tbonkers]

Hi @Kan_Li ,

The updated DCD file has made no difference and the crashes persist. 

[Kan_Li]

 Hi @tbonkers ,

Thanks for the info! Is it possible to put the data and heap/stack in another place such as SRAM? if the problem persists , that means the root cause is on application side, not the SDRAM. 

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[tbonkers]

Hi @Kan_Li ,

When running an application that does not use SDRAM I don't see any errors. I tested by using OCRAM for data, and writing/reading patterns into SDRAM - works with no issues, so I think I can safely conclude the problem is SDRAM-related. 

We have changed the SDRAM to another chip (ISSI) and the problem persists. I still can't be 100% sure it's a hardware/routing issue as reading and writing tests pass. Do you have any ideas on what tests to run to isolate the problem further?

[Kan_Li]

Hi @tbonkers ,

The attached patch is some code for SDRAM stress test, you may use "git apply" command with this patch in any folder to have the source files to be included in your project.

Hope that helps,

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[tbonkers · Accepted solution]

Hi @Kan_Li ,

I have a lead which points to firmware as the culprit. I've been using Zephyr v3.7.1 LTS up to now, yesterday I switched to Zephyr v4.2.99, and left two boards running overnight, and they run with no problems.

Thanks for you help.

[Kan_Li]

Hi @tbonkers ,

Great to know that! Please kindly let me know if you have any question.

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Kan_Li]

Hi @tbonkers ,

Is there any attempt from your app code to read/write the EPSR register? especially the T bit in this register, because Attempting to execute instructions when the T bit is 0 results in a fault or lockup.  I am wondering if the undefined instruction issue is also due to that, and unaligned memory access issue as well. 

I assume you are using a custom board and your project runs in XIP mode, right? Not sure if you have RT1160EVK as well, if yes, you may try your application code on the EVK and check if the result is the same, that would help to locate where the root cause is, hardware or software.

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------