Custom Board I.MX8QM - Kernel is Crashing When Systemd Services Starts - LPDDR4 Bandwidth Issues

[RameshChandrakasan]

Hi,@igorpadykov

We have designed custom Board based on i.mx8quadmax - MEK Evaluation board.

It boots -u-boot and working fine.

When it loads the Linux kernel at the near to end of login scripts it crashes.

Almost when it starts the services , it crashes(i.mx media image kernel). randomly different points.

Status:

1) It Passes the DDR Stress Test.

2) Build Root Linux kernel  is working fine.

3) Yocto - minimal image also crashing.

But, If disable services in yocto packages itself, it is working fine.

4) Sometime 10 out of 1 it boots properly without issue.

What could be the problem?

Pls suggest.

Reports attached here

1) Kernel crash report

2) DDR Stress test report

Regards

Ramesh

[Sanket_Parekh]

Hi @RameshChandrakasan 

I hope you are doing well.

"Unable to handle kernel NULL pointer dereference at virtual address 00000000000000c0" error could be occurred due to null-pointer dereferences in a kernel module.

The log crash seems to be caused by the kernfs_path_from_node() function in /fs/kernfs/dir.c

A similar kind of crash was fixed by the below commit in the kernel.

fs: kernfs: Fix possible null-pointer dereferences in kernfs_path_from_node_locked()

Have you made any different changes to kernel configuration and parameters from i.MX8QM - MEK board.

But, If disable services in yocto packages itself, it is working fine.
Please mention the service that was disabled.

Have you defined any customer services?

Please confirm if the rootfs is stored in mmc or sd card.

Thanks & Regards,

Sanket Parekh

[RameshChandrakasan]

Hi @Sanket_Parekh

Pleasure to Interact With you

1) Null Pointer Issue.

This bug is taken care in Kernel 5.15.32 as per the way you suggested.We have tested our board with kernel 5.15.32. The issue is appeared again. crashing..!!!

Parellel Memory bandwidth

We have tested the Board with kernel 5.15.32 - Core Minimal Image with systemd completely removed.

Then , It Boots every time.

Then , We ran the parallel memory bandwidth test.

https://community.nxp.com/t5/i-MX-Processors/Memory-Bandwidth-Measurement-Tool/m-p/989913

It starts loading the Memory slowly by increasing the thread count and Buffer size.

Like Thread count : 1,2,3..along with buffer size.

It crashes every time when the thread count becomes 3.

Hence , LPDDR4 bandwidth issue / DDR4 memory timing issue - identified.

Whether it is due to board fabrication OR DDR4 memory part issue

Pls suggest further

Regards

Ramesh

[Sanket_Parekh]

Hi @RameshChandrakasan ,

I hope you are doing well.

Please provide me with LPDDR4 Partnumber and RPA file used for further debugging.

Thanks & Regards,

Sanket Parekh

[RameshChandrakasan]

Adding @michael_kjar

[RameshChandrakasan]

Dear Michael,

Eval Board i.MX8QM Quad MAX - LPDDR4 part is obsolete, But RPA Tool Not providing and alternative part or Timing parameter

[RameshChandrakasan]

Adding Mr @igorpadykov