i.MX ProcessorsのトピックRe: DDR stress test cause system hang in Linux

DDR stress test cause system hang in Linux

johncoll — Wed, 26 Feb 2020 15:17:21 GMT

Hello NXPs,
Release: Yocto-Sumo (4.14.98_2.0.0_GA)
Board: i.MX8MQ EVK based custom board
We're working on iMX8MQ based custom-designed board with 2GB DDR4 RAM (MCIMX8M-EVK 3GB LPDDR4 RAM) now we're having an issue when doing stress test RAM in linux.
The system always hang when doing RAM stress test with more than 760MB RAM.

I've already done RAM testing with DDR Tool and mtest in Uboot and all PASSED, then configured the linux system recognized with 2GB RAM.
Does anyone provide the solution for this issue? Any guidance or suggestions will be really helpful.
Best regards,
John.

Updated: Add hang log and DDR Controller Configuration Spreadsheet

Re: DDR stress test cause system hang in Linux

igorpadykov — Wed, 26 Feb 2020 23:46:28 GMT

Hi John

may be recommended to use latest MX8M_LPDDR4_RPA_v24 tool with test:

i.MX8 MSCALE SERIES DDR Tool Release (V3.10)

and try attached patch.

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

Re: DDR stress test cause system hang in Linux

johncoll — Thu, 27 Feb 2020 14:24:54 GMT

Hi Igor

I've tested with new patch and system still hang like before, but this time kernel show some log like bellow

Any idea?

Best regards.

John

Re: DDR stress test cause system hang in Linux

igorpadykov — Fri, 28 Feb 2020 04:56:25 GMT

Hi John

1. what is full processor part number

2. please provide full log: uboot + linux

3. can you confirm that you used latest MX8M_LPDDR4_RPA_v24.xlsx tool

with latest ver.3.10 DDR test version below

i.MX8 MSCALE SERIES DDR Tool Release (V3.10)

and followed Chapter 4 How to bring up a new MX8MSCALE board

MSCALE_DDR_Tool_User_Guide.pdf included in package
4. please try with uboot and linux from nxp source.codeaurora.org/external/imx/linux-imx repository
linux-imx - i.MX Linux kernel

uboot-imx - i.MX U-Boot

5. what you do mean saying "doing stress test RAM in linux" - is it linux "memtester" or

some custom test ?

Re: DDR stress test cause system hang in Linux

johncoll — Fri, 28 Feb 2020 16:32:04 GMT

Hi Igor
My custom board are
CPU: MIMX8MQ6CVAHZAB
RAM: 2GB DDR4 MT40A512M16JY-075E
Uboot: imx_v2018.03_4.14.98_2.0.0_ga
Kernel: imx_4.14.98_2.0.0_ga
My DDR is DDR4 not LPDDR4 so I used MX8M_DDR4_RPA_v9.xlsx then based on that I modified it to match our RAM configuration, you can refer it in the attachement. I followed Chapter 4 in MSCALE_DDR_Tool_User_Guide.pdf to build uboot up and runing, the system can boot to linux and work fined. Then in Linux I used stress command to test memory with 760MB.
I have tested RAM using both DDR Tool and Uboot mtest, it's all show PASS. What I saw is that when application try to allocate or malloc a memory more than 760MB in Linux, it cause the hang issue, for example this command 'stress --vm 1 --vm-bytes 760M'

Re: DDR stress test cause system hang in Linux

igorpadykov — Sat, 29 Feb 2020 06:23:54 GMT

Hi John

suggest to try standard linux tool "memtester".

Best regards
igor

Re: DDR stress test cause system hang in Linux

johncoll — Mon, 02 Mar 2020 16:38:23 GMT

Hi Igor

I used "memtester" and same issue happened, this time with "memtester" the system will hange if input memory bigger than 768MB

Best regards

John

Re: DDR stress test cause system hang in Linux

igorpadykov — Tue, 03 Mar 2020 00:43:32 GMT

Hi John

memtester (opposite to ddr tool) also stresses power supplies, so errors using

memtester point to power supplies (ripples, instabilities) issues. May be recommended

to recheck board design using sect.3.6. Power connectivity/routing

i.MX8M Hardware Developer’s Guide

Re: DDR stress test cause system hang in Linux

johncoll — Fri, 06 Mar 2020 16:47:55 GMT

Hi Igor

I double checked with hardware team, then following test cases in Power Consumption Measurement document and comparing to IMX8MQ EVK board, there's no much different between our board and KIT.

Today I built new bl31.bin from without OP-TEE and the Linux system isn't hang anymore.

Is there any idea what might be the root cause?

Best regards

John

Re: DDR stress test cause system hang in Linux

igorpadykov — Sat, 07 Mar 2020 04:31:36 GMT

Hi John

if " there's no much different between our board and KIT", then

could you try to reproduce issue on nxp reference board

with software described on

i.MX Software and Development Tools | NXP

Best regards
igor

Re: DDR stress test cause system hang in Linux

haochengdong — Wed, 23 Sep 2020 13:49:01 GMT

We ran into the same problem and had no idea, did you solve it?

Re: DDR stress test cause system hang in Linux

johncoll — Tue, 16 Mar 2021 15:45:42 GMT

Hello.

We had new hardware version and there's some improved on power design but there is still same problem. Do you guys have any suggestion to debug this issue?

Re: DDR stress test cause system hang in Linux

alif_chen — Thu, 19 Aug 2021 12:25:34 GMT

Hello.

Is there any update on how to solve the issue?

We currently face the same issue on i.MX6ULL and wanna reference to the solution.

Thank you

Re: DDR stress test cause system hang in Linux

alif_chen — Thu, 19 Aug 2021 12:50:06 GMT

Hello,

We ran into the same problem, too.

Did you guys have any idea on how to solve it?

Thank you

Re: DDR stress test cause system hang in Linux

alif_chen — Fri, 20 Aug 2021 16:00:57 GMT

Hi all,

reply the issue myself.

It turns out that optee-os revise the DDR in the device tree with "reserved-memory".

However, the "#address-cells" and "#size-cells" are wrong.

I traced the code in optee-os, and then found a commit that helps.

https://source.codeaurora.org/external/imx/imx-optee-os/commit/?h=imx_5.4.47_2.2.0&id=995908f2a91c06ccd3e1bab7bc57325ecf270f62

The issue has been resolved by patching that.

Re: DDR stress test cause system hang in Linux

Russell_Kook — Fri, 03 Dec 2021 02:15:33 GMT

We are having the same problem with the i.MX8M plus.
The i.MX8M plus evk was referenced. However, the package type we use is MIMX8ML6CVNKZAB, and it passed the stress test of "mscale_ddr_tool_v3.30" without any problem.
The part name of the LPDDR we are using is MT53D1024M32D4DT-046 WT:D(4GB).

The Yocto version I use is 5.10-hardknott.

And I also corrected the memory size information of u-boot and ATF.

The linux boot is fine, but the memory test using memtesr or stress-ng fails.
If It set a test area that is exactly larger than 720 MB, the system crashes..
for example :
1. If it run up to 14th "memtester 50M 1000 &", there is no problem. On the 15th run, the system crashes.
2. "memtester 720M 1000 &" is fine but "memtester 750M 1000 &" crashes the system.
3. "stress-ng --vm 1 --vm-bytes 720" is fine, but "stress-ng --vm 1 --vm-bytes 750" crashes the system.

I'm looking forward to a lot of advice.

Re: DDR stress test cause system hang in Linux

BiyongSUN — Fri, 03 Dec 2021 15:45:58 GMT

You need to use linux command "free" to get how many memory for memtester.

Re: DDR stress test cause system hang in Linux

Russell_Kook — Mon, 06 Dec 2021 03:57:45 GMT

root@imx8mpevk:~# free
        total     used      free shared buff/cache available
Mem: 3763556    49684   3480184    8892      233688    3413216
Swap:       0    0         0

I typed because I do not have permission to upload a captured image.

You can see that there is a lot of free memory left.

I have attached my iomem info below.

root@imx8mpevk:~# cat /proc/iomem
30200000-3020ffff : 30200000.gpio gpio@30200000
30210000-3021ffff : 30210000.gpio gpio@30210000
30220000-3022ffff : 30220000.gpio gpio@30220000
30230000-3023ffff : 30230000.gpio gpio@30230000
30240000-3024ffff : 30240000.gpio gpio@30240000
30260000-3026ffff : 30260000.tmu tmu@30260000
30280000-3028ffff : 30280000.watchdog watchdog@30280000
30330000-3033ffff : 30330000.pinctrl pinctrl@30330000
30350000-3035ffff : 30350000.efuse efuse@30350000
30380000-3038ffff : 30380000.clock-controller clock-controller@30380000
30660000-3066ffff : 30660000.pwm pwm@30660000
30670000-3067ffff : 30670000.pwm pwm@30670000
30680000-3068ffff : 30680000.pwm pwm@30680000
30690000-3069ffff : 30690000.pwm pwm@30690000
30890000-3089ffff : 30890000.serial serial@30890000
30a20000-30a2ffff : 30a20000.i2c i2c@30a20000
30a30000-30a3ffff : 30a30000.i2c i2c@30a30000
30aa0000-30aaffff : 30aa0000.mailbox mailbox@30aa0000
30b50000-30b5ffff : 30b50000.mmc mmc@30b50000
30b60000-30b6ffff : 30b60000.mmc mmc@30b60000
30bb0000-30bbffff : 30bb0000.spi fspi_base
30bd0000-30bdffff : 30bd0000.dma-controller dma-controller@30bd0000
30bf0000-30bfffff : 30bf0000.ethernet ethernet@30bf0000
32ec0000-32ecffff : 32ec0000.media-blk-ctrl media-blk-ctrl@32ec0000
33000000-33001fff : 33000000.dma-apbh dma-apbh@33000000
38300000-38300423 : hantrodec0
38310000-38310423 : hantrodec0
38320000-383207cf : hx280enc
3d800000-3dbfffff : 3d800000.ddr-pmu ddr-pmu@3d800000
40000000-943fffff : System RAM
40480000-418bffff : Kernel code
418c0000-41b3ffff : reserved
41b40000-41d1ffff : Kernel data
43000000-4300bfff : reserved
4501a000-498dbfff : reserved
94400000-a43fffff : reserved
a4400000-bfffffff : System RAM
ba000000-bfffffff : reserved
100000000-17fffffff : System RAM
17b000000-17effffff : reserved
17f180000-17f1dffff : reserved
17f1e0000-17f7e0fff : reserved
17f7e1000-17f83cfff : reserved
17f83f000-17f847fff : reserved
17f848000-17fffffff : reserved

Re: DDR stress test cause system hang in Linux

BiyongSUN — Mon, 06 Dec 2021 07:52:50 GMT

So far, haven't seen problem.

root@imx8mqevk:~# free
total used free shared buff/cache available
Mem: 3037148 386204 2563424 9004 87520 2557760
Swap: 0 0 0
root@imx8mqevk:~#
root@imx8mqevk:~# uname -a
Linux imx8mqevk 4.14.98-imx_4.14.98_2.0.0_ga+g5d6cbea #1 SMP PREEMPT Sun Apr 14 10:53:57 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
root@imx8mqevk:~# memtester 2048M 1000
memtester version 4.3.0 (64-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 2048MB (2147483648 bytes)
got 2048MB (2147483648 bytes), trying mlock ...locked.

Loop 2/1000:
Stuck Address : ok
Random Value : ok
Compare XOR : ok
Compare SUB : ok
Compare MUL : ok
Compare DIV : ok
Compare OR : ok
Compare AND : ok
Sequential Increment: ok
Solid Bits : ok
Block Sequential : ok
Checkerboard : ok
Bit Spread : ok
Bit Flip : testing 359

Re: DDR stress test cause system hang in Linux

alif_chen — Mon, 06 Dec 2021 07:59:21 GMT

I enabled OPTEE. Finally, I find out the problem comes from reserved memory. It do not really locked.

The following commit solved my problem.

https://source.codeaurora.org/external/imx/imx-optee-os/commit/?h=imx_5.4.47_2.2.0&id=995908f2a91c06ccd3e1bab7bc57325ecf270f62