Custom iMX8mq board freezes after booting Linux

[matthewcampbell]

Hi All,

We have a custom iMX8mq board we are doing bring up on. Clocks and reset are very similar to the EVK, but we have deviated on some other hardware:

• 1Gb DDR4 (EVK has 3GB LPDDR4)
• Serial terminal on USART4 (EVK on USART1)
• PMIC/power supply configuration is different, but meets datasheet requirements
• Many different peripherals on SPI and I2C

First thing we did was run the DDR calibration and stress test. It passed >30 iterations of the stress test, and I have left it running over this weekend. The code generated was integrated with u-boot to do DDR training at boot time. Our board is using a Yocto Linux based off the official Linux L4.9.51 for i.MX 8MQuad GA release.

Here is the behavior:

I load U-boot and Linux using the NXP mfgtool. From the time the Linux kernel starts booting, 15 seconds later, the system freezes. I am able to log into the shell using the serial terminal prior to the freezing, but I only have about 4 seconds to do anything it freezes. The system seems to be completely unresponsive after this. I connected with JTAG to see what the processor is doing, and the PC is somewhere in the 0x0090_0000-0x0091_FFFF range which is on chip SRAM in the physical memory map. I am able to step the processor and when disassembled it looks like real code, but I'm not sure what it's doing. This is even more odd that this memory range is not in any valid virtual memory region that Linux sets up. I have no idea what code might be running in OCRAM, does Linux put something in there (interrupt/exception vectors or something)?

I have a couple theories. First, I think somehow the change from 3Gb to 1Gb of memory is causing Linux to go off and try to use memory that isn't there resulting in some kind of hard CPU exception. I have attempted to modify u-boot and the device tree to reflect 1Gb of memory, but not sure if I have done all that is needed.

Second, I based the devicetree for our custom board on the EVK device tree. However, it's possible that I either stripped out something essential, or

Finally, it might be possible that there is some additional changes we need to make to support DDR4 memory as opposed to LPDDR4.

I have spent 2 days trying different things and trying to learn something about what is going on. I've run out of ideas at this point, so any help would be appreciated.

[arunkvnss]

Hi, I also having a some what similar confusion on this, imx8 family supports configuring the pmic through the ATF (arm trusted firmware) ?. or whether it is configured through the IPC (inter processor communication to the SCFW firmware ) directly ?. During the frequency changes the ATF firmware is notified using the smc call, But how come the SCFW firmware will get to know about the changes ?

[igorpadykov]

Hi Matt

usually low power / cpufreq drivers codes are running from OCRAM, so

since pmic is different, issue may be caused by these drivers, described in

sect.2.3 Power Management attached Linux Manual.

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[matthewcampbell]

Hi Igor,

Thanks for the reply. That was our next step to check all power supplies and make sure they aren't dipping at any point around our freeze up.

I tried to dig up the code that may be running out of the OCRAM. Do you have any details on where to look in the Linux kernel for that. I did read the ref manual you attached, but it doesn't make mention of the iMX8. I've looked through the code in the Linux L4.9.51 for i.MX 8MQuad GA release of the kernel, and there doesn't seem to be any imx8 specific source files. Is it possible that it is mainly making reuse of iMX6 or iMX7 code?

[igorpadykov]

Hi Matt

sect.2.3.3.2.1 Source Code Structure shows specifc for i.MX8M files.

Best regards
igor

[matthewcampbell]

Thank you Igor. That seems to be specifically code for the CPU scaling, I was wondering what code is used for power management and other CPU features as there appears to be iMX6/7 versions, but no iMX8 (for examples arch/arm/mach-imx/mach-imx6q.c and pm-imx7.c in the same folder).

Also wondering, have there been any updates to any of the hardware guides or erratas we should be aware of? We are still having trouble tracking down this issue and want to validate our design against the requirements in these docs.

Thanks.

[igorpadykov]

Hi Matt

one can also look on:

linux/drivers/soc/imx/gpc-psci.c, pm-domains.c
linux/drivers/soc/soc-imx8.c

latest documentation is available on

i.MX 8M Applications Processor | Arm® Cortex®-A53, Cortex-M4 | 4K display resolution |NXP 

Best regards
igor

[matthewcampbell]

Hi Igor, that was great information. From that I figured out that the code I always end up in the ocram (0x910000~0x91FFFF) is the Arm Trusted Firmware bl31.bin. On the page for the imx8 DDR tool there is mention of needing to apply a patch to ATF if using custom memory. Is this true in my my case? Also, it looks like bl31.bin does some setup of DDR, is everything in there okay for DDR4 as opposed to the LPDDR4 used on the EVK?

Overall I'm starting to become suspicious of software changes needed to support our different memory configuration compared to the EVK (1GB DDR4 vs 3GB LPDDR4).

DDR tool page mentioning the patch:

i.MX 8M DDR Tool Release 

ATF code used in my Yocto buold:

imx-atf - i.MX ARM Truststed firmware 

[igorpadykov]

Hi Matt

MX8M_DDR_Tool_User_Guide.docx included in DDR tool package

describes how add support for new memory.

Best regards
igor

[matthewcampbell]

Hi Igor, that is the docuemt I followed to calibrate and prepare u-boot for our DDR4 memory. I'll re-review it to make sure I didn't miss anything.

What is troubling for me is that I can boot the same build (the only difference being U-boot DDR configuration and Linux device tree) on the EVK and it runs with no freeze. Has NXP physically booted an iMX8m board using DDR4 memory and is there a reference to this code somewhere?

[igorpadykov]

Hi Matt

if your board has not pmic EVK image may not run on it, as it communicates with pmic

from low power/cpufreq drivers. Also may be useful to check:

https://community.nxp.com/message/1005660 

Best regards
igor

[matthewcampbell · Accepted solution]

Hi Igor,

I found the cause of the issue. It turns out that the iMX8 software currently does not support dynamic voltage frequency scaling (DVFS) for DDR4 or DDR4L, just LPDDR4. It looks like the bl31.bin from ATF is responsible for performing the DVFS for DDR, which is why JTAG was reporting being stuck in there when my board froze. I fixed the problem by disabling the busfreq driver in my device tree. You can add the following to your device tree to fix it.

/ {

    busfreq {
        status = "disabled";
    };

    ... rest of your device tree here ...

}

There was no mention of this limitation in any of the documentation (the DDR tool, the BSP, etc) that I could find. I recommend updating at least the DDR tool documentation to call this out as it could save someone a lot of headaches in the future.

I also have the question of if DVFS will be supported in the future, and if simply disabling it is a viable long term soltuion, but I'll open a new thread on the forum to discuss this.

Thanks for your help along the way Igor!

best,

~Matt