AnsweredAssumed Answered

Debugging memory instability with 4.1/yocto 2.1 jethro on 6sl

Question asked by jayakumar2 on Nov 4, 2016
Latest reply on Nov 7, 2016 by igorpadykov

Hi,

 

I'm looking for advice on debugging a memory instability with linux 4.1. I noticed that when running on the linux 4.1 kernel, the 6sl board that I am testing on is unstable and regularly crashes with kernel panics within about 30 minutes. The exact same board in the same setup is very stable when running 3.0.35.

 

By running memtester, I found that I consistently get AND test errors on 4.1. Eg:
Compare AND : ok
FAILURE: 0xedb8450c != 0xedb6450c at offset 0x025987e0.

FAILURE: 0x98f7bce2 != 0x98f5bce2 at offset 0x005a47e0.

 

I have attached complete bootup sample logs for that.

 

The same memtest run on 3.0.35 passes consistently.

 

The memory test  errors seem to be mostly single bit errors which makes me wonder if the root cause is drive strength or memory refresh configuration or pll frequency that is different in 4.1.

Comparing the boot sections shows:
working case 3.0.35:
> Memory: 1033300k/1033300k available, 15276k reserved, 0K highmem

failing case 4.1:
< Memory: 700176K/1048576K available (7178K kernel code, 396K rwdata, 2440K rodata, 412K init, 426K bss, 20720K reserved, 327680K cma-reserved, 0K highme

I didn't do any custom memory configuration for our board, ie: use default 6slevk settings. I just used the default imx6sl devicetree and u-boot settings. The 2016 u-boot code does:
int dram_init(void)
{
gd->ram_size = imx_ddr_size();

As shown by the log, that seems to correctly detect the 1GB memory.

 

I'm now working on trying to figure out if there's some ddr pll speed issue but haven't yet figured out where this is configured.

 

Any advice would be welcome!

 

Thanks!

Original Attachment has been moved to: memtest_pass_rhu_goodbrd_3.0.35.txt.zip

Original Attachment has been moved to: memtest_fail_rhu_goodbrd.txt.zip

Outcomes