Hardware Overlay errors on IMX6Q with Android O8

[thomaselste]

Hi everyone,

on a custom IMX6Q-Board with 4 GB DDR3 RAM running Android O8 we have an issue concerning hardware overlays. Everything seems to work fine. But when starting an application the display shows for 2-3 seconds a greenish kind of pixel mess, which vanishes again, if the application has finished loading:

These display errors disappear if I disable overlays (Developer options/Disable HW overlays). Hence why I think it is a hardware overlay issue of the GPU.

DDR3 RAM was calibrated on our board using stress test tool V2.92.

And I found some older thread mentioning a similar behaviour when using more than 2 GB RAM. And indeed overlays work, if I limited the RAM to 2 GB using mem=2GB kernel parameter. As one mentioned solution was to change the GPU base address in the device tree, I checked the current settings. But as far as I can tell the GPU physical base address is already set to the suggested value in the imx6q.dtsi device tree file:

gpu@00130000 {
 compatible = "fsl,imx6q-gpu";
 reg = <0x00130000 0x4000>, <0x00134000 0x4000>,
 <0x02204000 0x4000>, <0x10000000 0x0>,
 <0x0 0x8000000>;

Apart from calibration, is there something else, which has to be done when running Android O8 with 4 GB of RAM? Or is there some other way, to avoid this overlay errors?

Best regards

Thomas

[jhkim3]

Hi Thomas,

Are you still experiencing this problem?
We recently tested with only CPU changes from IMX6Q to IMX6QP.
Running the application on IMX6QP does not cause the above problem, but causes MMU and GPU kernel dump.
===================================================
[  240.812767] GPU[8](ChipModel=0x320 ChipRevision=0x5303):
[  240.818120] **************************
[  240.822384] ***   MMU ERROR DUMP   ***
[  240.827394] **************************
[  240.832071]   MMU status = 0x00000002
[  240.840736]   MMU0: page not present
[  240.847752]   MMU0: exception address = 0xFB1A27C0
[  240.854019]     MTLB entry = 1004
[  240.857360]     STLB entry = 418
[  240.860657]     Offset = 0x000007C0 (1984)
[  240.864769]     Page table entry = 0x00000002
[  240.869192] GPU[8](ChipModel=0x320 ChipRevision=0x5303):
[  240.876574] **************************
[  240.880335] ***   GPU STATE DUMP   ***
[  240.884164] **************************
[  240.887924]   axi      = 0x0000003B
[  240.891457]   idle     = 0x7FFFFFF8
[  240.894961]     FE not idle
[  240.897761]     DE not idle
[  240.900587]     PE not idle
[  240.903391]   DMA appears to be stuck at this address:
[  240.908539]     0xFFFCE400
[  240.911355]   dmaLow   = 0x08020484
[  240.914855]   dmaHigh  = 0x019700CA
[  240.918351]   dmaState = 0x00000905
[  240.921892]     command state       = 5 (PAR_LOAD1_ST)
[  240.927046]     command DMA state   = 1 (CMD_START_ST)
[  240.932444]     command fetch state = 2 (FET_VALID_ST)
[  240.937790]     DMA request state   = 0 (REQ_IDLE_ST)
[  240.943269]     cal state           = 0 (CAL_IDLE_ST)
[  240.948607]     VE request state    = 0 (VER_IDLE_ST)
...
===================================================

I think there is probably some problem between 4GB and GPU.

In addition, we can't even access the menu to set the HW overlay because of the above problem(MMU and GPU dump)

We will share with you if the information is updated.

Additionally
We access the 4GB of memory on a 64bit bus with one HW CS0 pin.
How about your 4GB HW circuit configuration?

============================================
Device Information 
Memory type: DDR3
Manufacturer: Micron
Memory part number: MT41K512M16
Density of each DDR3 device (Gb): 8
Number of DRAM devices per chip select 4
Density per chip select (Gb)1: 32
Number of Chip Selects used2 1
Total DRAM density (Gb) 32
Number of ROW Addresses2 16
Number of COLUMN Addresses2 10
Number of BANKS2 8
Bus Width (input 16, 32, or 64 bits)2 64
Clock Cycle Freq (MHz)3 533
Clock Cycle Time (ns) 1.876
============================================

[thomaselste]

Hello Kim,

the problem persists on our hardware, too. We couldn't find a solution so far. Our DDR setup is basically the same as yours (same chip type, one chip select, 64 bit bus width, same row, column and bank number).

In the meantime thorstenwronski‌ was so kind to take look at our DDR calibration setup. But apart from some minor improvements, nothing needed to be changed. So for my part I would rule out some DDR setup or calibration problem at this point.

Best regards

Thomas

[jhkim3 · Accepted solution]

I think I found the cause of the problem.

Try to use only cma in the mxc-ion section of the device tree.

(arch/arm/boot/dts/imx6qdl.dtsi)

As follows:

         imx_ion {
             compatible = "fsl,mxc-ion";

             heap_cma@0 {
                 compatible = "fsl,cma";
                 heap-name = "cma";
                 heap-type = <4>;
                 heap-id = <0>;
             };
 /*
             heap_sys_contig@0 {
                 compatible = "fsl,sys-contig";
                 heap-name = "sys-contig";
                 heap-type = <1>;
                 heap-id = <1>;
             };

             heap_sys@0 {
                 compatible = "fsl,system";
                 heap-name = "system";
                 heap-type = <0>;
                 heap-id = <2>;
             };
 */
         };

If it is resolved, the reason is as follows.

As in the answers above, the GPU can only handle physical memory in the 0x10000000 to 0x80000000 area.

But if you use more than 2GB (like us 4GB)
The linux HIGH_MEMORY area is included from 0x80000000 to 0xFFFF0000.

Then, there is a case that ION SYSTEM CONTIG allocates an area of 0x80000000 ~ 0xFFFF0000 and informs the GPU.

That's why this happens.

NXP would never have tested OREO8 at 4GB.
Since they are not using ION SYSTEM CONTIG in versions prior to OREO8, they would have concluded that there is no problem at 4GB.

If you see a fix, please let me know as soon as possible.

(The content can be awkward because I am using a translator. please understand.)

[twmemphis]

Wow, really good that it is finally solved.

As you know, the Micron DDR3L 8Gb (MT41K512M16HA) is now End-Of-Life.

There are alternative solutions from other manufacturers which I would offer to any of you as FREE samples if you'd like to try! Just contact me please. I will ship them worldwide, but I wished to have feedback about the function in any i.MX6 system.

I have one more question: Would the i.MX6 also work with two ranks of DRAM? This requires two sets of Chip-Select (CS), two Clock-Enable (CKE), two On-Die-Termination (ODT) and two calibration (ZQ) pins.

If this was supported, there will soon be the first 16Gb DDR3L components (2 Gigabyte per chip) available from Intelligent Memory.

Regards,

Thorsten Wronski

Memphis Electronic AG (Germany)

[thomaselste]

Hello Kim,

that's great news! I've disabled all the ION memory sources in the device tree except the CMA one as you suggested. And the overlay errors are completely gone! Graphic effects are looking normal to me, now.  Looks to me as you really did find the solution for this problem.

My setup is now: CMA area below 2 GB defined in the device tree, no cma kernel command line parameter and ION using only CMA as source

I never thought of that the Android graphics systems might be using ION to allocate some buffers above the 0-2 GB range and hand them to the GPU for processing/displaying.

Thank you very much for investigating this and coming up with this solution

Thomas

[twmemphis]

I have some DDR3 config files for 4GB here to compare.

But if you have no problem when setting 2GB and you have problem with 4GB, then is looks as if you did not limit the max memory size to 3840MB  which is a requirement according to another thread on this forum.

Check the register MMDCx_MDASP bits 0 to 6 --> CS0 end. Here you can limit to 3840MB by setting to 7F

setmem/32  0x021b0040 = 0x0000007F   // CS0_End

Also I offer to check your DDR config files for possible errors. Just send me a private message.

I can also provide you with some samples of the IM8G16D3FCBG-125 (or -125I for industrial temp) components from Intelligent Memory.  My company ships those samples free of charge to potential customers.

Another option: The i.MX6 should also allow to work with two Chip-Selects. Eventually it would make sense to wire two CS, CKE, ODT, ZKE to each DRAM and then try the 2CS Version of the 8Gb component with part# IM8G16D3FCDG-125 ? That would give two seperate ranks of 2GB instead of one with 4GB.

Regards,

Thorsten Wronski

Memphis Electronic AG (Germany)

[thomaselste]

Hello Thorsten,

thank you for your reply.

Check the register MMDCx_MDASP bits 0 to 6 --> CS0 end. Here you can limit to 3840MB by setting to 7F

setmem/32 0x021b0040 = 0x0000007F // CS0_End

MDASP gets set to 0x7f via DCD. And I verified it again using U-Boot:

=> md 0x021b0040 1 
021b0040: 0000007f

Also I offer to check your DDR config files for possible errors. Just send me a private message.

Thank you for this kind offer. I took the liberty to send you a PM with my device configuration data. But I have the feeling, this is more of a GPU issue than a problem with the RAM as every memory test I've run succeded so far.

Configurations with 4 GB DDR running Android and using LCD interface seem to be rare, so this is probably not a very well tested setup. Or can maybe someone from NXP confirm, they have this exact configuration (IMX6Q, 4 GB DDR3, 16 Bit RGB LCD (not LVDS), Android O8) successfully running without HW overlay errors?

I can also provide you with some samples of the IM8G16D3FCBG-125 (or -125I for industrial temp) components from Intelligent Memory.  My company ships those samples free of charge to potential customers.

For this device it is probably to late. But I forwarded your offer to our hardware departement so we maybe will come back to your for future designs.

Best regards

Thomas

[Bio_TICFSL]

Hi Thomas,

Are you seeing this greenish kind of pixel mess on all application or any specific application? If it is specific application, can you please provide the same?"

Regards

[thomaselste]

Hi,

thank you for your reply. It happens every time for 2-3 seconds when an overlay animation gets displayed by Android. For example if you start a new application (doesn't matter which). After these 2-3 seconds, the display returns to normal again. If I had to guess, I would say something with the overlay framebuffer or its allocation isn't right. But for a simple "overlay framebuffer memory is overwritten" the pattern seems to be to regular.

Best regards

Thomas

[Bio_TICFSL]

Hi,

We are unable to reproduce issue at our end with imx6q processor having 4GB RAM and lvds(1024x768) display.

Can you please provide some more detail for the same?

• Which video interface are you using? And what is the resolution for it?
• Have you changed any uboot bootargs related to GPU or video outputs? It will be great if you can share uboot bootargs.
• Can you please also share init.rc file?

Regards

[thomaselste]

Hi,

thank you for your reply. Let me try to answer your questions.

We are using the LCD interface in RGB666 mode. The displays resolution is 800x480.

Bootargs are as follows:

console=ttymxc0,115200 init=/init video=mxcfb1:off video=mxcfb2:off video=mxcfb3:off
vmalloc=256M androidboot.console=ttymxc0 consoleblank=0 androidboot.hardware=freescale
cma=448M galcore.contiguousSize=33554432

The mxcfb0 entry is missing, as all other parameters are set in the device tree (but this seems to not make any difference, I tried using the mxcfb0 parameter, too):

mxcfb1: fb@0 {
   compatible = "fsl,mxc_sdc_fb";
   disp_dev = "lcd";
   interface_pix_fmt = "RGB666";
   mode_str ="ETM-WVGA";
   default_bpp = <24>;
   int_clk = <0>;
   status = "okay";
};
lcd@0 {
   compatible = "fsl,lcd";
   ipu_id = <0>;
   disp_id = <0>;
   default_ifmt = "RGB666";
   pinctrl-names = "default";
   pinctrl-0 = <&pinctrl_ipu1>;
   status = "okay";
};

And timings are done in the LCD interface driver (mxc_lcdif.c) via a struct entry:

 { "ETM-WVGA", 60, 800, 480, 33410, 88, 40, 33, 10, 128, 2, 0, FB_VMODE_NONINTERLACED, 0,}

And I have attached init.rc init.freescale.rc and init.freescale.imx6q.rc to this post.

Best regards

Thomas

[Bio_TICFSL]

Hi Thomas,

GPU hardware has limitation to access the physical memory address > 0x80000000, for 4GB memory, you can assign CMA memory to low 2GB memory in device tree to avoid such issue:

Kernel: arch/arm/boot/dts/imx6q.dtsi
Modify to following:
linux,cma {
                        compatible = "shared-dma-pool";
                        reusable;
                        alloc-ranges = <0x40000000 0x40000000>;
                        size = <0x40000000>;
                        linux,cma-default;
          };
 It means that CMA address begin from 0x40000000, CMA size is 0x40000000(1GB), 0x40000000 + 0x40000000 = 0x80000000.
You can adjust this two values, just make sure that CMA begin and end address < 0x80000000 

Regards

[jhkim3]

Hi

We have the same problem.

##################################################################

U-Boot 2017.03 (Apr 25 2019 - 14:38:58 +0900)

CPU:   Freescale i.MX6Q rev1.5 996 MHz (running at 792 MHz)
CPU:   Extended Commercial temperature grade (-20C to 105C) at 58C
Reset cause: POR
Model: Freescale i.MX6 Quad SABRE Smart Device Board
Board: MX6-SabreSD
DRAM:  3.8 GiB

......

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.17 (jhkim@spfw3) (gcc version 4.9.x 20150123 (prerelease) (GCC) ) #29 SMP PREEMPT Thu Apr 25 11:06:44 KST 2019
[    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c53c7d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt:Machine model: Freescale i.MX6 Quad SABRE Smart Device Board
[    0.000000] cma: Reserved 448 MiB at 0x2a000000
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] percpu: Embedded 15 pages/cpu @f76bd000 s31040 r8192 d22208 u61440
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 981263
[    0.000000] Kernel command line: console=ttymxc0,115200 init=/init video=mxcfb0:dev=ldb,if=RGB24,bpp=32 video=mxcfb1:off video=mxcfb2:off video=mxcfb3:off vmalloc=128M androidboot.console=ttymxc0 consoleblank=0 androidboot.hardware=freescale cma=448M galcore.contiguousSize=33554432 androidboot.selinux=permissive buildvariant=userdebug androidboot.serialno=1907b9d4ee6a3958 androidboot.soc_type=imx6q androidboot.storage_type=emmc
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Memory: 3419596K/3932156K available (12288K kernel code, 967K rwdata, 3960K rodata, 1024K init, 579K bss, 53808K reserved, 458752K cma-reserved, 3022844K highmem)

...

##################################################################

How can I solve this problem?

We connected Micron DDR3L MT41K512M16 x 4.

At the same ddr setting,

Only the MMDC1_MDCTL[17:16](DDR data bus size) bits could be controlled to change to 4GB or 2GB.

DATA 4, 0x021b0000, 0x851A0000 // 4GB

DATA 4, 0x021b0000, 0x85190000 // 2GB

The above problem occurs when setting 4GB, There is no problem when setting 2GB.

It does not seem to be a timing issue, nor is it a circuit problem.

[thomaselste]

Hi,

unfortunately the problem with our board persists, too. Btw: We are using the same DDR ICs: MT41K512M16

Best regards

Thomas

[thomaselste]

Hi,

sorry for my late reply. I was only now able to try out your suggestion.

My imx6q.dtsi didn't contain the linx,cma node, so I've added a complete reserved-memory entry:

reserved-memory {
   #address-cells = <1>;
   #size-cells = <1>;
   ranges;

   linux,cma {
      compatible = "shared-dma-pool";
      reusable;
      alloc-ranges = <0x40000000 0x40000000>;
      size = <0x40000000>;
      linux,cma-default;
   };
};

I had to remove the cma parameter from the kernel command line, or else the DT entry was ignored. Now the kernel seems to use the values correctly. The kernel log shows:

Reserved memory: created CMA memory pool at 0x40000000, size 1024 MiB 
OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool

Unfortunately this doesn't change anything regarding the overlay not showing correctly:

<script src="https://players.brightcove.net/6153537070001/default_default/index.min.js"></script>

To get some more information, what is going on, I've tried to enable some additional output of the video driver using the galcore.showArgs=1 parameter. Maybe this helps narrowing down the error:

Galcore version 6.2.2.96445 
Galcore options: 
 irqLine = 20 
 registerMemBase = 0x00130000 
 registerMemSize = 0x00004000 
 irqLine2D = 21 
 registerMemBase2D = 0x00134000 
 registerMemSize2D = 0x00004000 
 irqLineVG = 22 
 registerMemBaseVG = 0x02204000 
 registerMemSizeVG = 0x00004000 
 contiguousSize = 0x04000000 
 contiguousBase = 0x00000000 
 bankSize = 0x00000000 
 fastClear = -1 
 compression = -1 
 signal = 48 
 powerManagement = 1 
 baseAddress = 0x10000000 
 physSize = 0x80000000 
 logFileSize = 0 KB 
 recovery = 0 
 stuckDump = 0 
 gpuProfiler = 0 
 irqs = -1, 
-1, -1, 
-1, -1, 
-1, -1, 
-1, -1, 
-1, 

Thank you for your reply. Best regards

Thomas

[jhkim3]

Did you solve this problem?
If so, can you share a solution?