iMX8QXP: Flickering on dual channel LVDS display

[mod42]

We have a flickering problem with a dual channel LVDS display (1920x720) connected to our custom iMX8QXP board. When running a simple openGLES application (source and binary attached) we see some flickering in the upper left corner of the display (see also attached video). It looks like drawing is not done when the buffer is presented on the screen. We can reproduce this with the same openGLES code using VDK to init the window. If no drawing is done (eglSwapBuffers not called) the image is stable and no artifacts are visible)

The problem first showed up after updating from 5.10.72 based BSP to the current 5.15.71 based one. We can't reproduce this on the iMX8QXP MEK (using single channel IMX-LVDS-HDMI). It is also not reproducibl on our custom board using a single channel LVDS display (1024x600).

[qiang_li-mpu_se · Accepted solution]

Attached is the kernel patch to fix the g2d memory leak issue, the patch will be included in next BSP release.

By the way, from your captured video "flicker.mp4", the display content is not correct, it looks like only one LVDS channel is working. 

I tested the split mode LVDS panel "jdi,tx26d202vm0bwa" with dts file "imx8qxp-mek-jdi-wuxga-lvds0-panel.dtb", run the binary "waylandTest" with default released images, there is no flickering issue. 

[mod42]

Hi @qiang_li-mpu_se 

Thanks for the patch. This fixes the memory leak. Unfortunately I am not able to reproduce the flickering problem anymore. Even with the opengl renderer I don't see any artifacts. So, the case can be closed. Thank you. 

[Dhruvit]

Hi @mod42,

I hope you are doing well.

Please provide me with the device tree file for further debugging.

Please make sure that you have referred to  imx8x-mek-it6263-lvds0-dual-channel.dtsi

Did you make any other changes between  5.10.72  and 5.15.71 BSP?

Please refer to Chapter - 10 Wayland and Weston and 13.5.9.1 Setting vProfiler Environment Variables for OGL/OES Profiling in i.MX Graphics User's Guide

Thanks & Regards,
Dhruvit Vasavada

[mod42]

Hello @Dhruvit 

Please find the device tree (source, compiled and decompiled) attached.

Whats also different is that we now use gl-renderer for weston. In 5.10.72 we used g2d-renderer but this is causing a significant memory leak in the linux kernel (I can supply you with additional informations to this if you are interested)

imx8x-mek-it6263-lvds0-dual-channel.dtsi was used as template four our device tree.

What can I do using vProfiler to support you in reproducing our problem?

Regards

[Dhruvit]

Hi @mod42,

I hope you are doing well.

Please try using g2d pipeline.

Please provide me with more information regarding the memory leak.

Does using g2d with gstreamer also cause such problems?

Thanks & Regards,
Dhruvit Vasavada

[mod42]

Hi @Dhruvit 

With the g2d renderer the flickering is gone. But the memory leak is back

Here is what I have done based on the 5.15.71-.2.2.0 SD card image on IMX8QXP-MEK to trigger the leak:

- Recompile the kernel with CONFIG_DEBUG_KMEMLEAK
- Put the kernel on the SDCARD replacing existing Image
- Edit /etc/systemd/system/graphical.target.wants/weston.service 
  -> appending --use-g2d=1 to the ExecStart= entry so that weston uses g2d
- Boot the board
- Verify that weston uses g2d by inspecting /var/run/user/0/weston.log
  -> Loading module '/usr/lib/libweston-10/g2d-renderer.so'
- Start any OpenGLES application
  ->  I used /opt/imx-gpu-sdk/GLES3/S08_EnvMappingRefraction/GLES3.S08_EnvMappingRefraction_Wayland
- Kernel prints new memory leak findings from time to time
  -> kmemleak: 1870 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
- Inspect /sys/kernel/debug/kmemleak
  -> Lot of memory leaks are reported. All originating from the same location in the kernel
    unreferenced object 0xffff00081d05f400 (size 128):
      comm "weston", pid 571, jiffies 4294925206 (age 985.704s)
      hex dump (first 32 bytes):
        40 f4 05 1d 08 00 ff ff 20 da d0 09 00 80 ff ff @....... .......
        c5 66 9b a6 1e 00 00 00 e0 bc 43 0b 00 80 ff ff .f........C.....
      backtrace:
        [<00000000883faf4f>] kmem_cache_alloc+0x204/0x2b0
        [<00000000a2e0270b>] dpu_be_get_fence+0x28/0xf4
        [<000000001be5915c>] imx_drm_dpu_get_param_ioctl+0x60/0x100
        [<00000000a48a8e1c>] drm_ioctl_kernel+0xc8/0x120
        [<00000000ddd40938>] drm_ioctl+0x224/0x44c
        [<000000009760255e>] __arm64_sys_ioctl+0xac/0xf0
        [<000000003178fe37>] invoke_syscall+0x48/0x114
        [<0000000003655d72>] el0_svc_common.constprop.0+0xcc/0xec
        [<00000000c0fbd212>] do_el0_svc+0x24/0x90
        [<000000009f2808b8>] el0_svc+0x20/0x60
        [<00000000ba7d73d8>] el0t_64_sync_handler+0xe8/0x114
        [<00000000f9687662>] el0t_64_sync+0x1a0/0x1a4
- Watching /proc/slabinfo shows constantly increasing kmalloc-128 allocations
  -> watch -n 1 'cat /proc/slabinfo | grep kmalloc-128'

Letting the system run for hours in this state then leads to an out of memory condition and system hang. We discovered this on our custom board in our CI tests which are running without reboot for 24 hours. Due to only 1GB of RAM the OOM condition is reached quicker than on the MEK.

Using an unmodified kernel the memory leak is also there. Watching /proc/slabinfo will verify that. The only advantage of using CONFIG_DEBUG_KMEMLEAK is that it shows the location of the leak.

[Dhruvit]

Hi @mod42,

I hope you are doing well.

Does using the Gstreamer pipeline for longtime causes memory leak issues?

This seems to be weston's memory leak issue.

Try to get information like which part is causing a memory leak.

Please make sure that your application is releasing unused memory properly.

A similar issue was seen with the Weston application. Please make sure weston contains changes in patch mentioned below.

Thanks & Regards,
Dhruvit Vasavada

[mod42]

Hi @Dhruvit

Thanks for your reply. But the memory leak is not in weston itself (the patch is almost four years old and doesn't apply. It even mentions the file libweston/compositor-drm.c which is not there in lf-5.15.71-2.2.0 which is based on mainline weston 10.0.1)

The memory leak is in the linux kernel as mentioned in my previous post. The leak is presumably caused here called from here which is an ioctl called from weston.

I debugged weston and found that the origin of the ioctl is in libg2d

(gdb) bt
#0 ioctl () at ../sysdeps/unix/sysv/linux/aarch64/ioctl.S:26
#1 0x0000ffffa6a96ff8 in drmIoctl (fd=14, request=3221513282, arg=0xfffff3dd96f4) at ../../../../../../src/libdrm-imx-github/xf86drm.c:629
#2 0x0000ffffa61cb9a8 in dpuIoctlGetFenceFD () from /mnt/sysroot-imx8x/usr/lib/libg2d.so.2
#3 0x0000ffffa61f594c in g2d_renderer_repaint_output (output=0x27e9e370, output_damage=0xfffff3dd97b0) at ../../../../../../src/weston-imx-github/libweston/renderer-g2d/g2d-renderer.c:1177
#4 0x0000ffffa646c5ac in drm_output_render_g2d (state=0x27f54b80, damage=0xfffff3dd9908) at ../../../../../../src/weston-imx-github/libweston/backend-drm/drm-gbm.c:515
#5 0x0000ffffa645b3a0 in drm_output_render (state=0x27f54b80, damage=0xfffff3dd9908) at ../../../../../../src/weston-imx-github/libweston/backend-drm/drm.c:429
#6 0x0000ffffa645b7c0 in drm_output_repaint (output_base=0x27e9e370, damage=0xfffff3dd9908, repaint_data=0x27cd0770) at ../../../../../../src/weston-imx-github/libweston/backend-drm/drm.c:546
#7 0x0000ffffa6d0c574 in weston_output_repaint (output=0x27e9e370, repaint_data=0x27cd0770) at ../../../../../../src/weston-imx-github/libweston/compositor.c:2956
#8 0x0000ffffa6d0c7f0 in weston_output_maybe_repaint (output=0x27e9e370, now=0xfffff3dd99f0, repaint_data=0x27cd0770) at ../../../../../../src/weston-imx-github/libweston/compositor.c:3023
#9 0x0000ffffa6d0c9e4 in output_repaint_timer_handler (data=0x27cca2c0) at ../../../../../../src/weston-imx-github/libweston/compositor.c:3091
#10 0x0000ffffa6c9b074 in wl_timer_heap_dispatch (timers=0x27cc8888) at ../../../../../../src/wayland/src/event-loop.c:526
#11 wl_event_loop_dispatch (loop=0x27cc8840, timeout=timeout@entry=-1) at ../../../../../../src/wayland/src/event-loop.c:1020
#12 0x0000ffffa6c98a6c in wl_display_run (display=0x27cc8440) at ../../../../../../src/wayland/src/wayland-server.c:1408
#13 0x0000ffffa6f1f140 in wet_main (argc=1, argv=0xfffff3dda268, test_data=0x0) at ../../../../../../src/weston-imx-github/compositor/main.c:3681
#14 0x0000000000400724 in main (argc=5, argv=0xfffff3dda268) at ../../../../../../src/weston-imx-github/compositor/executable.c:33

which is called from here 

As all this stuff (libg2d.so.2, g2d-renderer.c and dpu-blit driver) are properiatary NXP stuff upstreaming this problems makes no sense. Maybe you or someone else can have a deeper look into this.

[mod42]

Hi @Dhruvit 

Do you have an update on this issue?

Regards