topic Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows in i.MX Processors

Bug in Vivante i.MX6 Wayland drivers when destroying windows

cola — Wed, 25 Apr 2018 06:38:48 GMT

We are getting the following client-side crash with our compositor when destroying windows. Apparently, the proxy object is nullptr though it never shall be.
We are using SDK release 5.0.11.p8.4.
Thread 2 (Thread 3662.3673):
#0  wl_proxy_add_listener (proxy=0x0, implementation=0x74b621cc <gcsWL_FRAME_LISTENER>, data=data@entry=0x5e424d0c) at /usr/src/debug/wayland/1.9.0-r0/wayland-1.9.0/src/wayland-client.c:464
No locals.
#1  0x74b53544 in wl_callback_add_listener (data=0x5e424d0c, listener=0x74b621cc <gcsWL_FRAME_LISTENER>, wl_callback=<optimized out>) at /home/bamboo/automation/3.14.52-1.1.1/graphics_pkg/temp_build_dir/build-imx6qsabresd/tmp/sysroots/imx6qsabresd/usr/include/wayland-client-protocol.h:317
No locals.
#2  gcoOS_SetDisplayVirtualEx (Display=<optimized out>, Window=0x5e424c84, Context=0x5e424d0c, Surface=<optimized out>, Offset=0, X=0, Y=0) at gc_hal_user_wayland.c:1692
        swapInterval = -1
        ret = <optimized out>
        i = <optimized out>
        wl_window = 0x5e424c84
        egl_buffer = 0x5e424d0c
        wl_buffer = 0x5e457d40
        display = 0x19d608
#3  0x74a843ac in veglSetDisplayFlip (Display=Display@entry=0x19d6bc, Surface=<optimized out>, BackBuffer=BackBuffer@entry=0x5e4255dc) at gc_egl_platform.c:249
        status = <optimized out>
#4  0x74a7ec6c in veglSwapWorker (Display=0x19d6bc) at gc_egl_swap.c:741
        display = 0x19d6bc
        displayWorker = 0x5e4255cc
        currWorker = 0x5e4255cc
        bStop = 0
        __user_ptr__ = <synthetic pointer>
#5  0x75a4cf5c in start_thread (arg=0x65556440) at /usr/src/debug/glibc/2.24-r0/git/nptl/pthread_create.c:335
        pd = 0x65556440
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {2123239463, 1853849923, 1700095040, 2130704288, 0, 338, 0, 0, 2130704288, 1700093820, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        robust = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x75d42408 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:86 from /opt/sdk/cebis/sysroots/cortexa9hf-neon-mel-linux-gnueabi/lib/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

Bio_TICFSL — Fri, 27 Apr 2018 17:01:43 GMT

Hi Andreas,

Could you provide your test? I can not reproduced it.

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

cola — Mon, 30 Apr 2018 13:55:58 GMT

Hi, I created a small test case and put it on GitHub:

GitHub - cordlandwehr/imx6_testcase_wayland-client-crash: Testcase for https://community.nxp.com/thread/474517?sr=inbox&…

Please see the include README.md for how to reproduce the bug.

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

Bio_TICFSL — Wed, 02 May 2018 14:58:40 GMT

Hi Andreas,

I believe there are some bugs fix for segfault and crashes in most recent driver. Could you try the attached most recent gpu 5.x p8 driver to see if your issue is fixed.

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

cphealy1 — Thu, 03 May 2018 17:25:37 GMT

As an alternative to a potentially improved Vivante driver, it is now possible to run the latest upstream Weston (4.0) with mainline kernel, libdrm, and Mesa. There are optimizations in upstream Weston specifically for the Vivante GPU that result in very good performance on the i.MX6 when running with mainline kernel, libdrm, and Mesa. (Mesa is what provides the opensource driver for the Vivante GPU in the i.MX6 when running a non NXP-BSP based kernel.)

I've been running with this stack for some time now and it works quite well. It would require you move from whatever version of kernel you are currently running to Linux 4.16 though which may not be an option in your case.

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

jrtigerlee — Fri, 04 May 2018 23:06:35 GMT

Hi Chris ,

Do you have any guideline to porting weston 4.0 with mainline kernel can share with us?

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

cphealy1 — Thu, 10 May 2018 21:01:55 GMT

Hi Jrtiger,

There isn't really much in the context of "porting" it. If you take stock Linux 4.16 (or newer) with stock libdrm 2.4.89 (or newer) with stock Mesa 18.0 and build stock Weston 4.0, you should have a working stack that just works.

In the kernel, libdrm, and Mesa, you need to make sure "etnaviv" is enabled. (etnaviv is vivante spelled backwards....)

With this stack, LVDS and HDMI output should work correctly and the GPU will be used to accelerate Weston for compositing as well as the GPU being available for any applications that use OpenGL ES 2.0. (Qt 5.10.1 is a good example of this. It's will use OpenGL ES 2.0 for acceleration.)

With a 1080p display and the i.MX6qp, many applications operate at 60fps.

With the mainline stack, not only is the GPU properly supported but the VPU (Video decoder) also just works. You'll want to have gstreamer 1.14. With this combination, no special NXP plugins are required and 1080p video playback will be ~1% CPU load.

Regards,

Chris

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

jrtigerlee — Thu, 10 May 2018 23:07:41 GMT

Hi Chris ,

I was using Yocto morty to build system, and will try to use Yocto rocko or sumo to create this new stuff.

Thanks for your experience sharing.

I'll glad that if you have more GPU benchmark sharing , like glmark2 score in 1080P resolution.

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

cphealy1 — Sat, 12 May 2018 01:35:22 GMT

Hi Jrtiger,

Below is the output of glmark2-es2-wayland. The display is 1920x1080. The SoC is the i.MX6qp and it's running Weston (Wayland compositor). Performance is a bit lower than should be expected as our HW platform has the GPU shader clock reduced from the default 720Mhz to 594Mhz. Within the next few months, I expect the framerate going up substantially as there is a compositor bypass feature being worked on for Weston that gets turned on when running an application fullscreen. (Composition is not needed when there's only one application on the screen.)

root@RDU2:~ glmark2-es2-wayland --fullscreen
=======================================================
    glmark2 2014.03
=======================================================
    OpenGL Information
    GL_VENDOR:     etnaviv
    GL_RENDERER:   Vivante GC3000 rev 5450
    GL_VERSION:    OpenGL ES 2.0 Mesa 18.0.0.20180404-1
=======================================================
[build] use-vbo=false: FPS: 295 FrameTime: 3.390 ms
[build] use-vbo=true: FPS: 360 FrameTime: 2.778 ms
[texture] texture-filter=nearest: FPS: 333 FrameTime: 3.003 ms
[texture] texture-filter=linear: FPS: 331 FrameTime: 3.021 ms
[texture] texture-filter=mipmap: FPS: 322 FrameTime: 3.106 ms
[shading] shading=gouraud: FPS: 267 FrameTime: 3.745 ms
[shading] shading=blinn-phong-inf: FPS: 191 FrameTime: 5.236 ms
[shading] shading=phong: FPS: 129 FrameTime: 7.752 ms
[shading] shading=cel: FPS: 95 FrameTime: 10.526 ms
[bump] bump-render=high-poly: FPS: 168 FrameTime: 5.952 ms
[bump] bump-render=normals: FPS: 217 FrameTime: 4.608 ms
[bump] bump-render=height: FPS: 164 FrameTime: 6.098 ms
libpng warning: iCCP: known incorrect sRGB profile
[effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 46 FrameTime: 21.739 ms
libpng warning: iCCP: known incorrect sRGB profile
[effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 12 FrameTime: 83.333 ms
[pulsar] light=false:quads=5:texture=false: FPS: 205 FrameTime: 4.878 ms
libpng warning: iCCP: known incorrect sRGB profile
[desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 26 FrameTime: 38.462 ms
libpng warning: iCCP: known incorrect sRGB profile
[desktop] effect=shadow:windows=4: FPS: 96 FrameTime: 10.417 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 46 FrameTime: 21.739 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 45 FrameTime: 22.222 ms
[buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 58 FrameTime: 17.241 ms
[ideas] speed=duration: FPS: 162 FrameTime: 6.173 ms
[jellyfish] <default>: FPS: 58 FrameTime: 17.241 ms
[terrain] <default>: FPS: 4 FrameTime: 250.000 ms
[shadow] <default>: FPS: 132 FrameTime: 7.576 ms
[refract] <default>: FPS: 20 FrameTime: 50.000 ms
[conditionals] fragment-steps=0:vertex-steps=0: FPS: 243 FrameTime: 4.115 ms
[conditionals] fragment-steps=5:vertex-steps=0: FPS: 63 FrameTime: 15.873 ms
[conditionals] fragment-steps=0:vertex-steps=5: FPS: 231 FrameTime: 4.329 ms
[function] fragment-complexity=low:fragment-steps=5: FPS: 135 FrameTime: 7.407 ms
[function] fragment-complexity=medium:fragment-steps=5: FPS: 64 FrameTime: 15.625 ms
[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 132 FrameTime: 7.576 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 116 FrameTime: 8.621 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 60 FrameTime: 16.667 ms
=======================================================
                                  glmark2 Score: 146
=======================================================

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

sanjeevsharma — Tue, 17 Jul 2018 05:43:24 GMT

Dear Bio_TICFSl,

We have tried with 5.0.11.p8.6 & v6.2.2.p0 Vivante gpu driver version and also tried wayland v1.11.1 but Crash is still observed. if you look at the application code GitHub - cordlandwehr/imx6_testcase_wayland-client-crash: Testcase for https://community.nxp.com/thread/474517?sr=inbox&…

thread 2 looks like it's about to destroy the EGLSurface for this wl_surface which already got destroyed. So the question is why is this wl_surface destroyed before its EGLSurface is destroyed.

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

sanjeevsharma — Fri, 20 Jul 2018 06:36:53 GMT

I discussed this issue with QT Community on #IRC channel(qt-labs) and described them this issue in detail. As per Thiago Macieira(Software Architect at Intel; Open Source advocate), The bug is not in Qt Code. GPU Vivante driver seems has some issues. This is caused by a race condition between the main thread and thread 2 (the thread that crashed). Thread 2 looks like it's about to destroy the EGLSurface for this wl_surface which already got destroyed)

He pointed out that "Main thread is doing a usleep() indicates that this software package has issue"

In backtrace he pointed out below section of code snippet which creating issue, mainly gcoOS_Delay which is highlighted.

#1 0x75dc7420 in usleep (useconds=useconds@entry=10000) at /usr/src/debug/glibc/2.24-r0/git/sysdeps/posix/usleep.c:32
ts =

{tv_sec = 0, tv_nsec = 10000000}

#2 0x755d7574 in gcoOS_Delay (Os=Os@entry=0x0, Delay=Delay@entry=10) at gc_hal_user_os.c:4033
_user_ = 1 '\001'
_user_ptr_ = <synthetic pointer>
#3 0x75503cf8 in _DestroySurfaceObjects (Thread=Thread@entry=0x3e2f4, Surface=Surface@entry=0x6636bc7c) at gc_egl_surface.c:1123
i = <optimized out>
status = gcvSTATUS_OK
#4 0x75503f80 in _DestroySurface (Thread=Thread@entry=0x3e2f4, Surface=Surface@entry=0x6636bc7c) at gc_egl_surface.c:2316

In my opinion,We can't insert delays/sleeps in our code, if we need to support multi-threading properly. since the crashing thread is running software from the same package that isn't doing multi-threading correctly, So problem looks like improper multi-threading.

Any input would be highly appreciated.

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

sanjeevsharma — Tue, 31 Jul 2018 09:22:10 GMT

Hello All,

is anyone from NXP who can look and help in this issue.

Regards

Sanjeev Sharma

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

PrabhuSundarara — Wed, 01 Aug 2018 14:06:10 GMT

Hi Sanjeev,

Could you please try with our latest 6.2.4P1 release with weston 4.0. There are lot improvements in the EGL related to buffere management.

may be you can send the test binary it will be quick to test.

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

sanjeevsharma — Mon, 06 Aug 2018 14:50:25 GMT

Thanks Prabhu,

please find attached test binaries.

#XDG_RUNTIME_DIR=/var/run compositor -platform eglfs #XDG_RUNTIME_DIR=/var/run client -platform wayland 

By iterative clicking with your finger at the red rectangle and the created yellow rectangle, you can reproduce the crash.

Note: compositor is  based on QtWayland.

Thanks in advance.

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

sanjeevsharma — Mon, 13 Aug 2018 07:05:28 GMT

Hi Prabhu,

Did you get time to execute test binary on 6.2.4P1 release ?

I would really appreciate your feedback here.

Regards
Sanjeev Sharma

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

chingling_wang — Thu, 16 Aug 2018 19:48:04 GMT

Hi, Sharma,

How can I unzip the attached binary? tar, bzip2 and window unzip are not working.

Thanks

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

chingling_wang — Fri, 17 Aug 2018 19:53:57 GMT

Hi, Sharma,

How can I unzip the attached binary? tar, bzip2 and window unzip are not working.

I can clone your gitHub, But, could you give me the instructions how to build it for imx6q?

Thanks

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

sanjeevsharma — Wed, 29 Aug 2018 07:43:30 GMT

Hi, Wang,

i am attaching unpack binaries for your reference. please try it and let me know.

Regards

Sanjeev Sharma

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

chingling_wang — Wed, 29 Aug 2018 20:46:39 GMT

Are your running commands:

XDG_RUNTIME_DIR=/var/run compositor -platform eglfs  &

XDG_RUNTIME_DIR=/var/run client -platform wayland ?

This is wayland backend, it cannot find or load eglfs plugin,  I got error.  And

I tried this way

./compositor -platform wayland &

./client -platform wayland

When I touched red rect, a yellow rect appears, no crash. The gpu I used is 6.2.4.p2, I used the latest sum xwayland sd card image.

Re: Bug in Vivante i.MX6 Wayland drivers when destroying windows

sanjeevsharma — Thu, 30 Aug 2018 05:08:06 GMT

Thanks Wang, you have to run client with wayland plugin only.

## Test Setup
- run compositor:

XDG_RUNTIME_DIR=/var/run compositor -platform eglfs &

- run client:

XDG_RUNTIME_DIR=/var/run client -platform wayland

Note: Compositor is based on QtWayland Qt 5.9.

## Trigger the Crash
- touch on the red window creates another green window
- touch an a green window destroys the latest green window

For performing the crash, just click on the red window, then the green (if it did not crash yet, repeat this).
After a couple of attempts, I get the crash.

We have tried 6.2.4P1 at our end. Compositor runs fine but client stuck after clicking 1 or 2 iteration, we don't see any crash but client is not taking more clicks(input).