gstreamer hang in vpu_WaitForInt

johannobermayr — Wed, 30 Nov 2016 14:49:26 GMT

Hello,

we have a own embedded system with i.MX6 Dualcore with Kernel 3.10.53 SMP
we have a testapplication with libgstreamer-1.0.so.0.203.0
we have a own thread with GstLoop

static int GstLoop(void *arg)
{
OSmutexLock(&gstLock);
gstInstance.loop = g_main_loop_new (NULL, TRUE);
OSmutexUnlock(&gstLock);

// run loop
g_main_loop_run (gstInstance.loop);

gst_element_set_state (gstInstance.pipeline, GST_STATE_NULL/*READY*/);

gst_object_unref (gstInstance.sink);
gst_object_unref (gstInstance.transform);
gst_object_unref (gstInstance.sinkBin);

/* could be that stopandcleanup() was called from GST context, than object_unref would hang. */
g_main_loop_unref(gstInstance.loop);
g_source_remove (gstInstance.bus_watch_id);
gst_object_unref (gstInstance.pipeline);

OSmutexLock(&stateLock);
memset(&gstInstance, 0, sizeof(gstInstance));
gstInstance.state = E_NOT_LOADED;
OSmutexUnlock(&stateLock);

return 0;
}

some seconds after starting the video we send a
g_main_loop_quit(gstInstance.loop);
to stop the video.

But we see, that the GstLoop will sometimes hang in the function pthread_mutex_lock called from g_mutex_lock with follow stack.
It can be work 2 days without trouble, or we have see, that this problem are 5 times per day.

Nov 29 12:33:39 test-arm user.info LRT: Thread is: gst_Loop:525:21916, pid:21916, objThis:0
Nov 29 12:33:39 test-arm user.info LRT:   Stack: 0xB5E41000-B5E4A000, size:36864, Min:15560
Nov 29 12:33:39 test-arm user.info LRT:   [0x000f06e4] LRT.exe(OSdumpstackContext+0xC4) SP[0xB5E46FB8], BP[0xB5E472C4]
Nov 29 12:33:39 test-arm user.info LRT:   [0x000f0c28] LRT.exe(OSbacktrace+0x38) SP[0xB5E46FB8], BP[0xB5E472D4]
Nov 29 12:33:39 test-arm user.info LRT:   [0x0013c8f4] LRT.exe(pthread_mutex_lock+0xB0) SP[0xB5E46FB8], BP[0xB5E4732C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb636362c] libglib-2.0.so.0.3800.2(g_mutex_lock+0x10) SP[0xB5E46FB8], BP[0xB5E47334]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb3b8356c] libgstimxvpu.so SP[0xB5E46FB8], BP[0xB5E4735C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc3ac] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x2C) SP[0xB5E46FB8], BP[0xB5E47394]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cca6c] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x6EC) SP[0xB5E46FB8], BP[0xB5E473FC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc0b4] libgstreamer-1.0.so.0.203.0(gst_element_set_state+0x94) SP[0xB5E46FB8], BP[0xB5E47414]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61ad8e8] libgstreamer-1.0.so.0.203.0(gst_bin_recalculate_latency+0xC58) SP[0xB5E46FB8], BP[0xB5E474D4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb5ecbd88] libgstplayback.so SP[0xB5E46FB8], BP[0xB5E47534]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc3ac] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x2C) SP[0xB5E46FB8], BP[0xB5E4756C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cca6c] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x6EC) SP[0xB5E46FB8], BP[0xB5E475D4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc0b4] libgstreamer-1.0.so.0.203.0(gst_element_set_state+0x94) SP[0xB5E46FB8], BP[0xB5E475EC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61ad8e8] libgstreamer-1.0.so.0.203.0(gst_bin_recalculate_latency+0xC58) SP[0xB5E46FB8], BP[0xB5E476AC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb5ed9ca4] libgstplayback.so SP[0xB5E46FB8], BP[0xB5E47734]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc3ac] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x2C) SP[0xB5E46FB8], BP[0xB5E4776C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cca6c] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x6EC) SP[0xB5E46FB8], BP[0xB5E477D4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc0b4] libgstreamer-1.0.so.0.203.0(gst_element_set_state+0x94) SP[0xB5E46FB8], BP[0xB5E477EC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61ad8e8] libgstreamer-1.0.so.0.203.0(gst_bin_recalculate_latency+0xC58) SP[0xB5E46FB8], BP[0xB5E478AC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61f0384] libgstreamer-1.0.so.0.203.0(gst_param_spec_fraction+0xE74) SP[0xB5E46FB8], BP[0xB5E47944]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb5ee24d4] libgstplayback.so SP[0xB5E46FB8], BP[0xB5E4797C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc3ac] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x2C) SP[0xB5E46FB8], BP[0xB5E479B4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc51c] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x19C) SP[0xB5E46FB8], BP[0xB5E479EC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cca6c] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x6EC) SP[0xB5E46FB8], BP[0xB5E47A54]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc0b4] libgstreamer-1.0.so.0.203.0(gst_element_set_state+0x94) SP[0xB5E46FB8], BP[0xB5E47A6C]
Nov 29 12:33:39 test-arm user.info LRT:   [0x000faddc] LRT.exe(GstLoop+0x70) SP[0xB5E46FB8], BP[0xB5E47A7C]
Nov 29 12:33:39 test-arm user.info LRT:   [0x0012499c] LRT.exe(trampoline+0x4C8) SP[0xB5E46FB8], BP[0xB5E47B84]
Nov 29 12:33:39 test-arm user.info LRT:   [0x00122340] LRT.exe(pthread_startthread_hook+0x3C) SP[0xB5E46FB8], BP[0xB5E47B94]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb6f360e8] libpthread-2.19.so(__pthread_get_minstack+0x1690) SP[0xB5E46FB8], BP[0xB5E47CDC]

we have overwritten the pthread_mutex_lock function, and have replace the pthread_mutex_lock with pthread_mutex_timedlock and 150seconds timeout.
after this we get the mutex data owner to see, witch task will have the mutex.
and we get the callstack for this task
Nov 29 12:33:39 test-arm user.info LRT: Callstack for thread:'???':21919
Nov 29 12:33:39 test-arm user.info LRT:   [0xb69a138c] libc-2.19.so(ioctl+0xC) SP[0xB43F2044], BP[0xB43F2054]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb5f7c77c] libvpu.so.4(vpu_WaitForInt+0x34) SP[0xB43F2044], BP[0xB43F207C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb3b6e178] libimxvpuapi.so.0(imx_vpu_dec_decode+0x368) SP[0xB43F2044], BP[0xB43F2154]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb3b8406c] libgstimxvpu.so SP[0xB43F2044], BP[0xB43F2254]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb6150488] libgstvideo-1.0.so.0.203.0(gst_is_video_overlay_prepare_window_handle_message+0x1438) SP[0xB43F2044], BP[0xB43F22DC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61506fc] libgstvideo-1.0.so.0.203.0(gst_is_video_overlay_prepare_window_handle_message+0x16AC) SP[0xB43F2044], BP[0xB43F2364]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb6152c04] libgstvideo-1.0.so.0.203.0(gst_is_video_overlay_prepare_window_handle_message+0x3BB4) SP[0xB43F2044], BP[0xB43F240C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61e39e8] libgstreamer-1.0.so.0.203.0(gst_flow_get_name+0x1C0C) SP[0xB43F2044], BP[0xB43F2494]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb610fbe4] libgstbase-1.0.so.0.203.0(gst_base_transform_set_passthrough+0x24C0) SP[0xB43F2044], BP[0xB43F24DC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61e39e8] libgstreamer-1.0.so.0.203.0(gst_flow_get_name+0x1C0C) SP[0xB43F2044], BP[0xB43F2564]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb60ed3c4] libgstbase-1.0.so.0.203.0(gst_base_parse_push_frame+0x83C) SP[0xB43F2044], BP[0xB43F2624]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb60f15dc] libgstbase-1.0.so.0.203.0(gst_base_parse_finish_frame+0x6B4) SP[0xB43F2044], BP[0xB43F26C4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb3bdab34] libgstvideoparsersbad.so SP[0xB43F2044], BP[0xB43F27C4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb60ea864] libgstbase-1.0.so.0.203.0(gst_base_parse_frame_new+0x548) SP[0xB43F2044], BP[0xB43F2874]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb60ee5e8] libgstbase-1.0.so.0.203.0(gst_base_parse_push_frame+0x1A60) SP[0xB43F2044], BP[0xB43F294C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61e39e8] libgstreamer-1.0.so.0.203.0(gst_flow_get_name+0x1C0C) SP[0xB43F2044], BP[0xB43F29D4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb5617438] libgstcoreelements.so SP[0xB43F2044], BP[0xB43F2AAC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb621a590] libgstreamer-1.0.so.0.203.0(gst_tag_setter_get_tag_merge_mode+0x3FC) SP[0xB43F2044], BP[0xB43F2B1C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb621b73c] libgstreamer-1.0.so.0.203.0(gst_task_join+0x368) SP[0xB43F2044], BP[0xB43F2B2C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb6345efc] libglib-2.0.so.0.3800.2(g_thread_pool_new+0x4A8) SP[0xB43F2044], BP[0xB43F2B6C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb63452e8] libglib-2.0.so.0.3800.2(g_test_get_filename+0x174) SP[0xB43F2044], BP[0xB43F2B84]
Nov 29 12:33:39 test-arm user.info LRT:   [0x00122340] LRT.exe(pthread_startthread_hook+0x3C) SP[0xB43F2044], BP[0xB43F2B94]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb6f360e8] libpthread-2.19.so(__pthread_get_minstack+0x1690) SP[0xB43F2044], BP[0xB43F2CDC]

so we see that this task will hang in the function vpu_WaitForInt.
But after getting this information (callstack) the task will continue.

Thanks for help
Johann

Re: gstreamer hang in vpu_WaitForInt

igorpadykov — Thu, 01 Dec 2016 09:57:43 GMT

Hi Johann

seems community gstreamer-imx plugins used in the case

(like libimxvpuapi, libgstimxvpu) and this could be posted on community mail list

meta-freescale Info Page

building for android · Issue #21 · Freescale/gstreamer-imx · GitHub

difference between community gstreamer-imx plugins and
nxp gstreamer plugins (fsl-gst-plugin/ imx-gst1.0-plugin packages) described on
trac.gateworks.com/wiki/Yocto/gstreamer
NXP plugins
https://github.com/Freescale/meta-freescale/tree/master/recipes-multimedia

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

i.MX Processorsのトピックgstreamer hang in vpu_WaitForInt

gstreamer hang in vpu_WaitForInt

Re: gstreamer hang in vpu_WaitForInt