AnsweredAssumed Answered

gstreamer hang in vpu_WaitForInt

Question asked by Johann Obermayr on Nov 30, 2016
Latest reply on Dec 1, 2016 by igorpadykov

Hello,

we have a own embedded system with i.MX6 Dualcore with Kernel 3.10.53 SMP
we have a testapplication with libgstreamer-1.0.so.0.203.0
we have a own thread with GstLoop

static int GstLoop(void *arg)
{
 OSmutexLock(&gstLock);
 gstInstance.loop = g_main_loop_new (NULL, TRUE);
 OSmutexUnlock(&gstLock);

 // run loop
 g_main_loop_run (gstInstance.loop);

 gst_element_set_state (gstInstance.pipeline, GST_STATE_NULL/*READY*/);

 gst_object_unref (gstInstance.sink);
 gst_object_unref (gstInstance.transform);
 gst_object_unref (gstInstance.sinkBin);

 /* could be that stopandcleanup() was called from GST context, than object_unref would hang. */
 g_main_loop_unref(gstInstance.loop);
 g_source_remove (gstInstance.bus_watch_id);
 gst_object_unref (gstInstance.pipeline);

 OSmutexLock(&stateLock);
 memset(&gstInstance, 0, sizeof(gstInstance));
 gstInstance.state = E_NOT_LOADED;
 OSmutexUnlock(&stateLock);

 return 0;
}

some seconds after starting the video we send a
 g_main_loop_quit(gstInstance.loop);
 to stop the video.
 
But we see, that the GstLoop will sometimes hang in the function pthread_mutex_lock called from g_mutex_lock with follow stack.
It can be work 2 days without trouble, or we have see, that this problem are 5 times per day.

Nov 29 12:33:39 test-arm user.info LRT: Thread is: gst_Loop:525:21916, pid:21916, objThis:0
Nov 29 12:33:39 test-arm user.info LRT:   Stack: 0xB5E41000-B5E4A000, size:36864, Min:15560
Nov 29 12:33:39 test-arm user.info LRT:   [0x000f06e4] LRT.exe(OSdumpstackContext+0xC4)  SP[0xB5E46FB8], BP[0xB5E472C4]
Nov 29 12:33:39 test-arm user.info LRT:   [0x000f0c28] LRT.exe(OSbacktrace+0x38)  SP[0xB5E46FB8], BP[0xB5E472D4]
Nov 29 12:33:39 test-arm user.info LRT:   [0x0013c8f4] LRT.exe(pthread_mutex_lock+0xB0)  SP[0xB5E46FB8], BP[0xB5E4732C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb636362c] libglib-2.0.so.0.3800.2(g_mutex_lock+0x10)  SP[0xB5E46FB8], BP[0xB5E47334]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb3b8356c] libgstimxvpu.so  SP[0xB5E46FB8], BP[0xB5E4735C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc3ac] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x2C)  SP[0xB5E46FB8], BP[0xB5E47394]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cca6c] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x6EC)  SP[0xB5E46FB8], BP[0xB5E473FC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc0b4] libgstreamer-1.0.so.0.203.0(gst_element_set_state+0x94)  SP[0xB5E46FB8], BP[0xB5E47414]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61ad8e8] libgstreamer-1.0.so.0.203.0(gst_bin_recalculate_latency+0xC58)  SP[0xB5E46FB8], BP[0xB5E474D4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb5ecbd88] libgstplayback.so  SP[0xB5E46FB8], BP[0xB5E47534]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc3ac] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x2C)  SP[0xB5E46FB8], BP[0xB5E4756C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cca6c] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x6EC)  SP[0xB5E46FB8], BP[0xB5E475D4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc0b4] libgstreamer-1.0.so.0.203.0(gst_element_set_state+0x94)  SP[0xB5E46FB8], BP[0xB5E475EC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61ad8e8] libgstreamer-1.0.so.0.203.0(gst_bin_recalculate_latency+0xC58)  SP[0xB5E46FB8], BP[0xB5E476AC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb5ed9ca4] libgstplayback.so  SP[0xB5E46FB8], BP[0xB5E47734]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc3ac] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x2C)  SP[0xB5E46FB8], BP[0xB5E4776C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cca6c] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x6EC)  SP[0xB5E46FB8], BP[0xB5E477D4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc0b4] libgstreamer-1.0.so.0.203.0(gst_element_set_state+0x94)  SP[0xB5E46FB8], BP[0xB5E477EC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61ad8e8] libgstreamer-1.0.so.0.203.0(gst_bin_recalculate_latency+0xC58)  SP[0xB5E46FB8], BP[0xB5E478AC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61f0384] libgstreamer-1.0.so.0.203.0(gst_param_spec_fraction+0xE74)  SP[0xB5E46FB8], BP[0xB5E47944]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb5ee24d4] libgstplayback.so  SP[0xB5E46FB8], BP[0xB5E4797C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc3ac] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x2C)  SP[0xB5E46FB8], BP[0xB5E479B4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc51c] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x19C)  SP[0xB5E46FB8], BP[0xB5E479EC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cca6c] libgstreamer-1.0.so.0.203.0(gst_element_change_state+0x6EC)  SP[0xB5E46FB8], BP[0xB5E47A54]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61cc0b4] libgstreamer-1.0.so.0.203.0(gst_element_set_state+0x94)  SP[0xB5E46FB8], BP[0xB5E47A6C]
Nov 29 12:33:39 test-arm user.info LRT:   [0x000faddc] LRT.exe(GstLoop+0x70)  SP[0xB5E46FB8], BP[0xB5E47A7C]
Nov 29 12:33:39 test-arm user.info LRT:   [0x0012499c] LRT.exe(trampoline+0x4C8)  SP[0xB5E46FB8], BP[0xB5E47B84]
Nov 29 12:33:39 test-arm user.info LRT:   [0x00122340] LRT.exe(pthread_startthread_hook+0x3C)  SP[0xB5E46FB8], BP[0xB5E47B94]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb6f360e8] libpthread-2.19.so(__pthread_get_minstack+0x1690)  SP[0xB5E46FB8], BP[0xB5E47CDC]

we have overwritten the pthread_mutex_lock function, and have replace the pthread_mutex_lock with pthread_mutex_timedlock and 150seconds timeout.
after this we get the mutex data owner to see, witch task will have the mutex.
and we get the callstack for this task
Nov 29 12:33:39 test-arm user.info LRT: Callstack for thread:'???':21919
Nov 29 12:33:39 test-arm user.info LRT:   [0xb69a138c] libc-2.19.so(ioctl+0xC)  SP[0xB43F2044], BP[0xB43F2054]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb5f7c77c] libvpu.so.4(vpu_WaitForInt+0x34)  SP[0xB43F2044], BP[0xB43F207C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb3b6e178] libimxvpuapi.so.0(imx_vpu_dec_decode+0x368)  SP[0xB43F2044], BP[0xB43F2154]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb3b8406c] libgstimxvpu.so  SP[0xB43F2044], BP[0xB43F2254]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb6150488] libgstvideo-1.0.so.0.203.0(gst_is_video_overlay_prepare_window_handle_message+0x1438)  SP[0xB43F2044], BP[0xB43F22DC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61506fc] libgstvideo-1.0.so.0.203.0(gst_is_video_overlay_prepare_window_handle_message+0x16AC)  SP[0xB43F2044], BP[0xB43F2364]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb6152c04] libgstvideo-1.0.so.0.203.0(gst_is_video_overlay_prepare_window_handle_message+0x3BB4)  SP[0xB43F2044], BP[0xB43F240C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61e39e8] libgstreamer-1.0.so.0.203.0(gst_flow_get_name+0x1C0C)  SP[0xB43F2044], BP[0xB43F2494]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb610fbe4] libgstbase-1.0.so.0.203.0(gst_base_transform_set_passthrough+0x24C0)  SP[0xB43F2044], BP[0xB43F24DC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61e39e8] libgstreamer-1.0.so.0.203.0(gst_flow_get_name+0x1C0C)  SP[0xB43F2044], BP[0xB43F2564]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb60ed3c4] libgstbase-1.0.so.0.203.0(gst_base_parse_push_frame+0x83C)  SP[0xB43F2044], BP[0xB43F2624]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb60f15dc] libgstbase-1.0.so.0.203.0(gst_base_parse_finish_frame+0x6B4)  SP[0xB43F2044], BP[0xB43F26C4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb3bdab34] libgstvideoparsersbad.so  SP[0xB43F2044], BP[0xB43F27C4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb60ea864] libgstbase-1.0.so.0.203.0(gst_base_parse_frame_new+0x548)  SP[0xB43F2044], BP[0xB43F2874]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb60ee5e8] libgstbase-1.0.so.0.203.0(gst_base_parse_push_frame+0x1A60)  SP[0xB43F2044], BP[0xB43F294C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb61e39e8] libgstreamer-1.0.so.0.203.0(gst_flow_get_name+0x1C0C)  SP[0xB43F2044], BP[0xB43F29D4]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb5617438] libgstcoreelements.so  SP[0xB43F2044], BP[0xB43F2AAC]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb621a590] libgstreamer-1.0.so.0.203.0(gst_tag_setter_get_tag_merge_mode+0x3FC)  SP[0xB43F2044], BP[0xB43F2B1C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb621b73c] libgstreamer-1.0.so.0.203.0(gst_task_join+0x368)  SP[0xB43F2044], BP[0xB43F2B2C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb6345efc] libglib-2.0.so.0.3800.2(g_thread_pool_new+0x4A8)  SP[0xB43F2044], BP[0xB43F2B6C]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb63452e8] libglib-2.0.so.0.3800.2(g_test_get_filename+0x174)  SP[0xB43F2044], BP[0xB43F2B84]
Nov 29 12:33:39 test-arm user.info LRT:   [0x00122340] LRT.exe(pthread_startthread_hook+0x3C)  SP[0xB43F2044], BP[0xB43F2B94]
Nov 29 12:33:39 test-arm user.info LRT:   [0xb6f360e8] libpthread-2.19.so(__pthread_get_minstack+0x1690)  SP[0xB43F2044], BP[0xB43F2CDC]

so we see that this task will hang in the function vpu_WaitForInt.
But after getting this information (callstack) the task will continue.

Thanks for help
 Johann


 

Outcomes