Segmentation fault using queue in pipeline gstreamer

[fabiendelafonta]

Hi,

I have a sabrelite board with last bsp from freescale and i develop a camera security software. I use Gstreamer for my app, and i need add text to video. I find this thread :How to add text watermark on video stream

But, I have a segmentation fault after few minute when i use plugin "queue" in pipeline. If i try this pipeline without "queue", its work but i have a jerky video.

"gst-launch -e mfw_v4lsrc capture-mode=5 fps-n=30 ! vpuenc codec=6 bitrate=20000000 ! matroskamux ! filesink location=/root/test1.mkv" : Ok but jerky video

"gst-launch -e mfw_v4lsrc capture-mode=5 fps-n=30 ! queue max-size-buffers=3 ! clockoverlay ! vpuenc codec=6 bitrate=2000000 ! queue ! matroskamux ! filesink location=/root/test2.mkv" : Segmentation fault after few minute.

Thanks in advance,

Fabien.

ps : I boot my board in nfsroot.

[fabiendelafonta]

Hi Edison.

Thank for your patch, it's work fine. I run my app with queue for 1 day and I have not had a crash.

Thanks, Fabien.

[henrybradlow]

I had the exact same problem, but it seems to have been solved by switching to queue2 instead of queue.

Hope this helps!

-Henry

[edison_fernande · Accepted solution]

Hi Henry,

Thanks for your help.

Just in case, I just found a solution for the queue. Just apply this patch to the src element.

The problem was being caused by the free_pool pointer in this element. This patch make it thread safe so the fault disappears.

Regards,

Edison

[fabiendelafonta]

Hi,

does anyone have a solution for the queue problem ?

thanks.

Fabien.

[edison_fernande]

Hi Fabien,

Did you find a solution for the seg fault?

Edison

[LeonardoSandova]

Fabien, could you try reducing the number of buffers being used by queue, as suggested in this thread: https://community.freescale.com/message/340827#340827 ?

[fabiendelafonta]

Hi,

I tried to compile and run my app with no buffers in queue (" queue max-size-buffers=0 max-size-bytes=0 max-size-time=0 ") and I have the same error as Edison.

root@freescale ~$ gdb ./RTSPsrvQ0

GNU gdb 6.6

Copyright (C) 2006 Free Software Foundation, Inc.

GDB is free software, covered by the GNU General Public License, and you are

welcome to change it and/or distribute copies of it under certain conditions.

Type "show copying" to see the conditions.

There is absolutely no warranty for GDB.  Type "show warranty" for details.

This GDB was configured as "arm-none-linux-gnueabi"...

Using host libthread_db library "/lib/libthread_db.so.1".

(gdb) run

Starting program: /root/RTSPsrvQ0

[Thread debugging using libthread_db enabled]

[New process 3223]

/***********************************************************/

/	VydecRtspServer	/
/	Build on Mon Jul 29 16:07:18 2013	/

/***********************************************************/

[New Thread 1074814976 (LWP 3223)]

MFW_GST_V4LSRC_PLUGIN 3.0.7 build on Jul 22 2013 08:50:18.

[New Thread 1094190192 (LWP 3226)]

[New Thread 1103185008 (LWP 3227)]

[New Thread 1111807088 (LWP 3228)]

[New Thread 1121973360 (LWP 3229)]

[INFO]  Product Info: i.MX6Q/D/S

vpuenc versions :smileyhappy:

	plugin: 3.0.7
	wrapper: 1.0.35(VPUWRAPPER_ARM_LINUX Build on Jul 11 2013 10:08:13)
	vpulib: 5.4.12
	firmware: 2.1.9.36350

[New Thread 1133061232 (LWP 3230)]

[New Thread 1142436976 (LWP 3231)]

ov5642_write_snapshot_para(new_frame_rate, new_mode) = 0

[New Thread 1207465072 (LWP 3232)]

[New Thread 1215956080 (LWP 3233)]

[New Thread 1225000048 (LWP 3234)]

[New Thread 1233945712 (LWP 3235)]

ERROR: v4l2 capture: mxc_v4l_dqueue() interrupt received

ERROR: v4l2 capture: mxc_v4l_dqueue() interrupt received

[INFO]  chromaInterleave 0, mapType 0, linear2TiledEnable 0

ERROR: v4l2 capture: mxc_v4l_dqueue() interrupt received

VPU interrupt received.

[New Thread 1262273648 (LWP 3236)]

ERROR: v4l2 capture: mxc_v4l_dqueue() interrupt received

VPU interrupt received.

[Thread 1262273648 (LWP 3236) exited]

VPU interrupt received.

Program received signal SIGSEGV, Segmentation fault.

[Switching to Thread 1207465072 (LWP 3232)]

0x403819b0 in IA__g_list_length (list=0x9) at glist.c:937

937	glist.c: No such file or directory.
	in glist.c

(gdb) bt

#0  0x403819b0 in IA__g_list_length (list=0x9) at glist.c:937

#1  0x406094d8 in mfw_gst_v4lsrc_create (src=<value optimized out>,

buf=0x47f86cf4) at mfw_gst_v4lsrc.c:292

#2  0x400a9674 in gst_push_src_create (bsrc=<value optimized out>,

	offset=<value optimized out>, length=4096, ret=0x47f86cf4)
	at gstpushsrc.c:117

#3  0x4009434c in gst_base_src_get_range (src=0xb2030,

offset=4628217828586029056, length=67, buf=0x47f86cf4) at gstbasesrc.c:1785

#4  0x40095f68 in gst_base_src_loop (pad=0xb4000) at gstbasesrc.c:1785

#5  0x401d5750 in gst_task_func (task=0xe92d8) at gsttask.c:207

#6  0x401d6c50 in default_func (tdata=<value optimized out>,

pool=<value optimized out>) at gsttaskpool.c:70

#7  0x403ad85c in g_thread_pool_thread_proxy (data=<value optimized out>)

at gthreadpool.c:726

#8  0x403ab57c in g_thread_create_proxy (data=0x3c5f0) at gthread.c:180

#9  0x40437b10 in start_thread () from /lib/libpthread.so.0

#10 0x40513958 in clone () from /lib/libc.so.6

(gdb)

Fabien.

[LeonardoSandova]

Fabien,

If you set max-size-buffers=1, does the seg-fault occur?

Leo

[fabiendelafonta]

Hi,

I tried to set max-size-buffers to 1 and I have the same error after a few hour.

Fabien.

[jack_mao]

Hi,

      have you tried gdb to get any clue?

       1. Enter command:    ulimit -c unlimited

      2. restart your application and wait for "segment fault", there should be core file

      3.  gdb <excute app>  core*

   e.g   gdb ./app core.12345

       Please also check your code,  if it use illegal pointer, access right issue(such as write to read only memory

Jack

[fabiendelafonta]

Hi,

I launch my server rtsp in gdb and i have :

root@freescale ~$ gdb ./rtspServer core
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "arm-none-linux-gnueabi"...
Using host libthread_db library "/lib/libthread_db.so.1".
/root/core: No such file or directory.
(gdb) run
Starting program: /root/rtspServer
[Thread debugging using libthread_db enabled]
[New process 3120]
/***********************************************************/
/                   VydecRtspServer                       /
/            Build on Mon Jul 15 11:11:54 2013              /
/***********************************************************/
[New Thread 1075015680 (LWP 3120)]
MFW_GST_V4LSRC_PLUGIN 3.0.7 build on Jul 11 2013 10:09:00.
[New Thread 1094386800 (LWP 3123)]
[New Thread 1103168624 (LWP 3124)]
[New Thread 1112282224 (LWP 3125)]
[New Thread 1121973360 (LWP 3126)]
[INFO]  Product Info: i.MX6Q/D/S
vpuenc versions :)
        plugin: 3.0.7
        wrapper: 1.0.35(VPUWRAPPER_ARM_LINUX Build on Jul 11 2013 10:08:13)
        vpulib: 5.4.12
        firmware: 2.1.9.36350
[New Thread 1132811376 (LWP 3127)]
[New Thread 1141699696 (LWP 3128)]
ov5642_write_snapshot_para(new_frame_rate, new_mode) = 0
[New Thread 1206719600 (LWP 3129)]
[New Thread 1215325296 (LWP 3130)]
[New Thread 1224541296 (LWP 3131)]
[New Thread 1233593456 (LWP 3132)]
[INFO]  chromaInterleave 0, mapType 0, linear2TiledEnable 0
[New Thread 1261630576 (LWP 3133)]
VPU interrupt received.
ERROR: v4l2 capture: mxc_v4l_dqueue() interrupt received
[Thread 1261630576 (LWP 3133) exited]
VPU interrupt received.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1206719600 (LWP 3129)]
0x403d69b0 in IA__g_list_length (list=0x8) at glist.c:937
937     glist.c: No such file or directory.
        in glist.c
(gdb) bt
#0  0x403d69b0 in IA__g_list_length (list=0x8) at glist.c:937
#1  0x406424d8 in mfw_gst_v4lsrc_create (src=<value optimized out>,
    buf=0x47ed0cf4) at mfw_gst_v4lsrc.c:295
#2  0x40195674 in gst_push_src_create (bsrc=<value optimized out>,
    offset=<value optimized out>, length=4096, ret=0x47ed0cf4)
    at gstpushsrc.c:117
#3  0x4018034c in gst_base_src_get_range (src=0xb2030,
    offset=4629713164399804416, length=17, buf=0x47ed0cf4) at gstbasesrc.c:1785
#4  0x40181f68 in gst_base_src_loop (pad=0xb4000) at gstbasesrc.c:1785
#5  0x40236750 in gst_task_func (task=0xe92d8) at gsttask.c:207
#6  0x40237c50 in default_func (tdata=<value optimized out>,
    pool=<value optimized out>) at gsttaskpool.c:70
#7  0x4040285c in g_thread_pool_thread_proxy (data=<value optimized out>)
    at gthreadpool.c:726
#8  0x4040057c in g_thread_create_proxy (data=0x10c340) at gthread.c:180
#9  0x4047bb10 in start_thread () from /lib/libpthread.so.0
#10 0x40557958 in clone () from /lib/libc.so.6

Fabien

[jack_mao]

Hi,

     In your trace, it failed in

0x403d69b0 in IA__g_list_length (list=0x8) at glist.c:937

937     glist.c: No such file or directory.

    I don't know if there is something missing.  we have not met this issue before, so please check your code.

[edison_fernande]

Hi,

I've been using gdb to debug the same issue. In my case i'm  sure glist.c is not missing (or any other file). When you follow the back trace you will see that at some moment the list is pointing to a wrong address (0x08 in Fabien's case). If i follow the back trace the common factor is the free_pool list in the mfw_gst_v4lsrc.c file that at some point is corrupted. For example, in the mfw_gst_v4lsrc_buffer_new function, when the list length is checked:

  if (g_list_length (v4l_src->free_pool) == 0) {

    GST_WARNING ("no buffer available in pool");

    return NULL;

  }

for some reason v4l_src->free_pool->next sometimes is pointing to a wrong address and that's why the pipeline crashes in glist.c. I've been trying to find out why that pointer is being corrupted with no success.

Any idea?, Am i wrong?

Thanks, Edison

[LeonardoSandova]

christapp, could you please try the debugging steps mentioned by jackmao. Junping, there is a similar issue (https://community.freescale.com/message/340508#340508) reported by Chris.

[jack_mao]

The command can run for some time, so the pipeline seems ok, does the recorded file can be played? if yes, please check the related memory status.

we doesn't meet such issue before, you could see the reference command from the user manual.

gst-launch mfw_v4lsrc fps-n=15 capture-mode=X ! queue ! $video_encoder_plugin codec=0 ! matroskamux ! filesink location=output.mkv sync=false

[fabiendelafonta]

Hi,

Yes I can play the record file. i tried some pipeline from the user manual and when I use the queue plugin I have a segfault after few hour.

My application encode a video in h264, make a video files of 2 minutes and send video to Rtsp Server. If I don't use the queue element, I have a jerky video.

[edison_fernande]

Hi,

I'm using a Sabre-SDP and with the same results. When i try to record from any of the on board cameras by using this pipeline:

gst-launch mfw_v4lsrc ! queue ! vpuenc codec=6 !  h264parse  ! mp4mux ! filesink location=test.mp4

the process stops with a segmentation fault (rarelly with a "no buffer available in pool" warning). If i remove the queue from the pipeline:

gst-launch mfw_v4lsrc ! vpuenc codec=6 !  h264parse  ! mp4mux ! filesink location=test.mp4

there is no segmentation fault.

The issue can be reproduced even using a fakesink instead of a filesink.

Debugging a little bit, i noticed that when the pipeline crashes is because the variable v4l_src->free_pool or v4l_src->free_pool->next (mfw_gst_v4lsrc.c) points to a not valid address (usually near 0 but not null) and at some point in the code someone tries to access that direction.

Any sugestion on how to solve it?...

Thanks

[LeonardoSandova]

Interesting and weird :smileysad:

Does the queue element gives you some performance on the encoding (of course before crashing)? jackmao have you seen this before?

Leo

[edison_fernande]

Yes, it's weird.

Actually the pipeline i'm trying to run is one with visualization and recording at the same time:

gst-launch mfw_v4lsrc device=/dev/video0 ! tee name=td ! queue ! vpuenc codec=6 ! h264parse ! mp4mux ! filesink location=/opt/scripts/media/video_record.mp4 td. ! queue ! mfw_v4lsink device=/dev/video17 axis-left=0 axis_top=0 disp-width=1024 disp-height=768

and with that one i do need the queues but the issue can be reproduced with the pipeline in my previous message. As i mentioned before you can use even a fakesink with same results.

Something interesting or weird is that the time the pipeline is running is random (from seconds to hours) and every time it crashes the free_pool->next pointer is pointing to what it seems to be the buffer index and not the buffer address. I'm using the default queue-size wich is 6 buffers and when the pipeline crashes is because some address (not greater than 6) is trying to be accessed (free_pool->next).

Thanks

[LeonardoSandova]

Try removing the bitrate property and put your root filesystem on a SD Card. The Jerky effect may be introduced to the NFS mounted system.

Leo