gstreamer streaming video + audio

[zhencongtee]

Hi,

I am trying to stream 2 videos, where 1 video is with sound, and the 2nd video is without sound.

The 2 video will be resized and be placed at different location at the display.

In order to stream 2 videos, below is the gst-launch command that i use:

     gst-launch filesrc location=a.mp4 typefind=true ! aiurdemux ! vpudec ! mfw_isink axis-left=0 axis-top=0 disp-width=640 disp-height=480 & Gst-launch filesrc location=b.mp4 typefind=true ! aiurdemux ! vpudec ! mfw_isink axis-left=640 axis-top=0 disp-width=640 disp-height=480

However, the pipeline above only stream the video and no audio available. How can i include the streaming of audio for the a.mp4? I tried:

     gst-launch filesrc location=a.mp4 typefind=true ! aiurdemux ! vpudec ! mfw_isink axis-left=0 axis-top=0 disp-width=640 disp-height=480 & gst-launch filesrc location=a.mp4 typefind=true ! aiurdemux ! beepdec ! alsasink & gst-launch filesrc location=b.mp4 typefind=true ! aiurdemux ! vpudec ! mfw_isink axis-left=640 axis-top=0 disp-width=640 disp-height=480

but the video and audio of the a.mp4 is not in sync.

Any idea what i am missing here?

I cannot use gplay or playbin2, as i need to stream 2 video at the same time, and need to resize and reposition the video.

Thank you.

Regards,

Zhen Cong Tee

[marcomadrigal]

Hi,

They are not going to be on sync since you are playing separate pipelines for audio and video. You should run something like this (not tested):

gst-launch filesrc location=a.mp4 typefind=true ! aiurdemux name=demux ! vpudec ! mfw_isink axis-left=0 axis-top=0 disp-width=640 disp-height=480 demux. ! queue ! beepdec ! alsasink & gst-launch filesrc location=b.mp4 typefind=true ! aiurdemux ! vpudec ! mfw_isink axis-left=640 axis-top=0 disp-width=640 disp-height=480

The main difference is that running two different pipelines for audio and video causes each one to run with its own clock, in the other hand getting both streams from the demuxer on the same pipeline ensures the pipeline is going to synchronize the streams with the same clock source.

Regards,

-Marco

[zhencongtee]

Hi Marco,

Sorry but do you have any update on my questions? I am stucked for this issue...

Thanks.

Regards,

Zhen Cong Tee

[zhencongtee · Accepted solution]

Hi all,

I manage to find the solution for this issue already.

In order to run the video and audio simultaneously and smoothly, command as below:

gst-launch filesrc location=<filename>.mp4 typefind=true ! aiurdemux name=demux demux. ! queue max-size-buffers=0 max-size-time=0 ! mfw_isink axis-left=0 axis-top=0 disp-width=720 disp-height=480 demux. ! queue max-size-buffers=0 max-size-time=0 ! audioconvert ! ‘audio/x-raw-int,channels=2’ ! alsasink

max-size-buffer=0 and max-size-time=0 is essential for the video to play smoothly.

Thanks all for help.

Regards,

Zhen Cong Tee

[marcomadrigal]

Hi Zhen Cong Tee,

Sorry for the delay, I got distracted on a project and today I realized about your question. Glad to know it is working now.

Best Regards,

-Marco

[zhencongtee]

Hi Marco,

Thanks for the reply. Had been busy lately and today got the time to test out this.

However, the result is not as i expected.

I tested:

gst-launch filesrc location=a.mp4 typefind=true ! aiurdemux name=demux ! vpudec ! mfw_isink axis-left=0 axis-top=0 disp-width=640 disp-height=480 demux. ! queue ! beepdec ! alsasink

1 video with audio pipeline.

During the video + audio playback, the console prompts me lot's of warning messages as below:

     WARNING: from element /GstPipeline:pipeline0/MfwGstISink:mfwgstisink0: A lot of buffers are being dropped.

     Additional debug info:

     /home/pico/workspace/yocto/edm_yocto_dora/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer/0.10.36-r2/gstreamer-0.10.36/libs/gst/base/gstbasesink.c(2875): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/MfwGstISink:mfwgstisink0:

     There may be a timestamping problem, or this computer is too slow.

I tried to use mfw_v4lsink as well, the similiar warning messages appear too.

     WARNING: from element /GstPipeline:pipeline0/MFW_GST_V4LSINK_INFO_T:mfw_gst_v4lsink_info_t0: A lot of buffers are being dropped.

     Additional debug info:

     /home/VERIFONE/zhencongt1/fsl-release-community/build_dora/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer/0.10.36-r2/gstreamer-0.10.36/libs/gst    /base/gstbasesink.c(2875): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/MFW_GST_V4LSINK_INFO_T:mfw_gst_v4lsink_info_t0:

     There may be a timestamping problem, or this computer is too slow.

In both case, the video is jerky, but audio playing is smooth. At the end of the video, the gstreamer prompt that the fps is only around 5.5fps. I am using a SOLO board. Dun think playing a video + audio can makes such big different right? I tried to do a 'top' command, and i see that the CPU load is only about 15%.

I tried to run the video and audio seperately (2 different process), and this problem does not happen. Video is smooth, audio is smooth.

Any idea why is this happening? Thank you.

Regards,

Zhen Cong Tee