Audio and video gstreamer

[JoshuaParrish]

Dear all,

I was hoping someone might point me out to a gstreamer routine that captures audio and video. 

Specifically audio from the mic input and video using the hw vpu. 

I have a software test setup 

gst-launch videotestsrc ! videorate ! video/x-raw-yuv,framerate=5/1 ! queue ! theoraenc ! queue ! mux. alsasrc device=hw:0,0 ! audio/x-raw-int,rate=48000,channels=2,depth=16 ! queue ! audioconvert ! queue ! vorbisenc ! queue ! mux. oggmux name=mux ! filesink location=stream.ogv

by the audio is consumed to slowly (due to the video processing). 

I know that the hw mfw_vpuencoder only ejects h264 for mp4, so I just need to know how to encode audio with it. 

There doesn't seem to be an mfw_aacencoder ?

Thanks in advance!

[JoshuaParrish]

For some reason I don't have qtmux available, I'll have to look into why. 

[daiane_angolini]

Are you trying to make an mp4 using avimux?

Did you try qtmux?

Joshua Parrish said:

So I actually just changed the filename to have an mp4 extension on the avimuxed stream and I now have a playable mp4 file. Doing this added some weird video effects but it's a start. 

[daiane_angolini]

I haven´t see that. My last VLC command line was for PC.

Joshua Parrish said:

Diane,

Is there anyway to get vlc to play nicely with the gstreamer/mfw plugins?

[JoshuaParrish]

Here is a verbose log of the ffmux_mp4 

gst-launch -v v4l2src num-buffers=50 ! videorate ! 'video/x-raw-yuv,width=640,height=480,framerate=(fraction)15/1' ! queue ! mfw_vpuencoder codec-type=2 ! queue ! mux. alsasrc num-buffers=240 ! audioconvert ! mfw_mp3encoder ! queue ! mux. ffmux_mp4 name=mux ! filesink location=test.mp4           BLN_MAD-MMCODECS_MP3E_ARM_02.02.00_ARM12  build on Apr 19 2011 14:18:53.MFW_GST_MP3_ENCODER_PLUGIN 1.9.7-1 build on Jan 28 2011 13:29:54.Setting pipeline to PAUSED ...[INFO]  Product Info: i.MX53VPU Version: firmware 13.4.24; libvpu: 5.1.4MFW_GST_VPU_ENCODER_PLUGIN 1.9.7-1 build on Jan 28 2011 13:24:19.libv4lconvert: warning more framesizes then I can handle!libv4lconvert: warning more framesizes then I can handle!/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: actual-buffer-time = 139319/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: actual-latency-time = 34829/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0.GstPad:src: caps = audio/x-raw-int, endianness=(int)1234, signed=(boolean)true, width=(int)32, depth=(int)32, rate=(int)44100, channels=(int)2/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)15/1Pipeline is live and does not need PREROLL ...Setting pipeline to PLAYING ...New clock: GstAudioSrcClock/GstPipeline:pipeline0/GstAudioConvert:audioconvert0.GstPad:src: caps = audio/x-raw-int, endianness=(int)1234, signed=(boolean)true, width=(int)16, depth=(int)16, rate=(int)44100, channels=(int)2/GstPipeline:pipeline0/GstAudioConvert:audioconvert0.GstPad:sink: caps = audio/x-raw-int, endianness=(int)1234, signed=(boolean)true, width=(int)32, depth=(int)32, rate=(int)44100, channels=(int)2/GstPipeline:pipeline0/MfwGstMp3EncInfo:mfwgstmp3encinfo0.GstPad:sink: caps = audio/x-raw-int, endianness=(int)1234, signed=(boolean)true, width=(int)16, depth=(int)16, rate=(int)44100, channels=(int)2/GstPipeline:pipeline0/MfwGstMp3EncInfo:mfwgstmp3encinfo0.GstPad:sink: caps = audio/x-raw-int, endianness=(int)1234, signed=(boolean)true, width=(int)16, depth=(int)16, rate=(int)44100, channels=(int)2/GstPipeline:pipeline0/MfwGstMp3EncInfo:mfwgstmp3encinfo0.GstPad:src: caps = audio/mpeg, mpegversion=(int)1, layer=(int)3, rate=(int)44100, channels=(int)2/GstPipeline:pipeline0/MfwGstMp3EncInfo:mfwgstmp3encinfo0.GstPad:src: caps = audio/mpeg, mpegversion=(int)1, layer=(int)3, rate=(int)44100, channels=(int)2/GstPipeline:pipeline0/GstQueue:queue2.GstPad:sink: caps = audio/mpeg, mpegversion=(int)1, layer=(int)3, rate=(int)44100, channels=(int)2/GstPipeline:pipeline0/GstQueue:queue2.GstPad:src: caps = audio/mpeg, mpegversion=(int)1, layer=(int)3, rate=(int)44100, channels=(int)2/GstPipeline:pipeline0/ffmux_mp4:mux.GstPad:audio_0: caps = audio/mpeg, mpegversion=(int)1, layer=(int)3, rate=(int)44100, channels=(int)2/GstPipeline:pipeline0/GstVideoRate:videorate0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)15/1/GstPipeline:pipeline0/GstVideoRate:videorate0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)15/1/GstPipeline:pipeline0/GstVideoRate:videorate0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)15/1/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)15/1/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)15/1/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)15/1/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)15/1/GstPipeline:pipeline0/MfwGstVPU_Enc:mfwgstvpu_enc0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)15/1/GstPipeline:pipeline0/MfwGstVPU_Enc:mfwgstvpu_enc0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)640, height=(int)480, framerate=(fraction)15/1[ERR]   VPU iram is less than search ram size[WARN]  VPU iram is less than needed, some parts don't use iram/GstPipeline:pipeline0/MfwGstVPU_Enc:mfwgstvpu_enc0.GstPad:src: caps = video/x-h264, height=(int)480, width=(int)640, framerate=(fraction)15/1ERROR: from element /GstPipeline:pipeline0/ffmux_mp4:mux: Internal GStreamer error: negotiation problem.  Please file a bug at http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer.Additional debug info:gstffmpegmux.c(494): gst_ffmpegmux_collected (): /GstPipeline:pipeline0/ffmux_mp4:mux:no caps set on stream 0 (video)Execution ended after 1268359429 ns.Setting pipeline to PAUSED .../GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = video/x-h264, height=(int)480, width=(int)640, framerate=(fraction)15/1>>VPU_ENC: Error -5 in Pushing the Output to Source PadSetting pipeline to READY .../GstPipeline:pipeline0/ffmux_mp4:mux.GstPad:audio_0: caps = NULL/GstPipeline:pipeline0/ffmux_mp4:mux.GstPad:src: caps = NULL/GstPipeline:pipeline0/GstQueue:queue2.GstPad:src: caps = NULL/GstPipeline:pipeline0/GstQueue:queue2.GstPad:sink: caps = NULL/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = NULL/GstPipeline:pipeline0/MfwGstMp3EncInfo:mfwgstmp3encinfo0.GstPad:src: caps = NULL/GstPipeline:pipeline0/MfwGstMp3EncInfo:mfwgstmp3encinfo0.GstPad:sink: caps = NULL/GstPipeline:pipeline0/MfwGstVPU_Enc:mfwgstvpu_enc0.GstPad:src: caps = NULL/GstPipeline:pipeline0/MfwGstVPU_Enc:mfwgstvpu_enc0.GstPad:sink: caps = NULL/GstPipeline:pipeline0/GstAudioConvert:audioconvert0.GstPad:src: caps = NULL/GstPipeline:pipeline0/GstAudioConvert:audioconvert0.GstPad:sink: caps = NULL/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = NULL/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = NULL/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0.GstPad:src: caps = NULL/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = NULL/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = NULL/GstPipeline:pipeline0/GstVideoRate:videorate0.GstPad:src: caps = NULL/GstPipeline:pipeline0/GstVideoRate:videorate0.GstPad:sink: caps = NULL/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = NULLSetting pipeline to NULL ...Freeing pipeline ...

Joshua Parrish said:

So I actually just changed the filename to have an mp4 extension on the avimuxed stream and I now have a playable mp4 file. Doing this added some weird video effects but it's a start. 

[JoshuaParrish]

So I actually just changed the filename to have an mp4 extension on the avimuxed stream and I now have a playable mp4 file. Doing this added some weird video effects but it's a start. 

[JoshuaParrish]

Hey guys,

I have a stream that is H264 + mp3 working; but I can't get it into an mp4 container. 

Any thoughts?

gst-launch v4l2src num-buffers=50 ! videorate ! 'video/x-raw-yuv,width=640,height=480,framerate=(fraction)15/1' ! queue ! mfw_vpuencoder codec-type=2 ! queue ! mux. alsasrc num-buffers=240 ! audioconvert ! mfw_mp3encoder ! queue ! mux. avimux name=mux ! filesink location=test.avi

vlc plays this file and shows me that it is a H264 mpeg-4 avc video stream and an mp3 stream so I know the stream is correct. 

When I try something like 

gst-launch v4l2src num-buffers=50 ! videorate ! 'video/x-raw-yuv,width=640,height=480,framerate=(fraction)15/1' ! queue ! mfw_vpuencoder codec-type=2 ! queue ! mux. alsasrc num-buffers=240 ! audioconvert ! mfw_mp3encoder ! queue ! mux. ffmux_mp4 name=mux ! filesink location=test.mp4

I get an internal data flow error. 

BLN_MAD-MMCODECS_MP3E_ARM_02.02.00_ARM12  build on Apr 19 2011 14:18:53.MFW_GST_MP3_ENCODER_PLUGIN 1.9.7-1 build on Jan 28 2011 13:29:54.Setting pipeline to PAUSED ...[INFO]  Product Info: i.MX53VPU Version: firmware 13.4.24; libvpu: 5.1.4MFW_GST_VPU_ENCODER_PLUGIN 1.9.7-1 build on Jan 28 2011 13:24:19.libv4lconvert: warning more framesizes then I can handle!libv4lconvert: warning more framesizes then I can handle!Pipeline is live and does not need PREROLL ...Setting pipeline to PLAYING ...New clock: GstAudioSrcClock[ERR]   VPU iram is less than search ram size[WARN]  VPU iram is less than needed, some parts don't use iramERROR: from element /GstPipeline:pipeline0/ffmux_mp4:mux: Internal GStreamer error: negotiation problem.  Please file a bug at http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer.Additional debug info:gstffmpegmux.c(494): gst_ffmpegmux_collected (): /GstPipeline:pipeline0/ffmux_mp4:mux:no caps set on stream 0 (video)Execution ended after 1233684684 ns.Setting pipeline to PAUSED ...>>VPU_ENC: Error -5 in Pushing the Output to Source PadSetting pipeline to READY ...Setting pipeline to NULL ...Freeing pipeline ...

Just as a double check this is the dump of the working avi. 

BLN_MAD-MMCODECS_MP3E_ARM_02.02.00_ARM12  build on Apr 19 2011 14:18:53.MFW_GST_MP3_ENCODER_PLUGIN 1.9.7-1 build on Jan 28 2011 13:29:54.Setting pipeline to PAUSED ...[INFO]  Product Info: i.MX53VPU Version: firmware 13.4.24; libvpu: 5.1.4MFW_GST_VPU_ENCODER_PLUGIN 1.9.7-1 build on Jan 28 2011 13:24:19.libv4lconvert: warning more framesizes then I can handle!libv4lconvert: warning more framesizes then I can handle!Pipeline is live and does not need PREROLL ...Setting pipeline to PLAYING ...New clock: GstAudioSrcClock[ERR]   VPU iram is less than search ram size[WARN]  VPU iram is less than needed, some parts don't use iramWARNING: from element /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Can't record audio fast enoughAdditional debug info:gstbaseaudiosrc.c(822): gst_base_audio_src_create (): /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:Dropped 10752 samples. This is most likely because downstream can't keep up and is consuming samples too slowly.Got EOS from element "pipeline0".Execution ended after 9419134592 ns.Setting pipeline to PAUSED ...Setting pipeline to READY ...Setting pipeline to NULL ...Freeing pipeline ...

Joshua Parrish said:

Diane,

Is there anyway to get vlc to play nicely with the gstreamer/mfw plugins?

[JoshuaParrish]

Diane,

Is there anyway to get vlc to play nicely with the gstreamer/mfw plugins?

[daiane_angolini]

http://www.imxdev.org/wiki/index.php?title=I.MX27_ADS_Board_Video_GST_Video_Streaming

I have already made video streaming over ethernet. 

I worked a lot trying to get both video and audio streaming, but sincerely I have never done this personally.

It should be the same command line for encoding, but instead of sending it to a filesrc, you should send it to rtp src.

I found the following command line. It sends both audio and video, but it receives only video. You can include the audio part and give it a try

sudo vlc-wrapper -I dummy --play-and-exit 3500kbit_113_BBC_World.videostrada --sout '#rtp{mux=ts,sdp=rtsp:///movie}'

gst-launch rtspsrc location=rtsp://$SERVERIP/movie ! rtpmp2tdepay ! 'video/mpegts' ! aiurdemux name=demux demux. ! queue ! mfw_vpudecoder !  mfw_v4lsink

[JoshuaParrish]

Lyon Wang, 

Thanks! I didn't know that was the case.

Daiane,

Thank you for the wiki. What I'm trying to do is stream over the network. But I'm having a heck of a time getting something to work. I have had the following show only video. (which that is all it is supposed to do) 

gst-launch -v mfw_v4lsrc ! mfw_mpeg4encoder ! rtpmp4vpay ! udpsink host=$my_ip port=5434

One of the problems I'm having comes from trying to put two streams together in a mpeg 4 generic rtp stream. 

I've also tried to package separately,

I'm having the same issues these people did. 

http://forums.freescale.com/t5/i-MX-Microprocessors/imx51-Gstreamer-H264-rtph264pay-problem/td-p/657...

What I'm trying to do is have a live video+audio stream to a website. 

I know there will be some issues with rtp. 

I would really just like to get a stream with audio and video together. Then I can rtp/shout2send/darwin stream it. 

Is there any chance you've seen a stream like that built?

[daiane_angolini]

Hi Lyon,

I have a text file with some work I made around this topic some time ago.

http://www.imxdev.org/wiki/index.php?title=I.MX51_EVK_Board_Video

(I´m already formatting the wiki page, but you can take it now)

It´s for iMX51, but I believe you can use almost the same command line (if not the same)

Please, let me know the results.

[LyonWang]

Yes， MP4 container can mux h264 video and mp3 audio, you can have a try

[JoshuaParrish]

I would like to use it in conjunction with h264. Is there a container that can support that?

As far as I knew AAC and h264 in mp4 container go together. 

can mp3 +h264 = mp4?

[LyonWang]

there is mfw_mp3encoder which used optimized mp3encoder library, you can use that one