How to do video encoding (h264 .mp4) file using gstreamer in IMX6Q?

[johnturnur]

Hello

I am new to gstreamer and in our application we need to capture the video using and transmit it through Network using I.MX6Q board.

I am having I.MX6Q SDP board which contains the MIPI and parallel camera. For start i want to capture the video using parallel camera and i want to encode (as H264 mp4 file) it using VPU  and save that .mp4 file on the SD card.

I am new to the gsteamer , and i want to know that it can be possible to encode the MIPI captured data in to .h264 format and stored it in SD card as .mp4 file?

Which gstreamer plugin will be required and what is the command to perform encoding operation?

I have already build rootfs using gst-fsl-plugin and i have gst-inspect | grep mfw command and it gives me the following plugin .

isink.imx:  mfw_isink: IPU-based video sink

aacdec.imx:  mfw_aacdecoder: aac audio decoder

ipucsc.imx:  mfw_ipucsc: IPU-based video converter

mp3dec.imx:  mfw_mp3decoder: mp3 audio decoder

v4lsink.imx:  mfw_v4lsink: v4l2 video sink

mp3enc.imx:  mfw_mp3encoder: mp3 audio encoder

audiopeq.imx:  mfw_audio_pp: audio post equalizer

mpeg2dec.imx:  mfw_mpeg2decoder: mpeg2 video decoder

vorbisdec.imx:  mfw_vorbisdecoder: vorbis audio decoder

amrdec.imx:  mfw_amrdecoder: amr audio decoder

h264.imx:  mfw_h264decoder: h264 video decoder

v4lsrc.imx:  mfw_v4lsrc: v4l2 based camera src

mpeg4dec.imx:  mfw_mpeg4aspdecoder: mpeg4 video decoder

Please help me.

where is the gstreamer's plugin help document ?

Thanks,

J.

[LeonardoSandova]

There some documents on the community which can help you. As a starting point you can use this one:GStreamer i.MX6 Pipelines and for your specific need checkGStreamer i.MX6 Encoding

In the other hand, the element to do hardware encoding is called 'vpuenc' and it is not listed on the log you post. Make sure you have installed it otherwise install it like this

for cmd in prep scbuild scdeploy; do ./ltib -m $cmd -gst-fsl-plugins; done

Leo

[johnturnur]

Hi Leo,

Thanks for your answer.

I have rechecked and vpuenc and vpudec both are installed.

I have refer the link and i tried to do encoding using

# Parallel (J9 port):

modprobe ov5642_camera

modprobe mxc_v4l2_capture

gst-launch mfw_v4lsrc ! queue ! vpuenc codec=0 ! matroskamux ! filesink location=output.mkv sync=false

When i  tried to copy the file on my pc from board and try to play the encoded file on my PC using VLC player and it does not play.

Actually i want to encode the .mp4 with H264 as encoder. So i want muxer as mp4. How i can achieve this? what is the gst command to encode .mp4 file ?

Regards,

J.

[johnturnur]

Hello 

I want to capture the video using parallel camera available on i.MX6Q SDP board and encode it with H.264 encoder and Mp4 as demuxer. I want to store that .mp4 file on SD card.

I have tried with following  gst command but it is not working.

gst-launch mfw_v4lsrc fps-n=15 capture-mode=4 ! queue ! vpuenc codec=6 ! mp4mux ! filesink location=out.mp4  sync=false

I have tried to play .mp4 file on board using below command

gst-launch playbin2 uri=file:///out.mp4

it does not display anything either on PC or on board.

SO it looks that Mp4mux is not working.

After that i have tried to encode it using matroskamux using below command.

gst-launch mfw_v4lsrc fps-n=15 capture-mode=4 ! queue ! vpuenc codec=6 ! matroskamux ! filesink location=output.mkv sync=false

and try to play it using play2bin command and it played fine on board but it fails to play on PC using VLC media player.

Can anyone help me to get the .mp4 encoding working with mp4mux pipeline?

Please help on this issue.

Thanks,

J.

[LeonardoSandova]

Hi John,

On the PC, can you try other players? mplayer, totem are good ones, for example. Also, create an encoded file (no container)

gst-launch mfw_v4lsrc fps-n=15 capture-mode=4 ! queue ! vpuenc codec=6 !  filesink location=output.codec6 sync=false

And play it on your host. Also, set capture modes, just to validate this is not a camera error.

Leo

[johnturnur]

Hello Leo,

Thanks for ur answer. I was busy show could not check the suggestion provided by you.

Anyway i have tired to runt the command

gst-launch mfw_v4lsrc fps-n=15 capture-mode=4 ! queue ! vpuenc codec=6 !  filesink location=output.codec6 sync=false

It has capture the file and try to play file using gplay on board it only play one frame only.

Also when i tried to capture the Mp4 file using H.264 codec

gst-launch mfw_v4lsrc fps-n=15 capture-mode=4 ! queue ! vpuenc codec=6 ! mp4mux ! filesink location=out.mp4  sync=false

i have got the file out.mp4 which is not playing either using gplay on board or any other video player on PC.

i am attaching for your reference.

Please help me to get this mp4 file working with H264 as encoder.

Regards,

J.

[eaglezhou]

Hi, John

    I think it is one known issue, there is some compatibility issue between vpuenc/h264 and mp4mux.

    because mp4mux need to encoder plugin provide config data(SPS/PPS) explicitly and it only identify avc1 format.

    so you can try another mp4 mux, such as 'ffmux_mp4' instead of 'mp4mux'.

Regard

Eagle

[LeonardoSandova]

Hi Peng, if you have a link to learn more about this issue, please post it.

Thanks

[uwekaiser]

Hi,

we had the same problems on an IMX53 device.

The plugin mp4mux needs  the "codec_data" from the encoder plugin to create the avcC Box in the mp4 file.

This can be achieved by  setting "h264-byte-stream" to "0".

But the mfw_vpuencoder has a bug in the function, which creates the "codec_data" from the  SPS/PPS  data.

Apply the following patch to mfw_gst_vpu_encoder.c

This worked for us.

Regards,

Uwe

--- a/gst-fsl-plugin-2.0.3/src/video/vpu_enc/src/mfw_gst_vpu_encoder.c

+++ b/gst-fsl-plugin-2.0.3/src/video/vpu_enc/src/mfw_gst_vpu_encoder.c

@@ -82,7 +82,7 @@ static MfwGstVPU_Enc *vpuenc_global_ptr = NULL;

#define VPU_PIC_TYPE ((vpu_enc->codec == STD_AVC) ? ((vpu_enc->outputInfo->picType>>1)&0x3) : ((vpu_enc->outputInfo->picType)&0x3) )

#define VPU_PIC_TYPE_IDR ( (vpu_enc->codec == STD_AVC) ? (!(vpu_enc->outputInfo->picType&0x1)) : (VPU_PIC_TYPE == 0))

#define VPU_PIC_TYPE_I ( VPU_PIC_TYPE_IDR || (VPU_PIC_TYPE == 0))

-#define NALU_HEADER_SIZE    5

+#define NALU_HEADER_SIZE    4

#define AVC_NALU_START_CODE 0x00000001

#ifdef SWAP

@@ -1390,19 +1390,19 @@ GstBuffer *

mfw_gst_vpuenc_packetize_avchdr (guint8 * sps_data, guint sps_size,

     guint8 * pps_data, guint pps_size)

{

   guint data_size = 6 + (2 + (sps_size - NALU_HEADER_SIZE)) +

       1 + (2 + (pps_size - NALU_HEADER_SIZE));

   GstBuffer *codec_data = gst_buffer_new_and_alloc (data_size);

   guint8 *data = GST_BUFFER_DATA (codec_data);

   gint8 i = 0;

   /* unsigned int(8) configurationVersion = 1; */

   data[i++] = 1;

   /* unsigned int(8) AVCProfileIndication; */

-  data[i++] = sps_data[NALU_HEADER_SIZE];

-  /* unsigned int(8) profile_compatibility; */

   data[i++] = sps_data[NALU_HEADER_SIZE + 1];

-  /* unsigned int(8) AVCLevelIndication; */

+  /* unsigned int(8) profile_compatibility; */

   data[i++] = sps_data[NALU_HEADER_SIZE + 2];

+  /* unsigned int(8) AVCLevelIndication; */

+  data[i++] = sps_data[NALU_HEADER_SIZE + 3];

   /* bit(6) reserved = '111111'b;

    * unsigned int(2) lengthSizeMinusOne; */

   data[i++] = 0xFF;             /* lengthSizeMinusOne = 3 */

@@ -1411,7 +1411,7 @@ mfw_gst_vpuenc_packetize_avchdr (guint8 * sps_data, guint sps_size,

   data[i++] = 0xE1;             /* numOfSequenceParameterSets = 1 */

   /* unsigned int(16) sequenceParameterSetLength ; */

   data[i++] = 0;

   data[i++] = sps_size - NALU_HEADER_SIZE;

   /* bit(8*sequenceParameterSetLength) sequenceParameterSetNALUnit; */

   memcpy (data + i, sps_data + NALU_HEADER_SIZE, sps_size - NALU_HEADER_SIZE);

   i += sps_size - NALU_HEADER_SIZE;

[XXiao1z]

is this a real bug? fsl-gst-plugin 3.0.7 still has not applied this patch, but if it is indeed a bug, FSL should be notified somehow?

[eaglezhou]

Hi， Leo

   I just find one similar issue discussed in : https://community.freescale.com/thread/304415

   but seems not answered completely either.

Regard

Eagle.

[johnturnur]

Hello Leo,

Is there any update on this? We are still facing the issue to encode .Mp4 file using gst plugin.

Your help will be appreciated.

Regards,

J.

[LeonardoSandova]

Hi John.

Before we escalate the problem, try

1. Specify the device property on the source element (make sure it points to your camera)

2. Change the capture-mode property, try lowering the values

3. Are you using a NFS system? if yes, try to use a SD mounted system, just to avoid any network problem.

4. Enable debug flags, just to be sure there is nothing wrong on the encoding and muxing taks (--gst-debug=*enc*:2,*mux*:2)

Just to validate your camera is working try

gst-launch mfw_v4lsrc <place the same properties here> ! mfw_v4lsink

Leo