Some GStreamer issues while capturing and compressing analog video

[LautaroCarmona1]

Here are some issues we found in Freescale GStremer Plugins while trying to capture and compress analog video with TVP5147 and i.MX51 processor.

We managed to capture raw video and audio and mux them into an AVI file. We also managed to compress raw video using the VPU encoder, but only if we get the raw video to be encoded from a file. We couldn't compress the video on-the-fly while capturing.

In order to capture video we are using mfw_v4lsrc. As this module provides the raw video source in NV12 format, we are converting it to I420 on-the-fly with the mfw_ipucsc module. After this, we are using videorate module to adjust framerate to 25/1. Finally we use filesink to store the captured raw video to a file.

Then, to compress the raw video offline, we use filesrc, followed bymxc_vpuencoder (codec-type=0, bitrate=2000), then avimux and finally filesink to store the compressed video to a file.

The issues we are facing are:

1. If we do not use videorate module, the video framerate is way too high. A video of about 10 seconds is played in less than 2 seconds. It seems that mfw_v4lsrc module is doing wrong timestamps. Then, if we do use videorate, the framerate is well adjusted but video looks choppy (we believe that this module is eliminating a lot of frames to do its work).
2. If we capture video with a size different than 640x480, the mfw_vpuencoder module seems to ruin the original video.
3. If we try to encode the video while it is being captured, the pipeline generates an AVI file with an appropriate size, but it cannot be played.
4. We cannot deinterlace the raw video being capture, we tried mfw_deinterlacer with no success.
5. We are capturing video with mfw_v4lsrc with the property preview=true. If, instead, we use preview=false the captured video is not vertically synched: it rolls vertically.

If you know something about these issues or if you have any suggestions please let us know. 

[LautaroCarmona1]

Hi, all! Sorry for the last response

Steve G, we are happy that our work was valuable to you! Thanks for saying that! We managed to capture and compress video on the fly, with these lines: 

NTSC:

gst-launch-0.10 -e mfw_v4lsrc sensor-width=720 sensor-height=480 capture-width=720 capture-height=480 preview=true preview-width=360 preview-height=240 preview-top=180 preview-left=220  ! 'video/x-raw-yuv,format=(fourcc)NV12,width=720,height=480' ! queue max-size-buffers=1000 ! mfw_vpuencoder codec-type=0 bitrate=2000 width=720 height=480 ! avimux name=mux ! filesink location=NTSCfps30.avi alsasrc device=hw:0,0 ! 'audio/x-raw-int,rate=44100,channels=2,depth=16' ! queue ! mfw_mp3encoder ! mux.

PAL:

gst-launch-0.10 -e mfw_v4lsrc sensor-width=720 sensor-height=576 capture-width=720 capture-height=576 preview=true preview-width=360 preview-height=288 preview-top=156 preview-left=220 ! 'video/x-raw-yuv,format=(fourcc)NV12,width=720,height=576,framerate=(fraction)30/1' ! mfw_ipucsc ! 'video/x-raw-yuv,format=(fourcc)I420,wifth=720,height=576,framerate=(fraction)30/1' ! videorate ! 'video/x-raw-yuv,framerate=(fraction)25/1' ! queue max-size-buffers=1000 ! mfw_vpuencoder codec-type=0 bitrate=2000 width=720 height=576 framerate=25 ! avimux name=mux ! filesink location=PALfps25.avi alsasrc device=hw:0,0 ! 'audio/x-raw-int,rate=44100,channels=2,depth=16' ! queue ! mfw_mp3encoder ! mux

Alex, thanks for your sharing your experience with us! After asking here in imxcommunity about the deinterlacer, we've been told that it works while playing, but now while capturing video, just as you say. About accelerated video, with the lines above, we didn't have problems with NTSC video, but with PAL video we still needed to use the videorate plugin. Anyway, we're still experiencing some lags with the audio sync with PAL.

Thanks both of you for replying!

[fcs]

I have had a similar problem, mfw_v4lsink produces rolling video if no preview is occuring. However due to latency it is preferable to use the preview function anyhow (it even continues operating after a system halt which is neat).

I have found that although the preview can be stretched horizontally to be 1024 pixels across, any height for the preview more than about 664 produces flickering lines in the displayed preview.

gst-launch-0.10 -e mfw_v4lsrc sensor-width=720 sensor-height=288 capture-width=720 capture-height=576 preview=true preview-width=1024 preview-height=664 fps-n=25 ! mfw_vpuencoder ! avimux name=mux ! filesink location=/mnt/usb1/PALfps25b.avi

I would like to stretch this to a height of 768 if possible. I am wondering if this problem can be fixed in the ipu_* drivers in the functions prpvf_start( void *private ), which initialise the preview and are called from drivers/media/video/mxc/capture/mxc_v4l2_capture.c function start_preview( cam_data *cam ), where cam holds the desired preview width and height.

Note sensor height is half the vertical resolution of the PAL signal, otherwise the image is displayed twofold in the preview, both squashed, one above the other. These appear to be the top and bottom fields (the PAL signal's "partial line" at the top of the first field is two pixels high with sensor-height=288 but only 1 pixel high in the twofold-squashed preview with sensor-height=576). Therefore I am previewing and encoding only half the vertical resolution.

The e-consystems video capture board e-CANMT_MX53x has a demo program capture.elf which demonstrates previews of captured video. It is able to preview the full vertical resolution by means of ioctl calls with option 5 "select interleaved mode". Interestingly we can't see any difference between option 4 "select Bottom field" and option 6 "select de-interlaced mode" in this demo program when the output is scaled to full screen. The partial line at the top of the screen is about 2 pixels high in both cases.

I am now interested in putting an overlay over the preview. In drivers/media/video/mxc/capture/ipu_prp_vf_sdc_bg.c there is an IRQ callback function prpvf_sdc_vsync_callback, which is absent from drivers/media/video/mxc/capture/ipu_prp_vf_sdc.c, so I am guessing this is to enable drawing an overlay over the captured video?

[alexgonzalez]

Updating the issues with accelerated video, in our case it was caused by the sensor auto exposure modifying the frame rate dynamically in response to the changes in light condition.

Turning off auto exposure the recording is fine.

Alex Gonzalez said:

Hi Lautaro,

I am using Digi Embedded Linux on a ConnectCore Wi-i.MX51. It's running a 2.6.35 kernel with a ported version of Freescale's 11.09 BSP, modified and bug fixed to run on the hardware. The user space is running the Freescale 2.0.3 multimedia kit.

We have encoded on the fly from a mt9v111 camera sensor connected to a CSI port.

The following pipeline works for us:

gst-launch mfw_v4lsrc device=/dev/video0 num-buffers=200 preview=0 ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1,format=(fourcc)UYVY' \
 ! mfw_ipucsc ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1,format=(fourcc)I420' ! queue ! mfw_vpuencoder codec-type=2 ! \
 queue ! avimux name=mux ! filesink location=test.avi

We can play back the recorded file with gst-launch and mplayer.

However we have common issues:

1) The video played back is accelerated.
2) We also need to capture in 640 x480.
3) If we use preview=1 we are seeing a seg fault.

We are still debugging these issues so I don't know if the problems lie with the plugins or the BSP.

We don't need to deinterlace, but the hardware deinterlacer is working if the source is an interlaced compressed video file. Did you try an open source software deinterlacer instead like ffdeinterlace?

Regards,

Alex

[alexgonzalez]

Hi Lautaro,

I am using Digi Embedded Linux on a ConnectCore Wi-i.MX51. It's running a 2.6.35 kernel with a ported version of Freescale's 11.09 BSP, modified and bug fixed to run on the hardware. The user space is running the Freescale 2.0.3 multimedia kit.

We have encoded on the fly from a mt9v111 camera sensor connected to a CSI port.

The following pipeline works for us:

gst-launch mfw_v4lsrc device=/dev/video0 num-buffers=200 preview=0 ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1,format=(fourcc)UYVY' \
 ! mfw_ipucsc ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1,format=(fourcc)I420' ! queue ! mfw_vpuencoder codec-type=2 ! \
 queue ! avimux name=mux ! filesink location=test.avi

We can play back the recorded file with gst-launch and mplayer.

However we have common issues:

1) The video played back is accelerated.
2) We also need to capture in 640 x480.
3) If we use preview=1 we are seeing a seg fault.

We are still debugging these issues so I don't know if the problems lie with the plugins or the BSP.

We don't need to deinterlace, but the hardware deinterlacer is working if the source is an interlaced compressed video file. Did you try an open source software deinterlacer instead like ffdeinterlace?

Regards,

Alex

[StevieRG]

Hello Lautaro, we also have problems using gstreamer for capture.

We made a bt.656 driver using your extremely useful published work on the TVP5147, many thanks for that valuable work. In our system there is no I2C TV chip, we just get a raw bt.656 interface from another video device, so we simply removed the I2C code.

The driver loads up and works very well for the /unit_tests/mxc_v4l2_tvin demo - a great (PAL) camera picture. But it does not work with gstreamer.

The most successful command line we found so far to get a live video preview is:
gst-launch-0.10 mfw_v4lsrc device="/dev/video0" ! 'video/x-raw-yuv,width=720,height=625,framerate=25/1' ! mfw_v4lsink
... this shows video but it is rolling. I noticed you tried preview=true, but for us we get "IPU Error - IPU_INT_STAT_10" and the driver seems to stop at VIDIOC_S_CTRL.

You describe using mfw_ipucsc and other gstreamer modules, could you share please some gstreamer command lines you have used? I'll give them a go and see if we can work things out too.

I'm wondering if anyone has had success with gstreamer capture on the i.MX at all?