i.MX53 IPU Deinterlacer

[CraigPetku]

Has anyone come across a white paper or a Linux driver which implements the HW deinterlacer found in the i.MX53?  After getting an ADV7180 running on CSI1, I now have odd/even fields that need to be combined (preferably using the IPU).  For a quick demo using the unit_tests it's easy just to crop and scale, but I'd really like to see the IPU do the heavy lifting.

[rogerio_silva]

About avoiding memcpy on mxc_v4l2_tvin.c, please, apply the following patch on  mxc_v4l2_tvin.c released with the latest i.MX53 BSP (11.09).

It will point the output buffer to the capture buffer. It's taking about 1.2% of cpu load instead of 50%.

Rgds

Rogerio

[ziltoidtheomnis]

Hi,

Thanks for the patch. It is exciting to find that a higher performance version of mxc_v4l2_tvin is possible.

I have tried the patch with latest i.MX53 BSP (11.09) and found that it fails on the ioctl in bold (return code < 0) running on an i.MX53 QSB.

@@ -327,7 +301,7 @@ int v4l_output_setup(void)

         memset(&buf_req, 0, sizeof(buf_req));

         buf_req.count = g_output_num_buffers;

         buf_req.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;

-        buf_req.memory = V4L2_MEMORY_MMAP;

+        buf_req.memory = V4L2_MEMORY_USERPTR;

         if (ioctl(fd_output_v4l, VIDIOC_REQBUFS, &buf_req) < 0

Any feedback on why the ioctl is failing would be very appreciated.

I have done some investigation on why the output device (/dev/video16) is failing on the ioctl and found the associated driver mxc_v4l2_output.c does not handle V4L2_MEMORY_USERPTR. Is this assumption on the ioctl failure correct?

Regards,

Ziltoid

[rogerio_silva]

Are you using the kernel from git? Try to use this branch:

http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/log/?h=imx_2.6.35_11.09.01

and check if V4L2_MEMORY_USERPTR is already supported on this kernel version. I think this support was added later than 2009 and this branch has newer patches.

[ziltoidtheomnis]

started with a freescale 2.6.35.3 (11.03)

patched the kernel to the latest 2.6.35 (11.09)

confirmed we have up to the latest patch - ENGR00252150 Fix MX53 IOMUX setup for all UARTs

[compmas2]

I have used this patch to the mxc_v4l2_tvin application successfully with the imx_2.6.35_maintain kernel (http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/log/?h=imx_2.6.35_maintain).  This is the kernel which is receiving regular updates and includes everything in the 11.09 release plus new bug fixes.  Use the following link to see where the mxc_v4l2_output driver in the imx_2.6.35_maintain kernel supports V4L2_MEMORY_USERPTR (linux-2.6-imx.git - Freescale i.MX Linux Tree).

[mithunk]

Hi Craig,

 Please can you explain how did you achieve frame sync for CSI->MEM channel??

 Because i am getting scroll for CSI->MEM and no scroll is found on CSI->IC path. 

Thanks and Regards,

Mith

Craig Petku said:

Ok, I got the ADV7180 working with the Android camera app.  I'll have several action items for my local rep (when I see him) e.g., explain the synchronization strategy between the CSI port and the IDMAC in CSI->MEM. 

It looks like in the CSI->MEM path the CSI free runs and the IDMAC generates the camera call back every time it completes capturing one frame of data without syncronizing SOF to the CSI port SAV code timing.  If so, there's potential for these to get out of sync.

In the CSI->IC path syncronization appears to be done appropriately resulting in a stable image always.

Of course this could just be the current state of Linux drivers and not a design limitation,.

Once you get the frames synced properly and adjust offsets to account for the porches, then the de-interlacer in the IDMAC will start to behave as well.

[ashutoshsrivast]

Hi Mithun,

Were you able to resolve this issue with TVP5150. I am facing same issues on iMX6 Android integration of

TVP5150 as a camera interface.

Best regards,

Ashu

[rogerio_silva]

Hi,

For scroll issue, please, try to apply the patches posted at i.MX535 Product Summary Page

Updates ad patches -> IMX53_LINUX_1109_BSP_PATCH

It worked for me.

Rgds

Rogerio

[AdamSutton]

Hi Ricky,

No, I'm no longer getting any scrolling. The only change I can remember that made it go away was changing the active_top=3 in the mcx_v4l2_capture code.

My driver sets very little, basically just enough to get it outputting video in the right TV mode:

TVP5150_REV_SELECT // AM1
TVP5150_OP_MODE_CTL // select TV mode
TVP5150_VD_IN_SRC_SEL_1 // select input source
TVP5150_VIDEO_STD // select TV standard
TVP5150_MISC_CTL // start the output

I've attached my driver for you to take a look at. I think I based it on an older version of the mainline driver (before it was updated to the soc framework) and then simplified it and added a few mods for my own purposes.

Adam

[mithunk]

Hi Adam,

       Whether your are getting any scroll issue with NTSC??

        If No, can please tell me the TVP5150 register value used. 

        I tried changing active_top =3 but did't solve the problem.

Adam Sutton said:

Hi All,

Good to see I'm not the only one having trouble getting this to work...

Some months ago I gave up getting the de-interlacing working, and used the crop first field hack. This was alright for early prototype work but now I really need to get things working properly.

I'm using a TVP5150 chip. I've modded the MXC driver to pass S_INPUT to the TVP driver as I need to use both analogue inputs. I've retained the ability to switch the MXC CSI input using a build switch (once I have a working solution it will be fixed anyway).

Currently I'm using an NTSC camera. Previous work (without de-interlacing) worked fine with PAL-I and NTSC, but I'm sticking to NTSC for now for getting the de-interlacing to work as people seem to have generally had more issues with this.

I've made some progress and now have a de-interlaced image. This is via the CSI->MEM route, using the active_top=3 mod suggested on here. The issue is that the de-interlacer appears to have misaligned the two fields (only 1 pixel out I think). No idea how this can be adjusted, do the CCIR_CODES play any part in this or is there some field offset param buried somewhere?

Finally what hope is there of me getting things working via the PRP route? So far I've had zero luck in getting this to de-interlace, but this does seem to offer greater flexibility in the output image format I can select (pixel format and size etc...). So I would ideally like to use this.

Regards

Adam

[AdamSutton]

Hi All,

Good to see I'm not the only one having trouble getting this to work...

Some months ago I gave up getting the de-interlacing working, and used the crop first field hack. This was alright for early prototype work but now I really need to get things working properly.

I'm using a TVP5150 chip. I've modded the MXC driver to pass S_INPUT to the TVP driver as I need to use both analogue inputs. I've retained the ability to switch the MXC CSI input using a build switch (once I have a working solution it will be fixed anyway).

Currently I'm using an NTSC camera. Previous work (without de-interlacing) worked fine with PAL-I and NTSC, but I'm sticking to NTSC for now for getting the de-interlacing to work as people seem to have generally had more issues with this.

I've made some progress and now have a de-interlaced image. This is via the CSI->MEM route, using the active_top=3 mod suggested on here. The issue is that the de-interlacer appears to have misaligned the two fields (only 1 pixel out I think). No idea how this can be adjusted, do the CCIR_CODES play any part in this or is there some field offset param buried somewhere?

Finally what hope is there of me getting things working via the PRP route? So far I've had zero luck in getting this to de-interlace, but this does seem to offer greater flexibility in the output image format I can select (pixel format and size etc...). So I would ideally like to use this.

Regards

Adam

[Ananth]

make sure g_input=1 in mxc_v4l2_tvin.c

    if (ioctl(fd_capture_v4l, VIDIOC_S_INPUT, &g_input) < 0)
    {
        printf("VIDIOC_S_INPUT failed\n");
        close(fd_capture_v4l);
        return TFAIL;
    }

for me output video worked in -fg foreground mode is selected.

[GopiNagaBharath]

Hi Ananthapadmanaban,

We using TVP5147M1 + i.MX53 (CSI1 Port).

When we rung the capture test command, we are seeing that we are getting only the first frame seen by the camera on LCD.

The analog input to the camera is composite video input, the i.MX53 is using 8-bit CSI data port with clock. The EAV & SAV are included in the data.

Do you have any idea what could be the issue?

[mithunk]

mxc_tvin_deinterlace.c application(default for PAL ) is provided by freescale separately , you can't find in mxc_v4l2_test.

please find the attachement.

I used mxc_v4l2_tvin.c which is working properly,without scrolling of display. but i get two frames in the display.

Thank you

Ananthapadmanaban said:

mxc_tvin_deinterlace.c where the file is available. I cannot able to find in mxc_v4l2_test.

For mxc_v4l2_tvin.c it works correct, if i set 0 to PAL and 3 to NTSC .active_top

<From application>

   if (g_current_std == V4L2_STD_NTSC)
    {
        g_cam_top = 3;
    }else
    {
        g_cam_top = 0;
    }

    memset(&crop, 0, sizeof(crop));
    crop.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    crop.c.left = g_cam_left;
    crop.c.top = g_cam_top;
    crop.c.width = g_cam_c_width;
    crop.c.height = g_cam_c_height;

    if (ioctl(fd_capture_v4l, VIDIOC_S_CROP, &crop) < 0)
    {
        printf("VIDIOC_S_FMT failed\n");
        close(fd_capture_v4l);
        return TFAIL;
    }

[henri_bragge]

Dearest Mithun or Ananthapadmanaban,

Can you please share code for "mxc_tvin_deinterlace.c" or the version of "mxc_v4l2_tvin.c" that uses overlay instead of memcpy? I have tried to look for it in the Android and Linux BSP packages and everywhere else too with no success.

Thanks!

[fcs]

Here is a non-memcpy version I found.

I too would like to use overlay instead of memcpy but I have found no code that does deinterlacing without memcpy.... why would this be!?

I would like to avoid the latency inherent in doing a memcpy when DMA can do it so much faster.

I too would like the source code for mxc_tvin_deinterlace.c

The 2009 version of mxc_v4l2_tvin.c used "/dev/graphics/fb0", I changed this to "/dev/fb0":

/*
* Copyright 2007-2009 Freescale Semiconductor, Inc. All rights reserved.
*/

/*
* The code contained herein is licensed under the GNU General Public
* License. You may obtain a copy of the GNU General Public License
* Version 2 or later at the following locations:
*
* http://www.opensource.org/licenses/gpl-license.html
* http://www.gnu.org/copyleft/gpl.html
*/

/*
* @file mxc_v4l2_tvin.c
*
* @brief Mxc TVIN For Linux 2 driver test application
*
*/

#ifdef __cplusplus
extern "C"{
#endif

/*=======================================================================
                                        INCLUDE FILES
=======================================================================*/
/* Standard Include Files */
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

/* Verification Test Environment Include Files */
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <asm/types.h>
#include <linux/videodev.h>
#include <linux/videodev2.h>
#include <linux/fb.h>
#include <sys/mman.h>
#include <math.h>
#include <string.h>
#include <malloc.h>

#include <linux/mxcfb.h>

#define TFAIL -1
#define TPASS 0

char v4l_device[100] = "/dev/video0";
int fd_v4l = 0;
int g_cap_mode = 0;
v4l2_std_id g_current_std = V4L2_STD_PAL;

/* PAL active frame size is 720 * 576, output is interlaced mode */
#define PAL_WIDTH             720
#define PAL_HEIGHT            288
/* Display area on LCD */
#define DISPLAY_WIDTH         1024
#define DISPLAY_HEIGHT        694
#define DISPLAY_TOP           74
#define DISPLAY_LEFT          0

int
mxc_v4l_tvin_test(void)
{
        int overlay = 1;
     v4l2_std_id id;
     struct v4l2_streamparm parm;
        parm.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
        parm.parm.capture.timeperframe.numerator = 1;
        parm.parm.capture.timeperframe.denominator = 0;
        parm.parm.capture.capturemode = 0;
        if (ioctl(fd_v4l, VIDIOC_OVERLAY, &overlay) < 0)
        {
            printf("VIDIOC_OVERLAY start failed\n");
   return TFAIL;
        }

       /* Detect standard monitor, the default video standard is PAL when the test programe is started */

  g_cap_mode =1;
  parm.parm.capture.capturemode = 1;//g_cap_mode;
  ioctl(fd_v4l, VIDIOC_S_PARM, &parm);
  while (1)
  {
         if (ioctl(fd_v4l, VIDIOC_G_STD, &id) < 0)
            {
               printf("overlay test programe: VIDIOC_G_STD failed\n");
               return TFAIL;
            }
      ioctl(fd_v4l, VIDIOC_S_STD, &id);
   sleep(1);
        }
    return 0;
}

int
mxc_v4l_tvin_setup(struct v4l2_format *fmt)
{
        struct v4l2_streamparm parm;
        struct v4l2_control ctl;
        struct v4l2_crop crop;
  int display_lcd = 0;

        if (ioctl(fd_v4l, VIDIOC_S_OUTPUT, &display_lcd) < 0)
        {
                printf("VIDIOC_S_OUTPUT failed\n");
                return TFAIL;
        }

        ctl.id = V4L2_CID_PRIVATE_BASE;
   ctl.value = 0;
        if (ioctl(fd_v4l, VIDIOC_S_CTRL, &ctl) < 0)
        {
                printf("set control failed\n");
                return TFAIL;
        }

  parm.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
        parm.parm.capture.timeperframe.numerator = 1;
        parm.parm.capture.timeperframe.denominator = 0;
        parm.parm.capture.capturemode = 0;
  if (ioctl(fd_v4l, VIDIOC_S_PARM, &parm) < 0)
        {
                printf("VIDIOC_S_PARM failed\n");
                return TFAIL;
        }

  parm.parm.capture.timeperframe.numerator = 0;
        parm.parm.capture.timeperframe.denominator = 0;
  if (ioctl(fd_v4l, VIDIOC_G_PARM, &parm) < 0)
        {
                printf("get frame rate failed\n");
                return TFAIL;
        }

  g_cap_mode = parm.parm.capture.capturemode;
        printf("cap_mode is %d\n", parm.parm.capture.capturemode);
        crop.type = V4L2_BUF_TYPE_VIDEO_OVERLAY;
        crop.c.left = 0;
        crop.c.top = 0;
        crop.c.width = PAL_WIDTH;
        crop.c.height = PAL_HEIGHT;
        if (ioctl(fd_v4l, VIDIOC_S_CROP, &crop) < 0)
        {
                printf("set cropping failed\n");
                return TFAIL;
        }

        if (ioctl(fd_v4l, VIDIOC_S_FMT, fmt) < 0)
        {
                printf("set format failed\n");
                return TFAIL;
        }

        if (ioctl(fd_v4l, VIDIOC_G_FMT, fmt) < 0)
        {
                printf("get format failed\n");
                return TFAIL;
        }

        return TPASS;
}

int
main(int argc, char **argv)
{
        struct v4l2_format fmt;
        struct v4l2_framebuffer fb_v4l2;
        char fb_device[100] = "/dev/fb0";
        int fd_fb = 0;
        struct fb_fix_screeninfo fix;
        struct fb_var_screeninfo var;
        struct mxcfb_color_key color_key;
        struct mxcfb_gbl_alpha alpha;
        unsigned short * fb0;
        unsigned char * cur_fb8;
        unsigned short * cur_fb16;
        unsigned int * cur_fb32;
        __u32 screen_size;
        int h, w;
int ret = 0;

        if ((fd_v4l = open(v4l_device, O_RDWR, 0)) < 0) {
                printf("Unable to open %s\n", v4l_device);
                return TFAIL;
        }

        fmt.type = V4L2_BUF_TYPE_VIDEO_OVERLAY;
        fmt.fmt.win.w.top =  DISPLAY_TOP ;
        fmt.fmt.win.w.left = DISPLAY_LEFT;
        fmt.fmt.win.w.width = DISPLAY_WIDTH;
        fmt.fmt.win.w.height = DISPLAY_HEIGHT;
  //printf("\nFramebuffer values=> top=%d,left=%d,width=%d,height=%d\n",fmt.fmt.win.w.top,fmt.fmt.win.w.left,fmt.fmt.win.w.width,fmt.fmt.win.w.height);

        if (mxc_v4l_tvin_setup(&fmt) < 0) {
                printf("Setup tvin failed.\n");
                return TFAIL;
  }

        memset(&fb_v4l2, 0, sizeof(fb_v4l2));

        if ((fd_fb = open(fb_device, O_RDWR )) < 0) {
                printf("Unable to open frame buffer\n");
                return TFAIL;
        }

        if (ioctl(fd_fb, FBIOGET_VSCREENINFO, &var) < 0) {
                close(fd_fb);
    printf("FBIOGET_VSCREENINFO\n");
                return TFAIL;
        }
        if (ioctl(fd_fb, FBIOGET_FSCREENINFO, &fix) < 0) {
                close(fd_fb);
    printf("FBIOGET_FSCREENINFO\n");
                return TFAIL;
        }

        /* Overlay setting */
        alpha.alpha = 0;
alpha.enable = 1;
if ( ioctl(fd_fb, MXCFB_SET_GBL_ALPHA, &alpha) < 0) {
                close(fd_fb);
    printf("MXCFB_SET_GBL_ALPHA\n");
                return TFAIL;
}

        color_key.color_key = 0x00080808;
        color_key.enable = 1;
        if ( ioctl(fd_fb, MXCFB_SET_CLR_KEY, &color_key) < 0) {
                close(fd_fb);
    printf("MXCFB_SET_CLR_KEY\n");
                return TFAIL;
        }

        screen_size = var.yres * fix.line_length; // 480*800

        /* Map the device to memory*/
        fb0 = (unsigned short *)mmap(0, screen_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd_fb, 0);
        if ((int)fb0 == -1)
        {
                printf("\nError: failed to map framebuffer device 0 to memory.\n");
                close(fd_fb);
                return TFAIL;
  }

        if (var.bits_per_pixel == 16) {
                for (h = DISPLAY_TOP; h < (DISPLAY_HEIGHT+ DISPLAY_TOP); h++) {
                        cur_fb16 = (unsigned short *)((__u32)fb0 + h*fix.line_length);
                        for (w = DISPLAY_LEFT; w < DISPLAY_WIDTH + DISPLAY_LEFT; w++) {
                                cur_fb16[w] = 0x0841;
                        }
                }
        }
        else if (var.bits_per_pixel == 24) {
                for (h = DISPLAY_TOP; h < (DISPLAY_HEIGHT + DISPLAY_TOP); h++) {
                        cur_fb8 = (unsigned char *)((__u32)fb0 + h*fix.line_length);
                        for (w = DISPLAY_LEFT; w < DISPLAY_WIDTH + DISPLAY_LEFT; w++) {
                                *cur_fb8++ = 8;
                                *cur_fb8++ = 8;
                                *cur_fb8++ = 8;
                        }
                }
        }
        else if (var.bits_per_pixel == 32) {
                for (h = DISPLAY_TOP; h < (DISPLAY_HEIGHT + DISPLAY_TOP); h++) {
                        cur_fb32 = (unsigned int *)((__u32)fb0 + h*fix.line_length);
                        for (w = DISPLAY_LEFT; w < DISPLAY_WIDTH + DISPLAY_LEFT; w++) {
                                cur_fb32[w] = 0x00080808;
                        }
                }
        }
        if (ioctl(fd_v4l, VIDIOC_G_FBUF, &fb_v4l2) < 0) {
                printf("Get framebuffer failed\n");
                return TFAIL;
        }
        fb_v4l2.flags = V4L2_FBUF_FLAG_OVERLAY;

        close(fd_fb);

        if (ioctl(fd_v4l, VIDIOC_S_FBUF, &fb_v4l2) < 0)
        {
                printf("set framebuffer failed\n");
                return TFAIL;
        }

        if (ioctl(fd_v4l, VIDIOC_G_FBUF, &fb_v4l2) < 0) {
                printf("set framebuffer failed\n");
                return TFAIL;
        }
        //fb_v4l2.fmt.width=800;
     //fb_v4l2.fmt.height=480;
  //fb_v4l2.fmt.bytesperline=1600;

        //printf("\n frame buffer width %d, height %d, bytesperline %d\n", fb_v4l2.fmt.width, fb_v4l2.fmt.height, fb_v4l2.fmt.bytesperline); // 800 480 1600
        ret = mxc_v4l_tvin_test();
  //printf("\nmxc_v4l_tvin_test-return value=%d\n", ret);
        close(fd_v4l);
        return ret;
}

[Ananth]

mxc_tvin_deinterlace.c where the file is available. I cannot able to find in mxc_v4l2_test.

For mxc_v4l2_tvin.c it works correct, if i set 0 to PAL and 3 to NTSC .active_top

<From application>

   if (g_current_std == V4L2_STD_NTSC)
    {
        g_cam_top = 3;
    }else
    {
        g_cam_top = 0;
    }

    memset(&crop, 0, sizeof(crop));
    crop.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    crop.c.left = g_cam_left;
    crop.c.top = g_cam_top;
    crop.c.width = g_cam_c_width;
    crop.c.height = g_cam_c_height;

    if (ioctl(fd_capture_v4l, VIDIOC_S_CROP, &crop) < 0)
    {
        printf("VIDIOC_S_FMT failed\n");
        close(fd_capture_v4l);
        return TFAIL;
    }

[mithunk]

Thanks Ananthapadmanaban for your reply,

Well i had tried the same, as you mentioned in the earlier post. 

i am using TV-in deinterlace application(mxc_tvin_deinterlace.c) which is working fine for PAL and display scrolls for NTSC.

but overlay applications(mxc_v4l2_overlay and mxc_v4l2_tvin) are working fine on display.

Any suggestion will be appreciated.

Ananthapadmanaban said:

modified drivers/media/video/mxc/capture/mxc_v4l2_capture.c .active_top to make it work.

Kumar said:

The output is rolling in the screen for NTSC.

Any chnages need to be done in CCIR code register for CSI0?

Any suggestion !!

Thanks in advance,

charles chang said:

I use ADV7180 + imx51.  android 10.3.1

the unit test program mxc_v4l2_tvin runs well, it output de-interlaced frame on overlay. and no need to memcpy each frame.

--  the mxv_v4l2_tvin code is different fromt imx53. though they have the same filename.

just a little bug , the default video standard is PAL, it sync fail when we input the NTSC video.

and 

sometime it hangs when video source is disappeared for a short period.

[Ananth]

modified drivers/media/video/mxc/capture/mxc_v4l2_capture.c .active_top to make it work.

Kumar said:

The output is rolling in the screen for NTSC.

Any chnages need to be done in CCIR code register for CSI0?

Any suggestion !!

Thanks in advance,

charles chang said:

I use ADV7180 + imx51.  android 10.3.1

the unit test program mxc_v4l2_tvin runs well, it output de-interlaced frame on overlay. and no need to memcpy each frame.

--  the mxv_v4l2_tvin code is different fromt imx53. though they have the same filename.

just a little bug , the default video standard is PAL, it sync fail when we input the NTSC video.

and 

sometime it hangs when video source is disappeared for a short period.

[mithunk]

The output is rolling in the screen for NTSC.

Any chnages need to be done in CCIR code register for CSI0?

Any suggestion !!

Thanks in advance,

charles chang said:

I use ADV7180 + imx51.  android 10.3.1

the unit test program mxc_v4l2_tvin runs well, it output de-interlaced frame on overlay. and no need to memcpy each frame.

--  the mxv_v4l2_tvin code is different fromt imx53. though they have the same filename.

just a little bug , the default video standard is PAL, it sync fail when we input the NTSC video.

and 

sometime it hangs when video source is disappeared for a short period.