imx6 simultaneous overlay and capture?

[pasiojala]

Hi all,
I'm trying to make overlay and capture work simultaneously on sabreDL board (will be moving on to the solo processor later on). I tried searching the forums, but couldn't really find a definitive answer to "is it possible at all to overlay and capture at the same time?"

What I basically try do is

initMyOverlayAndCapture();
overlay=1;
ioctl(fd_v4l, VIDIOC_OVERLAY, &overlay);
while (1){
 xioctl(fd_capture, VIDIOC_DQBUF, &buf);
 handleTheDataInMyBuffer(); //takes anything between 5 to 500 ms
 xioctl(fd_capture, VIDIOC_QBUF, &buf);

}

i.e I'd like to see the overlay work smoothly while I'm picking up samples from the camera and processing them. However, when done like this, I don't get any proper data in my buffer. If I never actually start the overlay with the ioctl line, data flows nicely to my buffer, so the basic pieces are working (but not together).

If I do something like

initMyOverlayAndCapture();
while (1){
 usleep(200*1000);
 overlay=0;
 ioctl(fd_v4l, VIDIOC_OVERLAY, &overlay);
 xioctl(fd_capture, VIDIOC_DQBUF, &buf);
 handleTheDataInMyBuffer(); //takes anything between 5 to 500 ms
 xioctl(fd_capture, VIDIOC_QBUF, &buf);
 overlay=1;
 ioctl(fd_v4l, VIDIOC_OVERLAY, &overlay);
}

I get moments of overlay, as expected, and proper data in between from the camera... but this doesn't quite fulfill the "smoothly" requirement I have.

BR,
Pasi

[rogerio_silva]

You can check this user application example:

v4l2-examples/example2.c at master · rogeriorps/v4l2-examples · GitHub 

It's based on /unit_tests/mxc_v4l2_tvin.out

On lines 397 and 399 it makes a memcopy to copy the image between input and output buffers. The input buffer is available at "capture_buffers[capture_buf.index].start" and it's available for you to make whatever image processing you need.

The output (to display) buffer is output_buffers[output_buf.index].start.

Another option is to use gstreamer to record a video and display at the same time.

Best regards,

Rogerio

[pasiojala]

Hi again. The approach from your example works smooth enough for me with smaller images, but when I capture a 2592x1944 in NV12 format, that results in about 7.5M of data to be copied between the capture and output buffers (and another 5M I need to copy for my analysis - I only use the Y part of the picture), taking about 600ms on average. This is no longer good for me.

Maybe this approach is not the correct one for me. Other than gstreamer, is there any other way to accomplish what I need:

-set up a fluent flow from camera to display (possibly scaling down), with no or minimal processor load

-in the meanwhile, grab a frame from the camera, process it (takes an undefined time), grab a new frame when ready to process a new frame, continue ad infinitum.

If I need to start studying gstreamer for this, is there a reason to believe it does it's magic better and faster, without memcpy taking up that much time?

Thanks,

Pasi

[rogerio_silva · Accepted solution]

Hi,

Maybe you can try the attached patch that applies on unit test mxc_v4l2_tvin.c.

It changes the output buffer from V4L2_MEMORY_MMAP to V4L2_MEMORY_USERPTR. This way the memcpy is not needed to copy every frame to display. You can use memcpy only when you want to get a frame.

Best regards,

Rogerio

[pasiojala]

Hey Rogerio,

thanks for the tip, I owe you a beer:)

I was actually earlier trying to fiddle around with the in and out buffers to avoid the memcpy, but  didn't realize you could simply just use the capature buffer for both input and output, as in your patch.

BR,

Pasi

[pasiojala]

Thanks Rogerio, 

I have to give this some thought. It still doesn't look optimal for me, as I'd rather use the VIDIOC_OVERLAY procedure to get my camera pic onscreen (I don't need to modify it, maybe just scale), and grab some samples for my own processing in the meantime. Unless, of course, if it's not actually possible to have the overlay and capture work at the same time.

I could maybe try getting my samples directly from fb (where I'd be overlaying), but in case I need to scale down the camera image to screen, I'd be losing precision doing this.

Anyway, thanks, I'll give your suggestion a try and see how well it works for me.

BR,
Pasi