Frame Rate drops during heavy system Load in i.MX6 MIPI-CSI2 camera

[rashmighoaelmur]

Hi,

We are using i.Mx6 QuadPlus Nitrogen Board with Kernel Version 3.14.52.
We are using proprietary Camera with our customized driver through MIPI-CSI2 interface.
A small region of interset is set (120 x 120) with minimum exposure and RGB888 format, then the camera product produces 120fps.
For testing we have used a simple command line application and do a continuous streaming.

Initially we are getting the frame rate as expected 120fps.
We tried increasing the CPU load during streaming, so we are compiling a piece of code with Makefile (Make -j4) which utilizes all CPU cores.
At that time, the frame rate dropped down to 38fps and it is not recovered back even after the stop of compilation.

We observed few things in the mxc v4l2 driver - (mxc_v4l2_capture.c - camera_callback) camera callback implementation.
    For testing, we have added a timer which calculates the number of times the interrupt hits in camera_callback function, which is the frame rate count
and the timer will restart after every 5 seconds.
    
    In camera_callback function, till this point the frame rate remains as 120fps.

    spin_lock(&cam->queue_int_lock);
    spin_lock(&cam->dqueue_int_lock);
    fps++; //variable to check fps count
    
    During the execution of below code, the fps rate dropped to 38.
    
    if (!list_empty(&cam->working_q)) {
        do_gettimeofday(&cur_time);

        done_frame = list_entry(cam->working_q.next,
                    struct mxc_v4l_frame,
                    queue);

        if (done_frame->ipu_buf_num != cam->local_buf_num)
            goto next;

        /*
         * Set the current time to done frame buffer's
         * timestamp. Users can use this information to judge
         * the frame's usage.
         */
        done_frame->buffer.timestamp = cur_time;

        if (done_frame->buffer.flags & V4L2_BUF_FLAG_QUEUED) {
            done_frame->buffer.flags |= V4L2_BUF_FLAG_DONE;
            done_frame->buffer.flags &= ~V4L2_BUF_FLAG_QUEUED;

            /* Added to the done queue */
            list_del(cam->working_q.next);
            list_add_tail(&done_frame->queue, &cam->done_q);

            /* Wake up the queue */
            cam->enc_counter++;
            wake_up_interruptible(&cam->enc_queue);
        } else
            pr_err("ERROR: v4l2 capture: camera_callback: "
                "buffer not queued\n");
    }

next:
    if (!list_empty(&cam->ready_q)) {
        ready_frame = list_entry(cam->ready_q.next,
                     struct mxc_v4l_frame,
                     queue);
        if (cam->enc_update_eba)
            if (cam->enc_update_eba(
                cam,
                ready_frame->buffer.m.offset) == 0) {
                list_del(cam->ready_q.next);
                list_add_tail(&ready_frame->queue,
                          &cam->working_q);
                ready_frame->ipu_buf_num = cam->local_buf_num;
            }
    } else {
        if (cam->enc_update_eba)
            cam->enc_update_eba(
                cam, cam->dummy_frame.buffer.m.offset);
    }

    cam->local_buf_num = (cam->local_buf_num == 0) ? 1 : 0;
    spin_unlock(&cam->dqueue_int_lock);
    spin_unlock(&cam->queue_int_lock);

    We need a code patch, to fix this issue.
    Kindly clarify and get back with us ASAP. Thanks in advance.

Please refer the below link:

https://github.com/boundarydevices/linux-imx6/blob/boundary-imx_3.14.52_1.1.0_ga/drivers/media/platf...
 

Best Regards,

Rashmi

[jimmychan]

Hi gary_bisson‌,

Do you have any suggestion/idea? Thanks.

[gary_bisson]

Hi,

No unfortunately I don't have an answer. Can this be reproduced with one of our supported cameras that streams 1080p @30fps?

Since this leverage a custom driver with a custom camera, many things can be involved here.

Regards,

Gary

[rashmighoaelmur]

Hi Gary,

We are not able to reproduce the same with omnivision camera since it capture frames @30fps. This problem occurs only with higher frame rates.

During Queue, Dequeue and camera_callback MXC core BSP driver only involves, there is no intervention of our customized driver here. Our driver involves during stream-on and stream-off and other control actions (change of any camera features).

After stream on, we are overloading CPU and frame rate goes down. We are able to recover from this situation, if we do stream off and stream on.

Will get the expected frame rate as 120fps then.

Since there is no intervention of our customized driver here. We are suspecting issue with mxc core driver or in the i.MX6 hardware.

Kindly help us to resolve this.

Thanks in Advance!

Regards,

Rashmi

[gary_bisson]

Hi,

I know the driver itself doesn't do much, but the big difference can be the format used, is it YUV as Omnivision?

How do you test, what software do you use to dqueue the frames? Are you doing any CSC? If so, are you doing it in SW or HW?

Regards,

Gary

[rashmighoaelmur]

Hi Gary,

I have attached the application which I am using to capture the frames. And also I am not aware of CSC( please let me know). I want to remind again that I am able to get back the expected fps if I stream OFF and ON again. I am suspecting on MIPI HW could be the cause of issue. Since we are disabling/enabling the MIPI while restarting the stream again.

Kindly suggest me the solution on this to get it resolved ASAP.

Thanks in advance.

Regards,
Rashmi