i.mx8M Mini OV5640 fps limitation

[raptor8]

Hello everyone,

I bought the imx8mmevk with the i.MX8M Mini SoC with the MINISASTOCSI accessory, flashed Linux (L4.14.78_1.0.0_ga_images_MX8MMEVK) and managed to stream from the OV5640 with gstreamer successfully.

But I was expecting to be able to use this sensor at 60fps as the datasheet of the sensor specify and the video encoders VP8/H264 are capable of outputting at least 720p60.

I have done a quick search for the cause of this but I see that's it's using 2 mipi_csi lanes

root@imx8mmevk:~# dmesg | grep "mipi"
[    1.257708] adv7511 1-003d: Probe failed. Remote port '/mipi_dsi@32E10000' disabled
[    1.549094] mxc_mipi-csi 32e30000.mipi_csi: 32e30000.mipi_csi supply mipi-phy not found, using dummy regulator
[    1.559458] mxc_mipi-csi 32e30000.mipi_csi: mipi csi v4l2 device registered
[    1.566436] CSI: Registered sensor subdevice: mxc_mipi-csi.0
[    1.572121] mxc_mipi-csi 32e30000.mipi_csi: lanes: 2, hs_settle: 13, clk_settle: 2, wclk: 1, freq: 333000000
[    1.582387] ov5640_mipi 2-003c: No sensor reset pin available
[    1.588182] ov5640_mipi 2-003c: 2-003c supply DOVDD not found, using dummy regulator
[    1.595981] ov5640_mipi 2-003c: 2-003c supply DVDD not found, using dummy regulator
[    1.603677] ov5640_mipi 2-003c: 2-003c supply AVDD not found, using dummy regulator
[    2.786869] mxc_mipi-csi 32e30000.mipi_csi: Registered sensor subdevice: ov5640_mipi 2-003c
[    2.797202] ov5640_mipi 2-003c: Camera is found

And the capabilities are tied to 30 fps max with V4L2

root@imx8mmevk:~# v4l2-ctl -D -d /dev/video0 --list-formats-ext
Driver Info (not using libv4l2):
        Driver name   : mx6s-csi
        Card type     : i.MX6S_CSI
        Bus info      : platform:32e20000.csi1_bridge
        Driver version: 4.14.78
        Capabilities  : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04200001
                Video Capture

                Streaming
                Extended Pix Format
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'YUYV'
        Name        : YUYV 4:2:2
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 720x480
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 2592x1944
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 0x0

Now I would like to know what is possible to do to get to 60fps with this sensor and from where the limitation is coming.

Is it from some registers in the driver? The clock speed used? Do I need to recompile something with other parameters?

I was also planning on using other sensors in the future, on the MIPI I/F or any other serial or parallel I/F. Are there others drivers included in the default linux image? How is it difficult to implement a new sensor?

I see many sensors listed on the official linux github that can interest me.

linux/drivers/media/i2c at master · torvalds/linux · GitHub 

Thank you for your time.

[andreasgustafss]

The ov5640 have a driver upstream that supports all camera modes, including 720p 60fps.

I would try to get the upstream driver to work on the kernel you're using.

It can be found here:

linux/ov5640.c at master · torvalds/linux · GitHub 

Both drivers utilize the v4l2-subdev system so a drop in replacement should be possible.

Good luck!

[raptor8 · Accepted solution]

I'm updating my question now as I'm rebuilding the Linux kernel now...
The driver is locked to 30 fps in multiple places like here:
https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/media/i2c/ov5640.c?h=imx_4.14.78_1...
https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/media/platform/mxc/capture/ov5640_...

And many other references to the framerate here

./drivers/media/i2c/ov5640.c
./drivers/media/platform/mxc/capture/ov5640_v2.c
./drivers/media/platform/mxc/capture/ov5640.c
./drivers/media/platform/mxc/capture/ov5640_mipi.c
./drivers/media/platform/mxc/capture/ov5640_mipi_v2.c
./drivers/media/platform/imx8/ov5640_mipi_v3.c
./drivers/media/platform/imx8/ov5640_v3.c

And the second part of my question was adding a new camera sensor, I suggest taking a look at the i.MX_BSP_Porting_Guide at chapter 13.2:

"13.2 Adding support for a new CMOS camera sensor"

Which explains how to add a driver to the make menuconfig to compile it.

[michael_glembot]

Could you please provice the i.MX_BSP_Porting_Guide? I only found https://www.nxp.com/docs/en/user-guide/i.MX_BSP_Porting_Guide_Linux.pdf , which does not support imx8mm. The registers are different.

[raptor8]

You should be able to download it from here: https://www.nxp.com/webapp/Download?colCode=imx-yocto-L4.14.98_2.0.0_ga 
It's available in the i.MX Software | NXP  under Documentation > Linux
(also if you can't login onto nxp website like me, try with another browser)

[diegoadrian]

Hello,

Unfortunately, the maximum fps that the board supports is 30fps. Probably you can make support 60 fps by changing the gstreamer v4l2src plug-in. However, I cannot guarantee anything, since the fps output depends on many things. I apologize if this could be a problem for you.

Best regards,

Diego.

[Amal_Antony3331]

Hi @diegoadrian 

I'm trying to integrate imx219 sensor with iMX8M Plus. I'm able to get the stream but the FPS that I'm getting is only 15 fps @480p. If  I decrease it to 144p, fps increases to 20. 

root@imx8mpevk:/opt/imx8-isp/bin# v4l2-ctl -D -d /dev/video2 --list-formats-ext
Driver Info:
        Driver name      : viv_v4l2_device
        Card type        : VIV
        Bus info         : platform:viv0
        Driver version   : 5.10.35
        Capabilities     : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : vvcam-video
        Model            : viv_media
        Serial           : 
        Bus info         : 
        Media version    : 5.10.35
        Hardware revision: 0x00000000 (0)
        Driver version   : 5.10.35
Interface Info:
        ID               : 0x03000003
        Type             : V4L Video
Entity Info:
        ID               : 0x00000001 (1)
        Name             : viv_v4l20
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Sink, Must Connect
          Link 0x0200001f: from remote pad 0x100000a of entity 'vvcam-dwe.0': Data, Enabled
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'YUYV' (YUYV 4:2:2)
                Size: Stepwise 176x144 - 1920x1080 with step 16/8
        [1]: 'NV12' (Y/CbCr 4:2:0)
                Size: Stepwise 176x144 - 1920x1080 with step 16/8
        [2]: 'NV16' (Y/CbCr 4:2:2)
                Size: Stepwise 176x144 - 1920x1080 with step 16/8
        [3]: 'GB10' (10-bit Bayer GBGB/RGRG)
                Size: Stepwise 176x144 - 1920x1080 with step 16/8

Gstreamer command that I have used is given below.

Server command:

gst-launch-1.0 v4l2src device=/dev/video2 ! videorate drop-only=true ! video/x-raw,width=640,height=480 ! jpegenc ! rtpjpegpay ! udpsink host=<client_ip> port=1234

Client command

gst-launch-1.0 udpsrc port=1234 caps="application/x-rtp" ! rtpjpegdepay ! jpegdec ! videoconvert ! fpsdisplaysink

Above given is the function to set the FPS.Is it due to whether I have missed any register to configure ?

Any help is appreciated.

Regards

Charles

[Cappelletto]

I have a similar situation with the imx8M-mini evk + ov5640 running at ~18fps instead of the expected 30 fps when using 640x480 resolution.

The same image runs smoothly at 29~30 fps when deployed in a imx8M Nano.