Data streaming issue and Frame Rate Problem in IMX8

[shrithi]

I have developed the custom driver for MIPI CSI2 FPGA IP support. MIPI CSI2 FPGA IP send the data to MIPI RX.

I have configured 4 data lanes for mipi and below configuration defined in device-tree

/* FPGA MIPI configuration */

&mipi_csi_0 { /* MIPI_CSI0: FPGA L MIPI TX */

#address-cells = <1>;

#size-cells = <0>;

status = "okay";

port@0 {

reg = <0>;

mipi_csi0_ep: endpoint {

remote-endpoint = <&fpga_L_mipi_ep>;

data-lanes = <1 2 3 4>;

bus-type = <4>;

};

};

};

&i2c_mipi_csi0 { /* MIPI_CSI0: FPGA L TX */

#address-cells = <1>;

#size-cells = <0>;

pinctrl-names = "default";

pinctrl-0 = <&pinctrl_i2c_mipi_csi0>;

clock-frequency = <400000>;

status = "okay";

fpga_L_mipi: fpga_L_mipi@3c {

compatible = "mipi,custom_mipi";

reg = <0x3c>;

mipi_csi;

status = "okay";

port {

fpga_L_mipi_ep: endpoint {

remote-endpoint = <&mipi_csi0_ep>;

data-lanes = <1 2 3 4>;

clocks-lanes = <0>;

};

};

};

};

When I enabled 4 lanes it will stuck during streaming

root@host:~# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! video/x-raw,format=RGB,width=1920,height=1080,framerate=10/1 ! filesink location=test2.raw

[  424.947801] bypass csc

[  424.950169] input fmt RGB4

[  424.952909] output fmt RGB3

Later I enabled 2 lines in device-tree. With 2lanes, some times we are able to get the data, but this is not stable. I used the FPGA MIPI TX IP which supports RGB24 format. we configured the FPGA MIPI TX at 125MHZ mipi clock for 1980x1080@30fps but we received less than 10fps in imx8.

root@host:~# v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RGB3 --stream-mmap

[ 140.157118] bypass csc

[ 140.159501] input fmt RGB4

[ 140.162205] output fmt RGB3

<<<<<<<<<< 8.73 fps

<<<<<<<< 7.53 fps

<<<<<< 7.53 fps

<<<< 6.66 fps

Can you please tell us what I am missing? How to configure clock in imx side?

Please help me to solve this problem.

Thank you.

[zhanghaijian]

hey ，shrithi！

 I have encountered the same problem with you，have you solved it right now？if you can，could you share your fpga mipi driver code，very appreciate that。

[shrithi]

I seen below hs setting value in drivers/staging/media/imx/imx8-mipi-csi2.c driver. How hs_setting value calculated based on width, height and framerate? Example: width=3056 height=3056 framerate=10 , in this case what is the hs_setting value?

static struct mxc_hs_info hs_setting[] = {
{2592, 1944, 30, 0x0B},
{2592, 1944, 15, 0x10},

{1920, 1080, 30, 0x0B},
{1920, 1080, 15, 0x10},
{1920, 1080, 25, 0x0C},

{1280, 720, 30, 0x11},
{1280, 720, 15, 0x16},

{1024, 768, 30, 0x11},
{1024, 768, 15, 0x23},

{720, 576, 30, 0x1E},
{720, 576, 15, 0x23},

{720, 480, 30, 0x1E},
{720, 480, 15, 0x23},

{640, 480, 30, 0x1E},
{640, 480, 15, 0x23},

{320, 240, 30, 0x1E},
{320, 240, 15, 0x23},

{176, 144, 30, 0x1E},
{176, 144, 15, 0x23},
};

[shrithi]

yes. I am using i.MX8 QM processor.

Can you explain about clock configuration?  How to calculate clk_core and clk_esc?    

clock-names = "clk_core", "clk_esc", "clk_pxl";

When I set clock configuration as below I am getting data with 4 data lanes and fps it showing 180fps.

assigned-clock-rates = <266000000>, <333000000>, <66000000>

Device-tree mentioned as below

/* FPGA MIPI configuration */

&mipi_csi_0 { /* MIPI_CSI0: FPGA L MIPI TX */

#address-cells = <1>;

#size-cells = <0>;

status = "okay";

port@0 {

reg = <0>;

mipi_csi0_ep: endpoint {

remote-endpoint = <&fpga_L_mipi_ep>;

data-lanes = <1 2 3 4>;

bus-type = <4>;

};

};

};

&i2c_mipi_csi0 { /* MIPI_CSI0: FPGA L TX */

#address-cells = <1>;

#size-cells = <0>;

pinctrl-names = "default";

pinctrl-0 = <&pinctrl_i2c_mipi_csi0>;

clock-frequency = <400000>;

status = "okay";

fpga_L_mipi: fpga_L_mipi@3c {

compatible = "mipi,custom_mipi";

reg = <0x3c>;

mipi_csi;

status = "okay";

port {

fpga_L_mipi_ep: endpoint {

remote-endpoint = <&mipi_csi0_ep>;

data-lanes = <1 2 3 4>;

clocks-lanes = <0>;

};

};

};

};

[joanxie]

why I ask for other dts files, I need to know how you set clock for mipi csi, I don't know what you change, but for imx8qm, I checked current bsp already supports 4 data lane,

https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/arch/arm64/boot/dts/freescale/imx8qm-mek.dts

&mipi_csi_0 {
#address-cells = <1>;
#size-cells = <0>;
virtual-channel;
status = "okay";

/* Camera 0 MIPI CSI-2 (CSIS0) */
port@0 {
reg = <0>;
mipi_csi0_ep: endpoint {
remote-endpoint = <&max9286_0_ep>;
data-lanes = <1 2 3 4>;
};
};
};

for imx8qm mipi csi clock,  only set core clock and esc clock, since current bsp can support 4 data lane, you can refer to the setting as below

mipi_csi_0: csi@58227000 {
compatible = "fsl,mxc-mipi-csi2";
reg = <0x58227000 0x1000>,
<0x58221000 0x1000>;
clocks = <&csi0_core_lpcg 0>,
<&csi0_esc_lpcg 0>,
<&csi0_pxl_lpcg 0>;
clock-names = "clk_core", "clk_esc", "clk_pxl";
assigned-clocks = <&csi0_core_lpcg 0>,
<&csi0_esc_lpcg 0>;
assigned-clock-rates = <360000000>, <72000000>;
power-domains = <&pd IMX_SC_R_CSI_0>, <&pd IMX_SC_R_ISI_CH0>;
power-domain-names = "pd_csi", "pd_isi_ch0";
status = "disabled";
};

[shrithi]

With this assigned-clock-rates = <360000000>, <72000000>; I am not able to stream the data. So I changed to  assigned-clock-rates = <200000000>, <333000000>; Is this proper?

Based on documents I assigned core clock as 200MHz. But I am not sure about esc clock.

[joanxie]

escape clock is for low power, normally customer doesn't need to change it, do you mean you can boot up 2 data lane with assigned-clock-rates = <360000000>, <72000000>; but failed with 4 data lane, then you change it to <200000000>, <333000000>; the 4 data lane setting works, right? you don't change anything else?

[shrithi]

We are able to get the data using 2 lanes with <360000000>, <72000000> (inconsistent). With <200000000>, <333000000> setting I am able to stream the data with 4 lanes. No other software changes done, TX mipi pixel clock changed to 74.25MHz, output clock is 222.75MHz and data rate is 445.5. Resolution is 1920*1080@30fps. 

[joanxie]

current imx8qm bsp already supports 4 data lane, I shared the default code with you before, you should check your own driver, total data rate= pixel clock * Bits-per-pixel, if your pixel clock is 74.25, the data rate shouldn't be 445.5, you need set the correct clock for the MIPI CSI2, then MIPI CSI2 would change it's register to meet your request, so your root cause isn't escape clock in the dts, should check your own driver if your driver set correct clock or not, for example, current bsp supports 4 data lane for max9286, in the max9286 driver, you can find

  /*
   * Pass mipi phy clock rate Mbps
   * fcsi2 = PCLk * WIDTH * CHANNELS / LANES
   * fsci2 = 72MPCLK * 8 bit * 4 channels / 4 lanes
   */
  max9286_data->format.reserved[0] = 72 * 8;

the max9286 uses 288M mipi clock, so pls check your FPGA driver if set this correct or not,

[shrithi]

We used below equations to calculate the clock and data rate
Pixel Clock Hz = HTOT*VTOT*FPS
Bandwidth bps = Pixel Clock * Bits Per Pixel
Data Rate Per Lane bps = Bandwidth / number of data lanes
MIPI D − PHY Clock Rate Hz = Data Rate Per Lane / 2

Pixel Clock = 2200×1125×30 = 74250000 Hz
Bandwidth = 74250000 * 24 = 1782000000 bps
Data Rate Per Lane = 1782000000/4 = 445500000 bps = 445.5 Mbps
MIPI Bit clock = 445500000 / 2 = 222750000 = 222.75 MHz

[joanxie]

bit_clk = Pixel clock * Bits per pixel / Number of lanes=74.25*24/4=445.5

[shrithi]

I have gone through the below mentioned document

https://www.nxp.com/docs/en/application-note/AN13573.pdf

Can you please reconfirm the calculation once?

[joanxie]

the equation 16 is for mipi clock, bit clock is another clock, you mentions mipi bit clock, I thought you mean bit clock

[shrithi]

How 1 pixel per clock and 4 pixel per clock vary in below equation for RGB888 format of data?

Bandwidth bps = Pixel Clock * Bits Per Pixel

Can you explain this? 

[joanxie]

for example, rgb888 send over 8bits, then needs 3 cycles/pixel, the depth is 24bpp

[shrithi]

How to calculate esc clock? I am suspecting clock configured by imx8 also. What maximum escape clock supported by imx8?

Currently I assigned 333MHz clock for esc clock.

When I stream data with 30fps, I am getting framerates in the range of 15 to 20.

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RGB3 --set-parm=30 --stream-mmap
Frame rate set to 30.000 fps
[ 201.417005] bypass csc
[ 201.419426] input fmt RGB4
[ 201.422509] output fmt RGB3
<<<<<<<<<<<<<<<<<<<<<<< 21.37 fps
<<<<<<<<<<<<<<<<<<<<< 21.29 fps
<<<<<<<<<<<<<<<<< 19.50 fps
<<<<<<<<<<<<<<<<<< 19.60 fps
<<<<<<<<<<<<<<<<<<<<<<<< 20.03 fps
<<<<<<<<<<<<<<<<<< 19.76 fps
<<<<<<<<<<<<<<<<<<<<< 19.72 fps
<<<<<<<<<<<<<<<<< 19.00 fps

The data will not be streamed if I set fps to 15fps. It will hang while streaming the data.
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RGB3 --set-parm=15 --stream-mmap
Frame rate set to 15.000 fps
[ 292.933426] bypass csc
[ 292.935842] input fmt RGB4
[ 292.938783] output fmt RGB3

By setting it to 60fps, I will achieve a framerate of 50.
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RGB3 --set-parm=60 --stream-mmap
Frame rate set to 60.000 fps
[ 341.047806] bypass csc
[ 341.050182] input fmt RGB4
[ 341.052936] output fmt RGB3
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 50.51 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 50.51 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 50.51 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 50.51 fps

The data on the TX side is set to 1920x1080 at 15 frames per second.
What caused the FPS to change every time? Does the hs_settle value cause the problem? The following values were given as hs_settle values in the driver.

drivers/staging/media/imx/imx8-mipi-csi2.c
static u8 rxhs_settle[3] = {0xD, 0xA, 0x7};
static struct mxc_hs_info hs_setting[] = {
{2592, 1944, 30, 0x0B},
{2592, 1944, 15, 0x10},

{1920, 1080, 30, 0x0B},
{1920, 1080, 15, 0x10},

{1280, 720, 30, 0x11},
{1280, 720, 15, 0x16},

{1024, 768, 30, 0x11},
{1024, 768, 15, 0x23},

{720, 576, 30, 0x1E},
{720, 576, 15, 0x23},

{720, 480, 30, 0x1E},
{720, 480, 15, 0x23},

{640, 480, 30, 0x1E},
{640, 480, 15, 0x23},

{320, 240, 30, 0x1E},
{320, 240, 15, 0x23},

{176, 144, 30, 0x1E},
{176, 144, 15, 0x23},
};

Can you please provide the more information?

[joanxie · Accepted solution]

you don't need change escape clock, as I mentioned before, you need check your FPGA side clock, if could, pls measure the mipi  clock  by oscilloscope to check if correct or not I also attached the mipi guide to reference

[shrithi]

After correcting the FPGA clock, I am able to get the data with the following configuration.

assigned-clock-rates = <360000000>, <72000000>;

Can the resolution of 3056x3056 be supported by imx8 mipi-csi2 and ISI?

[shrithi]

When we measured clock using CRO(oscilloscope) and clock is 500MHz for FPGA with double data rate.

[shrithi]

In our case FPGA is the TX for MIPI, When we did probe clock using oscilloscope, the clock observed is 500MHz.

The below clock configuration done in device-tree
 assigned-clocks = <&csi0_core_lpcg 0>,
                     <&csi0_esc_lpcg 0>;
 assigned-clock-rates = <200000000>, <360000000>;

Tested with below configuration
number of data lanes - 4
resolution - 1920x1080@30fps
Data format - RGB888
1 pixel per clock
double data rate

FPGA Mipi TX IP configured with 1 pixel per clock will operate with a esc clock frequency of 180MHz and above.
FPGA Mipi TX IP configuration with 4 pixels per clock will operate with a esc clock frequency of 360MHz.
Can you provide an explanation for how this is changing?

We tested ov5640 camera with default configuration using 2 lanes and its working. When we did probe clock using oscilloscope, the clock observed is 24MHz. Here everything is working fine.

One more observation is streaming is happening with 15fps hs_setting value, i.e, 0x10. If I give below command the it is showing 30fps
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RGB3 --set-parm=15 --stream-mmap
Frame rate set to 15.000 fps
[ 201.417005] bypass csc
[ 201.419426] input fmt RGB4
[ 201.422509] output fmt RGB3
<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<<<<<<<<<<<< 30.03 fps

I have confusion about esc clock and hs_setting value. So can you please provide us more information?
The severity of this case is high. Can you provide me with the solution as soon as possible?

[joanxie]

do you use imx8qm? and pls share dtsi files your current dts includes, I need check how you set mipi csi