Clocks issue when capturing 4K@30 RAW10 in i.MX8M

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Clocks issue when capturing 4K@30 RAW10 in i.MX8M

3,858 Views
jafet_chaves
Contributor I

Hi,

We've been working with an IMX334 image sensor on the dart-mx8m development kit. We are trying to capture a 4K@30 fps streaming. We configure the image sensor to use 4 lanes (891 Mbps/lane) and RAW10 but so far we only get distorted images (see images below). Frame rate is not correct either.

canvas2.png  capture_imx334.png

We've already validated that is possible to capture RAW10 images with two other sensors, the OV5640 and the AR0144 (750 Mbps/lane).

From these tests, we have observed that the main difference of the IMX334 with respect to the other sensors already validated, is the data rate and the number of CSI-2 lanes.

In the iMX8M documentation ("i.MX 8M Dual/8M QuadLite/8M Quad Applications Processors Reference Manual") one can find the following information about the MIPI Host Controller:

  - clk: RX Controller Core Clock input. This
         clock must be exactly equal to or faster
         than the receive byteclock,
         RxByteClkHS_ln0, from the RX DPHY.

  - clk_ui: ser interface clock. The frequency of
            clk_ui must be such that the data
            received on the data_out output is
            greater than or equal to the total
            bandwidth of the physical MIPI interface.
            Clk_ui has no relationship requirement
            with regards to ‘clk’ other than the
            bandwidth requirement mentioned
            previously.

   - esc_clk: Rx Escape Clock. This must be the
                   same escape clock that the RX DPHY
                   receives.

As an example, if RGB888 data is being received, the
user interface data width will be 24 bits wide (1 pixel per clk_ui rising edge and each
pixel is 24 bits). As an example, let the MIPI interface run at 1000Mbps with a single
lane. The data rate into the CSI-2 RX controller will be 125MBps so clk must run at
125MHz or higher. Since the user interface is 3 bytes wide, clk_ui must run at 125MHz/3
or higher to keep up with the incoming data.

Based on this, I assume clk must be >= byte rate and clk_ui must be >= pixel rate.

In our case, we have 891Mbps/lane which gives us: 445.4 MB/s and 356.4 MP/s so I'm assuming one should be clk and the other one clk_ui.

From the driver perspective, we have access to 3 clocks that are called: CORE, PHY and ESC. There is no
documentation about the relationship between the clk/ckl_ui in RM and CORE/PHY in driver but my guess is
that clk is what in the device tree is called CORE clock and clk_ui is what is called PHY. Based on this I tried changing the values of CORE and PHY clocks in iMX8 to values greater than the ones in my calculations but I have had no luck. With different clock values, I can see how the capture framerate changes but the captured data is never right.

Also, the documentation says that the clocks have to be exactly equal or greater than a certain value but in
practice this doesn't seem to be true as if I set the clocks too high, the capture stops working at all so it
looks like we have to use a value close to the actual capture data rate but the problem is we have no way of
generating the exact values of the calculations. If we have, for instance, a source clock of 1000MHz for CORE and
PHY clock, we can only get 1000, 500, 333, 250, 200, ... MHz clocks (we can only divide the clock by integer numbers).

So the questions would be:

*What is the relation between clk and clk_ui with the properties CORE/PHY in the device tree (arch/arm64/boot/dts/freescale/fsl-imx8mq.dtsi)?

*How to set clk, clk_ui, esc_clk properly, how it is related to what the image sensor outputs?

Thank you in advance for your answers!

--

Jafet Chaves
Embedded Software Engineer
RidgeRun | https://www.ridgerun.com/

9 Replies

2,870 Views
abdu2
Contributor I

Hi Jafet, 

We are also seeing a similar issue in iMX8M using a 4 lanes, 720Mbps configuration for 720p30, 1080p30, 4Kp30.

 

Have you got this solved @jafet_chaves ?

Appreciate if you could give some suggestion  @felix_ye  @Bio_TICFSL @edison_fernande 

Regards

Abdu Samad

 

0 Kudos
Reply

3,206 Views
jafet_chaves
Contributor I

Hi,

Just wanted to give you an update on this issue, we've been able to capture 3672x2180@30fps, 4 lanes, RAW10 successfully but the expected resolution is 4K@30fps, this could be related to sensor configuration (we're figuring it out internally). But if you have any input about the mismatch in the resolutions (from the i.MX8M perspective) please let us know.

Additionally, we had to apply some changes in the i.MX8M kernel (some of them were taken from newer kernel version as suggested by Bio_TICFSL others we're already added by us):

*CSI_CSICR18 bit 4 (BASEADDR_SWITCH_EN) was set to 1 (drivers/media/platform/mxc/capture/mx6s_capture.c)

*CSI_CSICR18 bits 19-18 (BASEADDR_SWITCH_EN) were set to 10 (drivers/media/platform/mxc/capture/mx6s_capture.c)

* We set assigned-clock-rates = <266000000>, <333000000>, <66000000> in the device tree (arch/arm64/boot/dts/freescale/fsl-imx8mq.dtsi).

--

Jafet Chaves

Embedded Software Engineer

RidgeRun | https://www.ridgerun.com/

0 Kudos
Reply

3,206 Views
edison_fernande
Contributor III

Hi Bio_TICFSL

By using the suggested clocking configuration (266, 333, 266) MHz we have been able to get the best results out of the imx334, however the capture is still not correct.

The sensor is configured to generate 3864x2180@30 frames with a data rate of 891Mbps/lane using 4lanes and this seems to be in the supported range of the iMX8 (1.5Gbps/lane).

If we use the recommended configuration, we can capture perfect frames if we configure the imx8 to receive a frame of 3672x2180 pixels (when the sensor is sending 3864x2180) .

If we configure the sensor to generate 3840x2164, we can capture "good" frames configuring the iMX8 to receive a frame of 3650x2164 and if we configure the sensor to generate frames of 3648x2164 we can see the frames using a width of 3466 so it looks like we are always capturing a width under the expected value by a fixed rate.

The sensor always  gives a constant data rate so in all these tests we have been receiving 891Mbps/lane.

I also run a second test to see if I was able to receive the video at the expected resolution.

I set the CORE clock to 266MHz, and sourced the PHY clock from the AUDIO PLL2 clock to be able to more precisely control the PHY clock frequency. With this configuration, I got the best results using a PHY clock of 351.4485MHz, with the sensor configured at 3848x2164 I was able to receive video configuring the iMX8 at 3460x2164, if I increase or decrease the PHY clock by really small values (350 or 352 MHz for instance) the framerate goes down and the captured frames no longer make sense.

Finally, is good to mention that the cases where I receive "good" video, if I capture several consecutive frames, I can see the video "shifting" to the left.

Do you know of any limitation of the iMX8 to capture at 891Mbps/lane using 4 lanes?

Do you know if the PHY clock frequency has to be of certain and very precise frequency? From my tests this seems to be the case but I can't find a relation between the data rate and the clocks.

Do you have any clue on what could be causing the iMX8 to be always capturing frames with a width smaller than the one the sensor is sending?

Any help will be really appreciated.

Edison

0 Kudos
Reply

3,206 Views
f_siebel
Contributor III

Maybe similar issue like on the I.MX8QM ? I'm not sure how much difference is between the IP cores inside the chip family.

Look at Errata https://www.nxp.com/docs/en/errata/IMX8_1N94W.pdf 

ERR05006 and the other ISI stuff.

0 Kudos
Reply

3,206 Views
Bio_TICFSL
NXP TechSupport
NXP TechSupport

Hello ye,

    Please check the table from 8M RM  for relation between clk and clk_ui with the properties CORE/PHY.A84A2B0.PNG

 

B.R,

0 Kudos
Reply

3,206 Views
felix_ye
Contributor II

Hi Bio_TICFSL,

With your clock configuration:

    assigned-clock-rates = <266000000>, <333000000>, <66000000>

 

Now we can capture the image (1280 * 720,  30 fps, RAW12)  with MIPI CSI 1-lane, 2-lane, 3-lane or 4-lane.

 

Thanks for your support !

Felix

0 Kudos
Reply

3,206 Views
felix_ye
Contributor II

Hi Jafet,

I also confused about the i.MX8MQ MIPI CSI clock configuration.

But I don't think "clk_ui is what is called PHY" as you said.

Just see this clock block diagram in i.MX8MQ RM P4994.

CSI.png

Felix

0 Kudos
Reply

3,206 Views
Bio_TICFSL
NXP TechSupport
NXP TechSupport

Hello Ye,

Could you try on 4.14.98_2.0.0_ga version? We have validate the function on  4.14.98_2.0.0_ga GA version.

      Please tell customer to set clk:

     assigned-clock-rates = <266000000>, <333000000>, <66000000>

     As for relation between clk and clk_ui with the properties CORE/PHY, I will consult IC team then give you a reply.

 

Regards

3,206 Views
felix_ye
Contributor II

Hi Bio_TICFSL,

Thanks a lot for your reply !

What we actually need is how to calculate these clock rates, depending on the data rates ( image width * image height * fps * bit-per-pixel ) and MIPI CSI data-lanes ( Now it can work only when data-lanes = 1).

By the way, we are now working on 4.14.98_2.0.0_ga version. Which data rates you have validated ?

Felix

0 Kudos
Reply