iMXRT1176 MIPI-CSI2 to UVC reducing the frame buffer's size

[Abisheikv2]

Hi,

I'm trying to develop a UVC device with RT1176 processor. While developing we saw that the CSI was using a frame buffer double the size than required. i.e, frame_height*frame_width*4 instead of the required 2. This was to use 24bit Parallel bus. We can't afford frame buffers of that size in our design. So we reduced the bus width to 16 bit and bits per pixel to 2 in CSI receiver's setting (the example source we are using is evkmimxrt1170_csi_mipi_yuv_cm7).

Bits per pixel of camera receiverChanging Databus width

When I observe the data, The frames are not proper and UV component is missing.

The one in left is how the actual data looks and the one in right is how we receive in RT1170.

Actual data of a Black frame     Data received in RT1170

 I tried changing the data format as well but no improvement. I'm not sure what we are missing here. Can someone pls shed some light?

Setting MIPI data format

[jeremyzhou]

Hi,
Thank you for your interest in NXP Semiconductor products and for the opportunity to serve you.
I think I need more information about your issue.
So I was wondering if you can introduce your testing environment, such as board, IDE, code, etc.
Furter, please list what modification you did with the evkmimxrt1170_csi_mipi_yuv_cm7 demo in detail.
Looking forward to your reply.
Have a great day.
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Abisheikv2]

Hey @jeremyzhou ,

I'm using the latest version of MCUExpresso IDE and SDK. The board is MIMXRT1170 EVK.

The changes done are as mentioned in the post. Changed the bus width of CSI to 16 , DEMO_CAMERA_BUFFER_BPP to 2, Set data type as YUV422 in CSICR18_MIPI_DATA_FORMAT fields. My question here is how to set the MIPI-CSI2 receiver to give out data in 16 bit to CSI receiver instead of 24 bit. or how can reduce the buffer size from what is used in sample code(2*1280*720*4(bpp, when the actual required is 2)). Thanks in advance.

[jeremyzhou]

Hi,
Thanks for your reply and clarification.
I‘m working on now and I'll reply you after I work it out.
Have a great day.
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Abisheikv2]

That would be great @jeremyzhou . Waiting for your reply.

[jeremyzhou]

Hi,
Sorry for replying late.
After changing the DEMO_CAMERA_BUFFER_BPP and dataBus as you mentioned, then try a lot of combinations of cameraConfig.pixelFormat and psBufferConfig.pixelFormat, I haven't gotten an ideal output (as the below figure shows).

I guess it is related to the PXP_SetCsc1Mode() function, however, it needs to spend much time to figure out the appropriate parameters.
So I'd like to suggest you refer workspace\evkmimxrt1170_csi_mipi_rgb_cm7, you can configure the CSI to receive the RGB565 format data, then transmit to the LCD directly without the PXP processing.
Maybe you can give it a try.

Have a great day.
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Abisheikv2]

Hi @jeremyzhou ,

Thanks for the effort. I'm stuck at the same situation. The PXP is not the problem here, if you can check the data(easier with memory monitor in IDE) before giving it to the PXP we can see that the UV component is missing. My assumption is we are setting the CSI receiver in 16 bit and MIPI-CSI2 is giving out data in 24 bit. My end application should send YUV422 over UVC so the RGB suggestion won't work out for me.

You can add the following line of code in static status_t OV5640_SetPixelFormat(camera_device_handle_t *handle, video_pixel_format_t pixelFormat) in fsl_ov5640.c to generate a static black frame and compare the data after our new setting and the default. 

OV5640_CHECK_RET(OV5640_WriteReg(handle, 0x503D, 0x83));

Can you please check any of your colleagues if they've seen such an issue. Or do they have a workaround for this.

Regards,

Abisheik

[jeremyzhou]

Hi,
Thanks for your reply.
Firstly, when observing the data whose start address is the cameraReceivedFrameAddr, I get a similar format as you show before, the two-pixel is identical to each other in the word unit.

Next, we haven't encountered a similar issue before.
Have a great day.
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Abisheikv2]

Hi,

 Thanks for the effort. Yes @jeremyzhou , That's the problem. What do you think is the issue here and how to resolve this? When can I expect a solution for this.

Update : 

"This project shows how to use the CSI with MIPI_CSI. In this example, the
camera device output pixel format is YUYV, the MIPI_CSI converts it to
YUV internally and sends to CSI. In other words, the CSI input data bus
width is 24-bit. The CSI saves the frame as 32-bit format XYUV8888. PXP
is used to convert the XYUV to RGB565 and show in the LCD panel."

I just saw this comment in the readme file. How to stop the MIPI_CSI from converting it to another format(YUV)? or change the output to our desired format?

[jeremyzhou]

Hi,
Hope it helps.
I've already escalated the issue to the software team, they may provide some insights into the issue.
So please be patient.
Have a great day.
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Abisheikv2]

Thanks @jeremyzhou . Awaiting an update.

[jeremyzhou]

Hi,
After digging deeper, I find that the CSI driver missing the configuration of the input MIPI data format, in another word, the below variable doesn't work at all.

    /* CSI input data bus is 24-bit, and save as XYUV8888.. */
    cameraConfig.pixelFormat                = kVIDEO_PixelFormatYUYV;//kVIDEO_PixelFormatXYUV;

Now, we're still working on it, and the AE team promises they will reply to me this week.
Have a great day.
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Abisheikv2]

Hey @jeremyzhou ,

Thanks for the effort. You're right that variable doesn't work. Even if we set the MIPI_DATA_FORMAT field in the CSI_CR18 register manually(verified with memory monitor) there's no difference in functionality. Take this also into consideration when you test. Awaiting AE team's reply.

[jeremyzhou]

Hi,

Thanks for your reply.
Now the AE team and I both are confused about the configuration of MIPI_DATA_FORMAT is ignored in the SDK library and AE already created a JIRAM ticket for the SDK team for further checking.
And I'll inform you ASAP if I get some feedback.
Have a great day.
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Abisheikv2]

@jeremyzhou  It's great that you guys are working on it. Do let me know when it's done

[jeremyzhou]

Hi，
VIDEO_MUX only can output 24bit format (as the below figure shows).
In another word, your purpose is impossible to make it.

Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Abisheikv2]

Hi, Can you please explain a little. So that means I can't avoid the zero padding. If that's the case does that mean I cannot use parallel cameras other than 24bit output? since both use the same VIDEO_MUX or is it specific to MIPI?.

I have a few queries then

1)Does that mean there is no way and get and use YUV422 data in i.MX RT1170?

2)If not, how to get back the proper data then? Is there a way to convert XYUV back to YUV422 or stop the conversion happening by default?

3)What is the purpose of MIPI DATA FORMAT field in CSICR18 then?

4)How does MIPI-CSI2 block differentiate between RGB and YUV in the example codes(to convert to XRGB8888 and XYUV8888)?

5)What other processor would you suggest for such an application?

Thanks

[jeremyzhou]

Hi,
Sorry for replying late.
1) Does that mean there is no way and get and use YUV422 data in i. MX RT1170?
-- It's Yes if you use the VIDEO_MUX to transfer data.
2) Is there a way to convert XYUV back to YUV422 or stop the conversion from happening by default?
-- I think you need to use the PXP or software algorithm to convert the XYUV to YUV422.
3) What is the purpose of the MIPI DATA FORMAT field in CSICR18 then?
-- It seems like the MIPI DATA FORMAT doesn't work as well it should, so in my opinion, it's a 'bug' for the module.
4) How does MIPI-CSI2 block differentiate between RGB and YUV in the example codes(to convert to XRGB8888 and XYUV8888)?
-- In my opinion, it's the same for MIPI-CSI2 to handle when receiving different formats from the camera, in another word, MIPI-CSI2 receive the coming byte data, then transfer it to VIDEO_MUX, it doesn't care the format of the coming byte data.
Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Abisheikv2]

Thanks for the confirmation @jeremyzhou. 

Are you sure we can use PXP for XYUV8888 to YUV422 conversion? because as per Reference Manual, PXP in RT1170 can only convert YUV to RGB.

[jeremyzhou]

Hi,
Thanks for your reply.
After confirming, the PXP doesn't support covert XYUV8888 to YUV422.

You can consider a software algorithm to do it just like the post shows.

So you 
Hope this is clear.

Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Abisheikv2]

Hi @jeremyzhou ,

Thanks for the suggestion. 

1. The video mux can give out only 24 bits data so the MIPI-CSI block converts any input data into 24 bit format. Is my understanding right?

2. I can see that MIPI-CSI is not just repacking the data but there is change in value. Can you get me the algorithm or the process happening in MIPI-CSI during conversion ?

3.Since MIPI-CSI repacks and converts every data. Is MJPEG possible? If so how to configure it?

Thanks in advance.