How to smoothly display camera frames in LVGL?

[Vinos]

Hello, I am using the MIMXRT1176DMAA chip. The board is equipped with a MIPI camera and a MIPI LCD, and the LCD primarily used for displaying LVGL GUI.

Currently, I would like to display the camera's images on the screen while overlaying some LVGL labels on top of the camera feed. I have attempted to display the camera frames through an LVGL Canvas, but this approach introduced additional processing overhead in transferring camera frames to the Canvas, resulting in high CPU usage and choppy camera feed.

Could you please advise if there is a solution to this issue? For instance, can the Layer feature of LCDIFV2 be utilized to display content on different layers? I would greatly appreciate any suggestions you could provide.

[EdwinHz]

Hi @Vinos,

I am not aware of how you are approaching this application, so here is my recommendation on how to do so:

Refer to the example "SDK_MIMXRT1170-EVK\boards\evkmimxrt1170\driver_examples\csi\mipi_rgb". The captured video from camera needs to be composited with framebuffer of LVGL. There are two methods to do this:

1) Copying the camera image from camera output buffer to framebuffer of LVGL. This can be done with memcpy but it will result in bad performance, so I'd recommend using DMA instead.

2) Using another layer of LCDIFv2 as output buffer of camera directly, then LCDIFv2 h/w composites different layers on display without copy.

BR,
Edwin.

[Vinos]

Thank you for your suggestion. I have tried the second solution, but in LVGL, the general background is white and there are some labels on top. The actual result is that the white background completely covers the camera image except for the labels. How can I make these white backgrounds transparent?

[EdwinHz · Accepted solution]

Hi again @Vinos,

Please look into the following link: https://github.com/lvgl/lvgl/issues/262 I believe it will prove very useful for your inquiry.

Let me know if it helps!
Edwin.

[Vinos]

After trying, I found that the camera screen only works smoothly when the LVGL task is closed. Currently, Layer0 is the GUI and Layer1 is the camera. I would like to add another Layer2 when Layer0 is closed and Layer1 is opened. In the center of Layer3, there should be a hollow circle with transparency in all parts except for the arc.

Therefore, I used the ARGB4444 format and performed a simple test by drawing a square on the LCD(Layer0 enable and Layer1 disable;or Layer0 disable and Layer1 enable). However, the resulting image was not transparent. Whenever the LCDIFV2_SetLayerBlendConfig function is called, there is no image on Layer3 on the LCD. However, if this function is commented out, the LCD displays the expected non-transparent image.

Below is the code,reference the example " lcdifv2_embedded_alpha_cm7" in SDK

#define LCD_WIDTH  480
#define LCD_HEIGHT 640
#define CENTER_LAYER 2

#define DEMO_FB0_ADDR ((uint32_t)s_frameBuffer[0])
AT_NONCACHEABLE_SECTION_ALIGN(
    uint8_t s_frameBuffer[1][LCD_HEIGHT][LCD_WIDTH][2],
    32);

void DEMO_FillFrameBuffer(void)
{
    uint32_t x, y;

    for (y = 0; y < LCD_HEIGHT; y++)
    {
        for (x = 0; x < LCD_WIDTH; x++)
        {
        	s_frameBuffer[0][y][x][0] = 0x11;
        	s_frameBuffer[0][y][x][1] = 0x11;
        }
    }
}

void lcd_center_layer_init()
{
	DEMO_FillFrameBuffer();
	/* Layer 1: ARGB4444 */
	const lcdifv2_buffer_config_t fb1Config = {
		.strideBytes = (LCD_WIDTH * 2),
		.pixelFormat = kLCDIFV2_PixelFormatARGB4444,
	};

	const lcdifv2_blend_config_t blend1Config = {
		.alphaMode = kLCDIFV2_AlphaEmbedded,
	};

	LCDIFV2_SetLayerBufferConfig(LCDIFV2, CENTER_LAYER, &fb1Config);

	LCDIFV2_SetLayerSize(LCDIFV2, CENTER_LAYER, 240, 240);

	LCDIFV2_SetLayerOffset(LCDIFV2, CENTER_LAYER, 0, 0);
        /* comment or not */
	LCDIFV2_SetLayerBlendConfig(LCDIFV2, CENTER_LAYER, &blend1Config);

	LCDIFV2_SetLayerBufferAddr(LCDIFV2, CENTER_LAYER, DEMO_FB0_ADDR);

	LCDIFV2_EnableLayer(LCDIFV2, CENTER_LAYER, true);

	LCDIFV2_TriggerLayerShadowLoad(LCDIFV2, CENTER_LAYER);
}

Is there any error in my code? Thank you.

[Vinos]

Solved.