How to use VPU to decode the image in google-coral

[Shao]

I use google-coral based on Mendel-OS and USB camera to capture image and inference segment .
Although the camera can support MJPEG-1280x720-30FPS, I only get 15FPS and very high CPU usage.

What I try

First, I try to use opencv library (VideoCapture ) to control my USB camera.

cap.open(/dev/video1, CAP_V4L);

Opencv will auto-covert frame to BGR format. I think the convert process is dependent on CPU and cause very high cpu usage.

I also try using opencv gstream pipline. 

cap = VideoCapture("v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegdec ! video/x-raw ! videoconvert ! appsink");

The gstreamer pipline method takes more cpu usage.

So I try to use v4l2 direct to capture the raw frame(MJPEG).
At the present stage, I use opencv to decode the frame to RGB to verify the capture frame correct or not.
I want to transfer MJPEG to YUYV and then I can extract Y-channnel for inference.
By the way, I know some camera support YUYV format but the FPS for YUYV format is slower than MJPEG in usual condition.

I survey the NXP official website. It provide several test and example code for YOCTO project. But I can't using it for google-coral based on Mendel-OS.

I also try to build the libimxvpuapi. Unfortunately I failed. I didn't find "usr/include/imx/mxcfb.h" in my SOM, which need to set for SYSROOT.

How do I using vpu on the google-coral to decode the frame?

[jimmychan]

We have different version of Linux Yocto BSP. You can download the BSP documentations from here:

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applicat...

e.g. Linux 5.10.52_2.1.0

Please read the Chapter 7 in the i.MX Linux Release Notes. You can find the gstreamer plugins for the corresponding i.MX processor in Table 15.

And please read the Chapter 7 in i.MX Linux User's Guide about the gstreamer use cases/examples.

[Shao]

Hi,
My board is google-coral Dev Board.
I follow the google-coral official website to build the Mendel Linux.

And I read the i.MX_VPU_Application_Programming_Interface_Linux_Reference_Manual.
In the section1.2, It mention the hantrodec.h and hx280enc.h.
But these file don't exist in my coral-board rootfs.
Should I change any kernel config option?

I also find the libhantro_h1.so.1 and libhantro.so.1 in /usr/lib in my coral board.
Could I use these two file to achieve VPU decoder? What is the related header file ?

[Shao]

Hi, 

The google-coral kernel version is 4.14-imx, i don't know which version is map.
I read the Linux 4.14.98_2.3.0 release note.
I can use command below but still very high cpu usage. 

cap = VideoCapture("v4l2src device=/dev/video1 ! image/jpeg,width=1280,height=720,framerate=30/1 ! vpudec ! videoconvert ! appsink");

Do I miss some information?

[jimmychan]

Board Support Package(BSP) is board dependency. There is different board dts/dtsi in the linux source code. Please contact your board's support person to provide you the BSP or the dts file so you can build the linux kernel for your board.