i.MX Processors中的主题 Re: eiQ - Distributed CNN inference

eiQ - Distributed CNN inference

nullbyte91 — Sun, 29 Aug 2021 06:11:15 GMT

Hi Team,

Is it possible to distribute the CNN inference engine to both CPU and GPU using ARMNN/TFLite?
We have our custom network that can run both CPU and GPU. We are looking for an option to distribute the inference engine to both CPU and GPU to get higher FPS.

Regards,
Jegathesan S

Re: eiQ - Distributed CNN inference

Yuri — Mon, 30 Aug 2021 04:44:13 GMT

@nullbyte91
Hello,

What i.MX device is used in the case?

Regards,
Yuri.

Re: eiQ - Distributed CNN inference

nullbyte91 — Mon, 30 Aug 2021 04:56:51 GMT

Hi @Yuri,

I'm using MX 8QuadMax.

Re: eiQ - Distributed CNN inference

Yuri — Tue, 31 Aug 2021 04:27:28 GMT

@nullbyte91
Hello,

from app team:

In short, there is no such option in eIQ that can support a CNN to run on both CPU and GPU at the same time.

However, it's possible from the application level to achieve that. As you can find in the eIQ demo, we can switch CPU/GPU inference in this application:

https://source.codeaurora.org/external/imxsupport/pyeiq/tree/eiq/apps/switch_video/switch_video.py?h=v3.0.0

Let's assume that GPU inference will take 0.1s to finish one frame and CPU inference will take 0.5s to finish one frame. We can give frame 6, 12 to CPU inference thread to calculate and frame 1-5, 7-11 to GPU inference thread to calculate. By the end of 1 second, we will have totally 12 frames finished inference. Comparing to use GPU only, it can raise FPS from 10 to 12.

But in reality, it's not easy to keep perfect sync between CPU and GPU. And usually GPU is much faster ,comparing to CPU, to run CNN inference. Therefore it may not be worthy to include CPU but adding this SW complexity.

Regards,
Yuri,