We are using google Coral SOM (based on the NXP imx8m) for an application using live camera neural-net analysis.
We face a performance issue to get gst image buffer in a python numpy array from a gstreamer stream.
Here is the test python source:
import time, os
import gi
from gi.repository import GLib, GObject, Gst, GstBase, Gtk
import numpy as np
gi.require_version("Gst", "1.0")
gi.require_version("GstBase", "1.0")
gi.require_version("Gtk", "3.0")
os.environ["GST_DEBUG"] = "*:2"
Gst.init(None)
pipeline = " ! ".join([
"v4l2src device=/dev/video0 io-mode=mmap",
"video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1",
"queue max-size-buffers=20 leaky=downstream",
"appsink name=appsink sync=false emit-signals=true max-buffers=10 drop=true"
])
pipeline = Gst.parse_launch(pipeline)
tic = time.perf_counter()
def on_new_sample(sink, preroll):
global tic
toc = time.perf_counter()
print("new sample", f"{(toc - tic)*1000}ms")
tic = toc
sample = sink.emit("pull-preroll" if preroll else "pull-sample")
gstbuffer = sample.get_buffer()
success, map_info = gstbuffer.map(Gst.MapFlags.READ)
if not success:
raise RuntimeError("Could not map buffer data!")
# numpy_frame = np.zeros((1080, 1920, 3), dtype=np.uint8)
numpy_frame = np.frombuffer(map_info.data, dtype=np.uint8)
gstbuffer.unmap(map_info)
return Gst.FlowReturn.OK
appsink = pipeline.get_by_name("appsink")
appsink.connect("new-sample", on_new_sample, False)
loop = GLib.MainLoop()
pipeline.set_state(Gst.State.PLAYING)
loop.run()
Each frame tooks about 37ms to process, causing frame skip.
When we do not read the frame buffer into a numpy array the frame is processed in less than 30ms (no frame skip). For that we replace:
numpy_frame = np.frombuffer(map_info.data, dtype=np.uint8)
by
numpy_frame = np.zeros((1920, 100, 2), dtype=np.uint8)
to simulate a same size numpy array writing.
We suspect an issue with the DMA. But it seems hard to debug.
We also tested with io-mode=mmap, dmabuf or auto (other are not supported by the camera) and we got same results.
Note that we do not have this issue when we replace the video source by a test one: "videotestsrc pattern=solid-color"
We use linux 4.14.98 from Mendel (4.14.98-imx, aarch64 GNU/Linux).
Our camera is an OV5640, working with the following drivers:
- drivers/media/platform/mxc/capture/mx6s_capture.c
- drivers/media/platform/mxc/capture/ov5640_mipi_v2.c
- linux-imx/drivers/media/platform/imx8/mxc-mipi-csi2_yav.c
