Streaming [IMX8MP] at maximal ISP rate (over 60 fps)

[mmarohnic]

Hello Community,

I would like to stream IMX sensor on maximal ISP rate (using functions like auto exposure, auto white balance etc)

In the i.MX 8M Plus Applications Processor Reference Manual (section 13.1.5 in the attachment) it is stated that the maximum frame rate is 120fps at 1080p.

However, using gstreamer we have only been able to achieve 60fps via:

gst-launch-1.0 -v v4l2src device=/dev/video2  ! "video/x-raw,format=YUY2,width=1920,height=1080,framerate=60/1" ! queue !  waylandsink

 although the maximal frame rate for this sensor is 90fps.

This restriction seems to be due to v42lsrc plugin restriction on framerate.

• is there any nxp sample application already available for this use case?
• If there is no such application I have noticed that there are some examples of  applications in isp-imx repository (video_test.cpp and vvext.cpp),  could you point us in the right direction/ file example on how to develop this application ?

Thanks for any suggestions!

[joanxie]

which ISP do you use? ISP0 or ISP1? they are not the same performance,

[mmarohnic]

I used CSI1 port so it should be ISP0

[joanxie]

refer to the AN 13736, Note: i.MX8MPlus can only support up to 3840 x 2160 at 30 fps.

https://www.nxp.com.cn/docs/en/application-note/AN13736.pdf

[mmarohnic]

I am working with 1080p which should have 120fps in the application note that you have shared.

[joanxie]

yes, I tested os08a20 with imx8mp, can reach 4k@30， which means you can get higher frame rate than 60 with 1080p resolution, this is my test result

[mmarohnic]

Thanks for the effort.  Can you tell me how did you test this?

I can reach 90fps (restriction on sensor)when using 

v4l2-ctl -d2 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --stream-mmap 

however, i can see in the logs (journalctl -u imx8-isp.service) that auto exposure and other ISP functions are not called.

However, when using gstreamer I can only get 60 fps. 

I thought that this restrictions are from gstreamer or some restrictions on NXP driver on frame rate while using ISP functions

[joanxie]

sorry for my late, I was handle urgent cases before, gstreamer doesn't show the real frame rate the ISP can reach, we have verified ISP0 can reach 4k@30 ， dual ISP can up to 1080p@60

[mmarohnic]

@joanxie 

I am still having trouble achieving this rate, could there be some other restrictions from waylandsink or display?

I tried moving to 1280 x 720 resolution.

Here the command

gst-launch-1.0 -v v4l2src device=/dev/video2 ! "video/x-raw,format=YUY2,width=1280,height=720,framerate=90/1" ! queue! fpsdisplaysink sync=false  video-sink=fakesink signal-fps-measurements=true text-overlay=false fps-update-interval=10000

gives 135 frame-rate (i do not know why it is not 90)

However, the command

gst-launch-1.0 -v v4l2src device=/dev/video2 ! "video/x-raw,format=YUY2,width=1280,height=720,framerate=90/1" ! queue ! fpsdisplaysink sync=false video-sink=waylandsink signal-fps-measurements=true text-overlay=false fps-update-interval=10000

gives only 60fps.

- Please note that we would need some way to test the isp rate via gstreamer or some other way that is reliable and where isp functions like white balance, auto exposures and others fro m isp are also used 

- Also we should be able to use gstreamer or some other way to visualize the stream with maximal frame rate. Please suggest some approach.

[joanxie]

I replied to you in your another post about g2d converter, you can test again, 

[mmarohnic · Accepted solution]

Streaming

Using the v4l2-ctl one can stream at maximal rate using ISP (lSP logs can be seen via journalctl -u imx8-isp.service, isp-imx should be build in debug mode)

v4l2-ctl -d2 --set-fmt-video=width=1920,height=1080,pixelformat=YUY2 --stream-mmap

Here I was able to get 90fps. Sensor is set to output RG10 format, so if one sets pixelformat=RG10, one can achieve maximal rate but the ISP functions will not be activated. Note that .max_fps field in isp-vvcam repo driver implementation must be set to 60fps.

Visualization

The visualization seems to be bounded by wayland, if i set imxvideoconvert_g2d plugin for example with 

gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,format=YUY2,width=1920,height=1080,framerate=70/1 ! queue ! imxvideoconvert_g2d ! queue ! video/x-raw,format=RGB16,width=320,height=180 ! queue ! waylandsink window-width=320 window-height=180 --no-position

the image has artifacts and it is not clear whether it is streaming at max rate.

Even the 

gst-launch-1.0 -v videotestsrc ! video/x-raw,format=YUY2,framerate=70/1,width=320,height=180 ! queue ! fpsdisplaysink sync=false video-sink="waylandsink window-width=320 window-height=180 --no-position" signal-fps-measurements=true text-overlay=false fps-update-interval=10000

achieves only 60fps so it seems that this limit is related to wayland.

[joanxie]

how about trying "fpsdisplaysink text-overlay=false signal-fps-measurements=true sync=false videosink="waylandsink sync=false "?

[mmarohnic]

Adding this part (videosink="waylandsink sync=false ") did not help with frame rate.

Some further notes:

• By using fpsdisplaysink with fakesink I can stream at 90 fps at 1280x720 resolution (but not on 1920x1080), the ISP functions work since when i cover the sensor the auto exposure algorithm is active.

gst-launch-1.0 -v v4l2src device=/dev/video2 ! "video/x-raw,format=YUY2,width=1280,height=720,framerate=90/1" ! queue ! fpsdisplaysink sync=false video-sink=fakesink signal-fps-measurements=true text-overlay=false fps-update-interval=10000

• This does not happen when I use stream without ISP where I can achieve 135 fps.

v4l2-ctl -d2 --set-fmt-video=width=1280,height=720,pixelformat=RG10 --stream-mmap

• Using videotestscr instead of v4l2src I can get 100fps via
  

gst-launch-1.0 -v videotestsrc ! video/x-raw,format=YUY2,framerate=120/1,width=1280,height=720 ! queue ! fpsdisplaysink sync=false video-sink=fakesink signal-fps-measurements=true text-overlay=false fps-update-interval=10000

So it might be that converting sensor format from RAW10 to YUY2 is where some time is lost so any idea how to optimize this would be great.

[mmarohnic]

This could be a wayland problem since

gst-launch-1.0 -v videotestsrc ! video/x-raw,format=YUY2,framerate=70/1,width=1280,height=720 ! queue ! fpsdisplaysink sync=false video-sink=waylandsink signal-fps-measurements=true text-overlay=false fps-update-interval=10000

also streams at 60fps. Any recommendation on this two problems would be most welcome.

- Please note that we would need some way to test the isp rate via gstreamer or some other way that is reliable and where isp functions like white balance, auto exposures and others fro m isp are also used 

- Also we should be able to use gstreamer or some other way to visualize the stream with maximal frame rate. Please suggest some approach.

[mmarohnic]

 Here is the missing attachment from the previous message