imx8: pyeiq facial detection & face eye detection not working with live camera(v4l2, ov5640)

[vsuneja63]

pyeiq facial detection & face eye detection working fine with USB cam but same is not working on nxp ov5640 camera on imx8mq evk (5.4.24 bsp) target. Streaming is fine but false detection happening with wrong marking and that is also random. Its fps is around 17. What could be the issue for such behavior?

[kenizgandhi]

Hello @Alifer_Moraes , @manish_bajaj , 

I want to run some object detection applications on imx8mpevk board.

Linux version 5.10.9-1.0.0+g32513c25d8c7 (oe-user@oe-host) (aarc1
Machine model: NXP i.MX8MPlus EVK board

I have connected mipi csi2 to the board and able to capture video using mipi ov5640 camera using gst-launch-1.0 v4l2src device=/dev/video1 ! autovideosink and display it on HDMI monitor. 

But when I try to run object detection demo application using pyeiq --run object_detection_tflite --video_src=/dev/video1 . 

I am getting error like :

[16732.920316] mxc-mipi-csi2.0: unsupported csi-sam command -1068476902.
Using /dev/video1 as video device
Your video device could not be initialized. Exiting...

Can you please help me solve this error. 

Thank You. 

Kind Regards,

Keniz

[manish_bajaj]

vsuneja63@gmail.com‌,

I will suggest to first check on what port ov5640 is connected and are your able to get it running using standard gstreamer command, and then use that port. I don't think /dev/video2 is correct. Their are 2 MIPI CSI port in 8MQ and it should be either /dev/video0 or /dev/video1.

-Manish

[vsuneja63]

Hi Manish,

Thanks!

ov5640 cam is on /dev/video0 only, /dev/video2 is for USB webcam. As i mentioned recent error is wrt recorded video (recorded by ov5640 cam), not with live stream. Not sure why its showing /de/video2 but i am passing recorded video only as input. On the connected CSI port i could see live streaming happening through ov5640 camera(with pyeiq app & gstreamer command), face-eye false detection happening randomly. Again all case scenarios are absolutely working with USB web cam but do not with ov5640. Expecting to reproduce live camera & recorded video scenarios with ov5640 cam on your end.

[manish_bajaj]

vsuneja63@gmail.com‌,

We have not tested and verified it on 8MQ. Are you able to get ML with live camera (using OV5460) working?

I would suggest to check where this /dev/video2 coming from. 

Can you share the script/command you are using? Working/nonworking...

-Manish

[vsuneja63]

--video_fwk=opencv flag was suggested by Alifer, without this flag nothing work.

[vsuneja63]

Hi Manish,

ML with live ov5640 camera is only streaming but detection not working. Tested again with recorded video of ov5640 camera not getting /dev/video2 but same error video device could not initialize. Tried with other youtube videos(just changed video source in the below command), detection working fine but streaming is little slow.

Sharing the error logs below:

root@imx8mqevk:/home/pyeiq# pyeiq --run face_and_eyes_detection --video_src=/home/test_h264.avi --video_fwk=opencv
#########################################
# #
# PyeIQ - face_and_eyes_detection #
# #
#########################################

====== AIUR: 4.5.5 build on Jul 30 2020 15:31:43. ======
Core: AVI_PARSER_03.06.07 build on May 31 2019 02:09:31
file: /usr/lib/imx-mm/parser/lib_avi_parser_arm_elinux.so.3.1
Track 00 [video]: Disabled
Codec: 0, SubCodec: 0
------------------------
[ WARN:0] global /usr/src/debug/opencv/4.2.0.imx-r0/git/modules/videoio/src/cap_gstreamer.cpp (1758) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module aiurdemux0 reported: GStreamer encountered a general stream error.
[ WARN:0] global /usr/src/debug/opencv/4.2.0.imx-r0/git/modules/videoio/src/cap_gstreamer.cpp (888) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /usr/src/debug/opencv/4.2.0.imx-r0/git/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Your video device could not be initialized. Exiting...

[Alifer_Moraes]

Hi,

Which version of PyeIQ are you using? This error you got on the log was fixed on the new version we released last week (2.1.0), you can update your package with the following command:

# pip3 install eiq --upgrade

Regards,

Alifer

[vsuneja63]

Hi Alifer,

Thanks!

Tried to upgrade but looks like its already up-to-date. As I mention issue is with ov5640 camera recorded videos only not with other videos. I request if you can investigate with same environment(imx8mqevk with ov5640 camera, recorded video & live camera) setup on your side. 

[Alifer_Moraes]

Hello,

I tried to run it on the same environment you said (i.MX 8MQ EVK - BSP 5.4.24-2.1.0 and OV5640). I didn't see random detection, but it wasn't detecting properly if the face was far from the camera, otherwise, it worked fine.

Regards,

Alifer

[vsuneja63]

Hi Alifer,

Thanks!

I tested here again but same no detection happening, tried with new imx8mq board & new camera module(ov5640). No face-eye detection even if I am close to less than a feet. Not sure what could be the difference. Can we enable debug mode & verify the logs for detection? Or any other way if you can suggest.  

[Alifer_Moraes]

Hi vsuneja63@gmail.com‌,

PyeIQ was tested and validated only on i.MX 8MP and i.MX 8QM, we never ran tests on i.MX 8MQ, but it should work.

Could you give us more details about the error you are facing? Which lines are you using to run the demo? Do you have any log messages or screenshot to help us figure out what may be wrong?

Regards,

Alifer

[vsuneja63]

Hi,

Thanks!

g2d support is not there in imx8mq evk. Does that could be the reason for this? Here are the logs:

The specified video_src was not found.
Searching for default video device...
Using /dev/video0 as video device
[ WARN:0] global /usr/src/debug/opencv/4.2.0.imx-r0/git/modules/videoio/src/cap_gstreamer.cpp (935) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1

[Alifer_Moraes]

Hello,

As Manish said, PyeIQ is just a reference and a quick way to run demos and there are a series of factors that may impact performance and result, especially on i.MX 8MQ, we didn't validate PyeIQ on 8MQ.

I don't think this is related to imxvideoconvert_g2d, because it is not supported by i.MX8 MQ and you said you were able to stream video without error.

Which line are you using to run the demo? Could you try to use "--res=vga" with your line and see if the demo will work properly?

Regards,

Alifer

[vsuneja63]

Hi,

Recorded video by v4l2 camera & passing that as video source throwing GStreamer warning. No Streaming at all. Here are the logs:

Using /dev/video2 as video device
Resolution not supported. Using 640x480 instead.

====== AIUR: 4.5.5 build on Jul 30 2020 15:31:43. ======
        Core: AVI_PARSER_03.06.07  build on May 31 2019 02:09:31
 file: /usr/lib/imx-mm/parser/lib_avi_parser_arm_elinux.so.3.1
    Track 00 [video]: Disabled
        Codec: 0, SubCodec: 0
------------------------
[ WARN:0] global /usr/src/debug/opencv/4.2.0.imx-r0/git/modules/videoio/src/cap_gstreamer.cpp (1758) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module aiurdemux0 reported: GStreamer encountered a general stream error.
[ WARN:0] global /usr/src/debug/opencv/4.2.0.imx-r0/git/modules/videoio/src/cap_gstreamer.cpp (888) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /usr/src/debug/opencv/4.2.0.imx-r0/git/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Your video device could not be initialized. Exiting...

[Alifer_Moraes]

Hello,

Please, could you send every detail of your environment? Board specs, BSP version, camera model, etc, so I can reproduce those errors to try and find a solution?

In the meantime, could you try to run the demo with the flags --res=vga --video_fwk=opencv?

Thank you,

Alifer

[vsuneja63]

Hi Alifer,

Thanks!

Tested v4l2 camera recorded video for face eye detection with the flags "--res=vga --video_fwk=opencv", but Same issue.

[Alifer_Moraes]

Hello,

Thank you for the update. I'm still investigating what may be causing this issue and as soon as I find something I'll let you know.

BR,

Alifer

[vsuneja63]

Hi Alifer,

Thanks!

Did you get a chance to reproduce the scenario with ov5640 camera on imx8mq-evk? Its a blockage for me, looking forward for your response. 

[manish_bajaj]

vsuneja63@gmail.com‌,

Can you confirm if you are referring below error for 8MQ?

Using /dev/video2 as video device
Resolution not supported. Using 640x480 instead.

-Manish

[vsuneja63]

Hi Manish,

Thanks!

Yes, this error is there if using recorded video by ov5640 camera as input.