Need guidance on video streaming on i.mx 8m, linux, ffmpeg & openmax

[doug4350]

Hi NXP community.

My first post here...

I purchased the i.mx 8m evaluation kit to work on video streaming from ip cameras.  I have successfully got the board to boot from a Linux BSP sd card, can ssh to the board and poke around.  There was no gnu gcc compiler out of the box, so I spent a couple days figuring out and setting up a cross compiling environment on my Mac.  It works by creating a Docker container with the Ubuntu 18.04 image and gcc build tools.  After getting the docker image created and running, I followed the I.MX8 Hello World Linux post here as a guide: i.MX 8 Hello World Linux - ArmV7a 32-bit and ArmV8a 64-bit  

I am porting an ffmpeg and openmax based video streaming platform to the i.mx8 and am at a cross roads. 

I have the ffmpeg library compiled from source and can connect with a test program to an ip camera and get data from the camera to the i.mx8 evaluation board.  Cool so far.  I haven't tried any decompression or other steps yet.

Being that there appears to be a long ride on any road I now choose to go down, can anyone recommend whether to keep openmax or suggest an alternative that fits better?  I've spent some time looking for openmax downloads for the board, but finally took a step back and decided to ask the community.

The platform I'm porting from will accept streams from multiple cameras, decode them and do some manipulation on the raw raster image, re-encode them and stream back out to client viewers who periodically connect to the platform.  Re-encoding is only done when client viewers are connected.  The client viewers are able to select at will which camera to view from.  That's kind of a high level description of my end goal & hope it makes sense.

I'd really appreciate any direction to how-to docs, examples, source downloads for recommendations you all think will work for me.

Take care,

- Doug Slattery

"Nothing is impossible if ImPossible"

[doug4350 · Accepted solution]

I've finally got my development environment setup and running on the mx8mqevk platform.  The Hantro, FFMPEG and OpenMax IL libraries are compiled and I can link a binary.  I haven't done much testing, but will open another discussion for that if needed.

Here's some notes for anyone else coming along that may be in a similar situation needing to build an image or create a cross-compiling environment on Docker for Mac:

After a few retries of increasing the docker container size to ultimately 200GB, that gave me sufficient space to build the image from the BSP I needed with all the packages.

I used this Docker image to start with: satoshikumano/ubuntu-18.04-cpp:0.0.1

and this command to start a container with the image.

Note: -v <host dir>:<container dir>  is an option to share a directory between the host and the docker container.  This is one way to copy files to/from the host/container which I find more convenient & easier than the docker cp command.

   ./docker run -it -d -p 80:80 -v <host dir>:<container dir> --name ubuntu satoshikumano/ubuntu-18.04-cpp:0.0.1 bash

The i.MX_Yocto_Project_User's_Guide.pdf file was quite helpful, but there were still issues I had to work through.

One example is on pgs 5 & 6 on the Yocto Project Setup chapter,  I had to use

   repo init -u https://source.codeaurora.org/external/imx/imx-manifest -b imx-linux-sumo -m imx-4.14.78-1.0.0_ga.xml

instead of

   repo init -u https://source.codeaurora.org/external/imx/imx-manifest -b imx-linux-rocko -m imx-4.9.88-2.0.0_ga.xml

Probably due to the pdf being a little behind.

In build/conf/local.conf, I had this line set to the following to include all the recipes I needed:

EXTRA_IMAGE_FEATURES ?= "dbg-pkgs dev-pkgs tools-sdk tools-debug tools-profile debug-tweaks"

There were several linux package dependencies to install besides the ones shown in section 3.1 Host Packages

a few were:

apt-get install python3 python3-pip cpio vim locales dpkg libncurses5-dev libncursesw5-dev xcb-proto

Also, while building, there were various compile and install stage failures, and I found for a reason yet unknown, that if I went in and ran the appropriate run_docompile or run_doinstall script from a different shell window, it would go through, at which point, the yocto build could be restarted and continue on.

Some places I had to actually tweak build scripts as one instance, it was trying to execute python 2.x script with a python 3.x interpreter.

Other cases, there were gcc compiler errors which I had to trek through to get working.

There were some cases where the compiler was building for the local environment rather than the target environment which ended up being the $PATH pointing to the incorrect compiler.  Adjusting the $PATH variable resolved it.

Also, one part of the build was complaining about the locale not being correct, so some parts of these steps fixed it:

    dpkg-reconfigure locales

    apt-get install locales

    # local-gen en_US.UTF-8

    update-locale LC ALL=en_US.UTF-8 LANG=en_US.UTF-8

    export LANG=en_US.UTF-8

There were link errors on some modules complaining about missing xcb, Xau and Xdmcp libraries.  I ended up tweaking either the offending Makefile or Makefile.ac.  Others were resolved by setting this env variable:

export X_EXTRA_LIBS="-lxcb -lXau -lXdmcp"

Here's another build variable that fixed another build failure on one of the recipes:

# error: Undefined reference to _IO_fwide:

# add -D_IO_fwide=fwide to CFLAGS

And lastly, strange compiler errors indicating bugs in gcc which turned out to be OOM errors.  I increased the ram footprint of the docker image to 6GB.

This is what the final .sdcard image looked like:

-rw-r--r-- 1 pi pi 8682209280 Feb 23 09:49 fsl-image-validation-imx-imx8mqevk-20190222165938.rootfs.sdcard

Take care,

- Doug

"Nothing is impossible if ImPossible"

[diegoadrian]

Hello,

Unfortunately, have apt-get on a Yocto image is not possible, since Yocto is a costume made image for the embedded systems, all the package needs to be pre-installed by recipes. Some of the repositories are already included on our BSP and they are easy to add in the final Yocto image, for example, the gcc compiler. For OpenMax, we do have the Gstreamer plug-ins that could help you with the all the streaming of the video.

We do have a Yocto training that I recommend you to follow if you want to learn about Yocto. The training is so well explained. Please find below the link for the Training:

https://community.nxp.com/docs/DOC-94035 

Hope this information can help you.

Best Regards,

Diego.

[doug4350]

Thanks for the feedback Diego.

I'm working through creating a custom image with Yocto.  I'm getting several build failures working through them one by one until the image is built.  I'm on my 2nd attempt, the first was missing some layers.

The Yocto Training link you mentioned is dead.  Just FYI, I kind of have a handle on it now.

I still want to keep this thread open if that's ok since the goal is to get my streaming server ported to the NXP platform.

Take care, 

- Doug

"Nothing is impossible if ImPossible"

[diegoadrian]

Hello,

My bad.

Please see if this link to the training works:

https://community.nxp.com/docs/DOC-94849 

Best Regards,

Diego.

[doug4350]

Hi everyone, 

I've made some progress, but am not finding a needed library describe in this guide:

IMX8MDQlQRM.pdf chapter 14 - Video Processing Unit (VPU) 

It says that it's using the Hantro hw decoder library under the Bellagio implementation of OpenMax.  More specifically, OMX IL API for 8170/8190/9170/9190/G1/G2.

I've managed to locate and build the Bellagio OMX library, but can't find much useful information on Hantro.

Anyone have any experience or information on the Hantro decoder libs?   I will also be doing transcoding, so will be following that path for re-encoding after getting decoding working.

I found a gstreamer example using Hantro which I'm trying to pick apart to become familiar with Hantro, but it's not using the OpenMax IL.

Thanks,

- Doug

"Nothing is impossible if ImPossible"

[diegoadrian]

Hello,

The Hantro branch for the video processing unit is already set as default in our BSP. You can confirm it if you go to the menuconfig configuration described in the Linux User Guide.

Best Regards,

Diego.

[doug4350]

Hi Diego and thanks for the response.

@Moderator - How can I have messages auto-approved so they don't have to be reviewed by you before posting to the forum?  This is the only forum I belong to that is doing that past the first post...

I'm was at CES last week and now am getting back into normal work mode.

@Everyone - I seem to be missing some items I need on the BSP.  I'm new to Yocto, and only imaged an SD card from in the fsl-image-qt5-validation-imx-xwayland-imx8mqevk.sdcard image in the L4.9.88_2.0.0_images_MX8MQ.tar.gz BSP.  I have a feeling I need to add more packages via Yocto to be productive.

Can anyone point me to a preconfigured image or package(s) with the above BSP that also contains: (The image I'm using is missing these items)

- apt-get and other linux system management tools.

- gcc compiler & C/C++ development tools.

- menuconfig and related tools

- ffmpeg and OpenMax IL (I've been able to cross compile them, but it would be more convenient building on the board itself)

I've  done very little modification on the os image, so I'm ok with scrapping it and starting again with an image containing the missing pieces.  If I at least have apt-get and the board specific drivers & libs installed, I can fetch any os packages I need via apt-get if they're available or build from source.

Thanks,

- Doug

"Nothing is impossible if ImPossible"

[diegoadrian]

Hello,

In the Linux user guide, on the section 7.3.12, you can see an example of how to stream a video with Gstreamer.  Hope this information can help you.

Best Regards,

Diego.

[doug4350]

Hi Diego and thanks for the reply.

What you are referring to looks like command line utility related, whereas I'm looking for libraries and source examples in c/c++ that will tap into the vpu/gpu which is where OpenMax comes in.

I stumbled on a couple of nuggets after reading & searching some more which looks more like what I need:

i.MX_Linux_Reference_Manual.pdf section 6.6 - Video Processing   [seems to be g-streamer source code oriented]

IMX8MDQlQRM.pdf chapter 14 - Video Processing Unit (VPU)         [looks to have OpenMax details I'm needing, now if that's the case, I just need to find the OpenMax libraries.]

I may have more questions, so will leave the thread open for now.

Take care,

- Doug Slattery

"Nothing is impossible if ImPossible"