IPU multiple video out + Overlay

[Tarek]

Hi,

How do I create multiple instances of the IPU for playing multiple video streams each in a separate window at the same time?

Also which instance will display the Overlay GUI which should cover all video windows?

I'm using:

i.MX6Q - Nitrogen board - BSP 12.08 - Openembedded build system - Poky distribution - Yocto project

Thanks

[FranciscoCarril]

Please take a look to these examples:

GStreamer iMX6 Multi-Overlay

[Tarek]

Hi Rancisco,

I can't use Gstreamer because I need to demux the stream using a modified version of FFMPG. Thus I'm using the VPU API's directly. In my application I have a decoder while loop calling av_read_frame() continuously which gives a compressed Packet "packet.data and packet.size". I pass this buffer to VPU and obtain a decoded Frame YUV. I need to pass the output buffer to the display task. And do the same for multiple streams.

Is there anyway I can integrate Gstreamer into the decoder loop to avoid all the VPU - IPU - GPU complications?

If not, is there a more detailed guidance on how to replicate the Gstreamer implantation?

[LeonardoSandova]

Hi Tarek,

Is there a way you can change any of the ffmpeg demuxer GStreamer elements to get the same result? Perhaps this way will take you longer, but once you have the demuxer element ready, you can plug it in into a standard pipeline and use the mfw_isink (Freescale video sink element), where the latter has internally all the logic you want to implement.

Leo

[Tarek]

Hi Leonardo,

I found an interesting gst-plugin that should do the job for me.

ppsrc — Easy way for applications to inject buffers into a pipeline

I found the source for this plugin as part of gst-plugins-base. I hove compiled and included this into my image.

In my image the libraries are installed correctly as expected:

./usr/lib/libgstapp-0.10.la

./usr/lib/libgstapp-0.10.so.0.25.0

./usr/lib/libgstapp-0.10.so

./usr/lib/libgstapp-0.10.so.0

./usr/lib/gstreamer-0.10/libgstapp.la

But Gstreamer doesn't seam to recognize this plugin

#gst-inspect  appsrc

No such element or plugin 'appsrc'

Please can you ask the Gstreamer experts to help with installing this plugin?

Thanks,

Tarek

[LeonardoSandova]

Hi Tarek,

I am glad you found a gstreamer option. I think you have three options so gst can find the compiled plug-in:

1. Move your share objects to /usr/lib/gstreamer-0.10 (sudo cp /usr/lib/libgstapp* /usr/lib/gstreamer-0.10) or

2. Check the ./configure script and indicate where the share objects will be installed (do not remember the exact parameter name) or

3. in your pipeline, add the parameter --gst-plugin-path=/usr/lib

Let me know if it works.

Leo

[Tarek]

Hi Leo,

I found the problem, I have installed the development pkg gst-plugins-base-app-dev but missed the main pkg gst-plugins-base-app. Now I can create appsrc element successfully but I still have problems.

I think the gst VPU plugin is unable to recognize the stream type, although when I used the vpu api's it was fine.

I've got the following pipeline:

/*app-source -> vpu-decoder -> mfw_v4lsink-sink*/

gst_element_link ((GstElement*)ctx->appsrc,ctx->vdecoder );

gst_element_link ( ctx->vdecoder, ctx->vdsink );

I'm pushing packets every time I receive need-data signal but after some packet-pushes I get vpudec critical error.

Do I need to add typefind element to the pipeline?

0:00:00.689993667  1669 0x33865580 DEBUG                GST_BUS gstbus.c:767:gst_bus_source_dispatch:<bus1> source 0x33864350 handler returns 1

0:00:00.690191833  1669 0x33865580 DEBUG                GST_BUS gstbus.c:500:gst_bus_timed_pop_filtered:<bus1> got message 0x33b02cf8, state-changed from avpacket, type mask is 4294967295

0:00:00.690273167  1669 0x33865580 DEBUG                GST_BUS gstbus.c:762:gst_bus_source_dispatch:<bus1> source 0x33864350 calling dispatch with state-changed message from element 'avpacket': GstMessageState, old-state=(GstState)GST_STATE_READY, new-state=(GstState)GST_STATE_PAUSED, pending-state=(GstState)GST_STATE_VOID_PENDING;

0:00:00.690561500  1669 0x33865580 DEBUG                GST_BUS gstbus.c:767:gst_bus_source_dispatch:<bus1> source 0x33864350 handler returns 1

vdecode.c(852): push_packet

0:00:00.690957833  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c008

vdecode.c(852): push_packet

0:00:00.691188000  1669 0x33864980 DEBUG                 appsrc gstappsrc.c:988:gst_app_src_create:<avpacket> we have buffer 0x33b0c008 of size 37905

0:00:00.691412833  1669 0x33864980 DEBUG                basesrc gstbasesrc.c:2045:gst_base_src_do_sync:<avpacket> we have no clock

0:00:00.691486000  1669 0x33864980 DEBUG                basesrc gstbasesrc.c:2231:gst_base_src_get_range:<avpacket> buffer ok

0:00:00.691587500  1669 0x33864980 DEBUG              GST_EVENT gstpad.c:5402:gst_pad_send_event:<video-decoder:sink> have event type newsegment

0:00:00.691670500  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c070

0:00:00.691705167  1669 0x33864980 WARN                  vpudec vpudec.c:1810:gst_vpudec_sink_event: Unsupport newsegment format 2

0:00:00.691823333  1669 0x33864980 DEBUG               GST_PADS gstpad.c:5430:gst_pad_send_event:<video-decoder:sink> sent event, result 0

vdecode.c(852): push_packet

0:00:00.692099166  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c0d8

vdecode.c(852): push_packet

0:00:00.692252666  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c140

vdecode.c(852): push_packet

0:00:00.692390000  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c1a8

vdecode.c(852): push_packet

0:00:00.692523166  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c210

vdecode.c(852): push_packet

0:00:00.692649833  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c278

vdecode.c(852): push_packet

0:00:00.693015166  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c2e0

vdecode.c(852): push_packet

0:00:00.693301666  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c348

vdecode.c(852): push_packet

0:00:00.693585500  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c410

vdecode.c(852): push_packet

0:00:00.693864000  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c478

vdecode.c(852): push_packet

0:00:00.694147166  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c4e0

vdecode.c(852): push_packet

0:00:00.694425833  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c548

vdecode.c(852): push_packet

** (console:1669): CRITICAL **: gst_vpudec_chain: assertion `vpudec->context.handle' failed

0:00:00.695730500  1669 0x33864980 INFO                 basesrc gstbasesrc.c:2562:gst_base_src_loop:<avpacket> pausing after gst_pad_push() = wrong-state

0:00:00.695841500  1669 0x33864980 DEBUG                basesrc gstbasesrc.c:2588:gst_base_src_loop:<avpacket> pausing task, reason wrong-state

0:00:00.695928333  1669 0x33864980 DEBUG               GST_PADS gstpad.c:5646:gst_pad_pause_task:<avpacket:src> pause task

0:00:00.696020833  1669 0x33864980 DEBUG                   task gsttask.c:698:gst_task_set_state:<avpacket:src> Changing task 0x33b0b050 to state 2

0:00:00.724539000  1669 0x33865580 DEBUG                 appsrc gstappsrc.c:1486:gst_app_src_push_buffer_full:<avpacket> queueing buffer 0x33b0c5b0

vdecode.c(852): push_packet

[LeonardoSandova]

Not sure if this helps but the appsrc element has a 'typefind' property, so you try it before inserting the 'typefind' element. Also, you can use the capsfilter element between the appsrc and the decoder. Both ways specify the decoder sink caps

Leo

[Tarek]

Hi Leo,

The typefind element is the only thing that works in my scenario. I still have more problems but I will create a new thread for that.

Thanks a lot for your help