Text and video streaming over network

[sachindeshpande]

Hello All,

I want to stream text and video over network. The purpose is to overlay the text and video and send it over the network. i am using gstreamer for this.

Can any one tell me as how can i send both together and what is the pipeline for the server and client ?

Thanks in advance

Regards

Sachin

[BiyongSUN]

pc:

gst-launch-0.10 -v videotestsrc   ! x264enc ! rtph264pay ! udpsink host=10.192.246.76 port=8890

board:

export VSALPHA=1
gst-launch udpsrc caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' port=8890 ! rtph264depay ! vpudec ! mfw_isink sync=false axis-top=100 axis-left=100 disp-width=320 disp-height=240

[EricNelson]

Hi Sachin,

From your description, you need to insert text into the presumably YUV bitstream of each video frame before it's handed off to the video encoder, producing an altered video stream that has the text overlay.

There is hardware support in the IPU for layers in the display subsystem, but it doesn't sound like that's what you need.

Instead, you simply need something to render text into YUV buffers. The GPU can provide acceleration of either the drawing primitives here or blending,

but this is not really an expensive operation and may be over-kill.

There is a gstreamer text overlay that may do what you're after ("pango", I think), but I haven't used it.

It may also disrupt the buffer allocations in the pipeline.

What you're after is a pipeline like this:

  Input (camera or decode output as YUV) -> textoverlay -> VPU

You need to make sure that the hardware buffers provided on the input side are handed off to the VPU.

[LeonardoSandova]

Thanks Eric. In the past, I have tried the pango's overlay but performance degrades considerably for big frames resolutions... I believe the time consuming part is the mallocs and memcpys involved...

[benhenricksen]

Hi Leonardo,

Can you give an indication of the performance you achieved with Pango - frame-size, frame-rate, and which processor variant etc?

Thanks,

Ben

[EricNelson]

Hi Leonardo,

That confirms my suspicion that the pango plugin doesn't preserve the hardware-allocated buffers.

Sachin, this means that you probably need a custom plugin to keep the pipeline fully accelerated.

I doubt that you need to hardware accelerate the plugin much, if at all. If your app is okay writing

white text on top of the video, it's really easy to layer-in Freetype's output directly into a Y layer,

such that fully opaque in the text render buffer becomes fully white.

If you need more complex blending, you should consider using the GPU to allocate a full ARGB

layer, and have it blend onto the output buffer. Since the GPU supports YUV buffers, this works

really well.

You can find some examples in the opengl plugin.

[LeonardoSandova]

If you are streaming FROM the target TO a host, I do not think you will get a good performance (because putting test into a video requires SW processing, which is too expensive for high resolutions). is this your case? There are some documents on the community regarding UDP/RTP streaming....

Leo

[sachindeshpande]

Hello Leo,

Thanks for the reply,

Yes you are correct I want to stream the video with image and text overlayed. I was able to build the pipeline and stream over network. I am using x86 as my server and I.MX6Q as client. But the final goal is to use I.MX6Q as server as well as client.

Yes the performance for the high resolution was not good.  I am trying to implement the following scenerio

1) Input video stream from STB / IP to server

2) Stream the video in Multicast mode

3) Periodically overlay Image and Text over the video and steam

For the above purpose I am doing the overlay. Could you please suggest the optimized way of achieving this ?

Thanks and Regards

Sachin

[sachindeshpande]

Hello Leo,

Does i.mx6q has support for hardware mixing. What I mean is there a way of doing the image overlay on top of video stream using some hardware module in i.mx6q board.

I tried that but yes with higher resolution the video is very bad. Could you please point as how can I do this in optimal way using hardware support.

Regards

Sachin

[sachindeshpande]

Hello Leo,

Any suggestion on the above question?

Regards

Sachin

[LeonardoSandova]

Hi Sachin, sorry for the delay. I got no precise input for this one. I will push this question internally so an IPU specialist can help you better.

Leo