RTSP gstreamer and vpudec

[holgerweber]

Hello,

I try to display a video stream with gstreamer. I use the following command, which works:

gst-launch-0.10 rtspsrc location=rtsp://192.168.25.60/stream1.sdp ! rtpmp2tdepay ! mpegtsdemux !  h264parse ! ffdec_h264 max-threads=2 ! mfw_v4lsink

The performance is not as good as it should be so I want to use the hw decoder:

gst-launch-0.10 rtspsrc  location=rtsp://192.168.25.60/stream1.sdp ! rtpmp2tdepay ! mpegtsdemux ! vpudec ! mfw_v4lsink

But nothing happens, no video is displayed. I only get this output:

MFW_GST_V4LSINK_PLUGIN 3.0.5 build on Dec 14 2012 13:58:00.
Setting pipeline to PAUSED ...
[INFO] Product Info: i.MX6Q/D/S
vpudec versions :smileyhappy:
  plugin: 3.0.5
  wrapper: 1.0.28(VPUWRAPPER_ARM_LINUX Build on Dec 14 2012 12:23:57)
  vpulib: 5.4.10
  firmware: 2.1.8.34588
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[INFO] bitstreamMode 1, chromaInterleave 1, mapType 0, tiled2LinearEnable 0
[WARN] VPU iram is less than needed, some parts don't use iram
>>V4L_SINK: Actually buffer status:
  hardware buffer : 24
  software buffer : 0

I also tried aiurdemux which replaces mpegtsdemux but there is a bug with rtsp streaming.

Any ideas?

[holgerweber]

The problem was decoding of low latency streams (intra-refresh streams). I had to recompile the vpu wrapper lib as discussed here: Re: H.264 decoding using ffmpeg + using GPU for display acceleration

Playback is now running with this pipeline:

gst-launch --gst-debug=vpudec:0 udpsrc uri=udp://192.168.24.39:4444 ! mpegtsdemux ! h264parse ! vpudec low-latency=true framedrop=true ! mfw_isink axis-top=100 axis-left=100 disp-width=640 disp-height=360 sync=false

But the latency is much to high! It's about 1000 ms we want to have a latency of 200ms or better.

Any ideas how I can reduce the latency?

[holgerweber · Accepted solution]

Some time later....

I read this thread https://community.freescale.com/message/358922#358922
which mentioned to change the vpudec.c file. It's also possible to change the value sDecOpenParam.reorderEnable to 0  in the vpu_wrapper.c file.

This reduced the delay of the whole system from about 1sec. to 200ms.

PATCH:

1. open vpu_wrapper.c
2. change pObj->iframesearch_allowed=1 to 0
3. change sDecOpenParam.reorderEnable=1 to 0

[reinholdfrederi]

Hi,

I would like to reduce latency time without change vpu_wrapper.c.

I have changed in vpudec this parameter: dis-reorder=true (equal to sDecOpenParam.reorderEnable=0).

Now i'm looking for pObj->iframesearch_allowed in vpudec parameters ?

Thanks for your help.

[LeonardoSandova]

Any reason you did not use h264parse on the second pipeline?

Please enable debugging when launching your pipeline (--gst-debug=*:5)

Leo

[holgerweber]

You right, I changed the command to:

gst-launch-0.10 --gst-debug=vpudec:5 rtspsrc location=rtsp://192.168.25.60/stream1.sdp ! rtpmp2tdepay ! mpegtsdemux ! h264parse  ! vpudec ! mfw_v4lsink

With no success but I get this messages:

MFW_GST_V4LSINK_PLUGIN 3.0.5 build on Dec 14 2012 13:58:00.

Setting pipeline to PAUSED ...

[INFO] Product Info: i.MX6Q/D/S

vpudec versions :smileyhappy:

  plugin: 3.0.5

  wrapper: 1.0.28(VPUWRAPPER_ARM_LINUX Build on Dec 14 2012 12:23:57)

  vpulib: 5.4.10

  firmware: 2.1.8.34588

0:00:00.178237667  7627    0x17050 LOG                   vpudec vpudec.c:392:vpudec_core_mem_alloc_dma_buffer: Call VPU_DecGetMem return 0x0

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

0:00:02.267122001  7627 0x2e700798 INFO                  vpudec vpudec.c:1812:gst_vpudec_sink_event: Get newsegment event from 0:43:59.915622222to 99:99:99.999999999 pos 0:00:00.000000000

0:00:02.267608335  7627 0x2e700798 INFO                  vpudec vpudec.c:1181:gst_vpudec_setcaps: Get upstream caps video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true

0:00:02.268074668  7627 0x2e700798 INFO                  vpudec vpudec.c:1190:gst_vpudec_setcaps: Get codec std 6

0:00:02.268709668  7627 0x2e700798 INFO                  vpudec vpudec.c:1230:gst_vpudec_setcaps: got downstream allow caps video/x-raw-yuv, format=(fourcc)NV12, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw-yuv, format=(fourcc)I420, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw-yuv, format=(fourcc)TNVP, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw-yuv, format=(fourcc)YV12, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw-yuv, format=(fourcc)TNVF, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw-yuv, format=(fourcc)YV12, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]

[INFO] bitstreamMode 1, chromaInterleave 1, mapType 0, tiled2LinearEnable 0

0:00:02.272831001  7627 0x2e700798 INFO                  vpudec vpudec.c:1283:gst_vpudec_setcaps: Use new tsm scheme

0:00:02.273043001  7627 0x2e700798 INFO                  vpudec vpudec.c:1165:gst_vpudec_setconfig: Set drop policy 0

0:00:02.273333335  7627 0x2e700798 LOG                   vpudec vpudec.c:1414:gst_vpudec_chain: Chain in with size = 30698

0:00:02.274277001  7627 0x2e700798 LOG                   vpudec vpudec.c:1467:gst_vpudec_chain: buf status 0x101 data 30698

0:00:02.274480335  7627 0x2e700798 INFO                  vpudec vpudec.c:1555:gst_vpudec_chain: Got not enough input message!!

0:00:02.281955001  7627 0x2e700798 LOG                   vpudec vpudec.c:1414:gst_vpudec_chain: Chain in with size = 30514

0:00:02.282935001  7627 0x2e700798 LOG                   vpudec vpudec.c:1467:gst_vpudec_chain: buf status 0x101 data 30514

0:00:02.283137001  7627 0x2e700798 INFO                  vpudec vpudec.c:1555:gst_vpudec_chain: Got not enough input message!!

        And after some more of them:

0:00:02.413860001  7627 0x2e700798 INFO                  vpudec vpudec.c:1555:gst_vpudec_chain: Got not enough input message!!

0:00:02.423047001  7627 0x2e700798 INFO                  vpudec vpudec.c:1181:gst_vpudec_setcaps: Get upstream caps video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)640, height=(int)360, framerate=(fraction)15/1, parsed=(boolean)true, pixel-aspect-ratio=(fraction)1/1

0:00:02.423533668  7627 0x2e700798 INFO                  vpudec vpudec.c:1190:gst_vpudec_setcaps: Get codec std 6

0:00:02.423642668  7627 0x2e700798 INFO                  vpudec vpudec.c:1194:gst_vpudec_setcaps: Set Width 640

0:00:02.423720335  7627 0x2e700798 INFO                  vpudec vpudec.c:1198:gst_vpudec_setcaps: Set Height 360

0:00:02.424047001  7627 0x2e700798 INFO                  vpudec vpudec.c:1230:gst_vpudec_setcaps: got downstream allow caps video/x-raw-yuv, format=(fourcc)NV12, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw-yuv, format=(fourcc)I420, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw-yuv, format=(fourcc)TNVP, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw-yuv, format=(fourcc)YV12, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw-yuv, format=(fourcc)TNVF, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw-yuv, format=(fourcc)YV12, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]

[INFO] bitstreamMode 1, chromaInterleave 1, mapType 0, tiled2LinearEnable 0

0:00:02.439768668  7627 0x2e700798 INFO                  vpudec vpudec.c:1283:gst_vpudec_setcaps: Use new tsm scheme

0:00:02.439918001  7627 0x2e700798 INFO                  vpudec vpudec.c:1165:gst_vpudec_setconfig: Set drop policy 0

0:00:02.440091335  7627 0x2e700798 LOG                   vpudec vpudec.c:1414:gst_vpudec_chain: Chain in with size = 30698

[WARN] VPU iram is less than needed, some parts don't use iram

0:00:02.440803668  7627 0x2e700798 LOG                   vpudec vpudec.c:1467:gst_vpudec_chain: buf status 0x201 data 30698

0:00:02.440956335  7627 0x2e700798 INFO                  vpudec vpudec.c:1125:gst_vpudec_core_start: Get min framebuffer count 18

0:00:02.441048001  7627 0x2e700798 INFO                  vpudec vpudec.c:1025:gst_vpudec_set_downstream_pad: Get initinfo padwidth 640, padheight 368, left 0, right 640, top 0, bottom 360, interlace 0

0:00:02.441275001  7627 0x2e700798 INFO                  vpudec vpudec.c:1074:gst_vpudec_set_downstream_pad: Try set downstream caps video/x-raw-yuv, format=(fourcc)NV12, width=(int)640, height=(int)368, pixel-aspect-ratio=(fraction)1/1, crop-top=(int)0, crop-left=(int)0, crop-right=(int)0, crop-bottom=(int)8, num-buffers-required=(int)24, framerate=(fraction)15/1, alignment=(int)1

0:00:02.447800001  7627 0x2e700798 LOG                   vpudec vpudec.c:392:vpudec_core_mem_alloc_dma_buffer: Call VPU_DecGetMem return 0x0

>>V4L_SINK: Actually buffer status:

  hardware buffer : 24

  software buffer : 0

0:00:02.515363001  7627 0x2e700798 LOG                   vpudec vpudec.c:1467:gst_vpudec_chain: buf status 0x821 data 0

0:00:02.516016668  7627 0x2e700798 INFO                  vpudec vpudec.c:1512:gst_vpudec_chain: Got drop information!!

0:00:02.521368335  7627 0x2e700798 LOG                   vpudec vpudec.c:1414:gst_vpudec_chain: Chain in with size = 30890

0:00:02.521752001  7627 0x2e700798 LOG                   vpudec vpudec.c:1467:gst_vpudec_chain: buf status 0x1 data 30890

0:00:02.522304335  7627 0x2e700798 LOG                   vpudec vpudec.c:1467:gst_vpudec_chain: buf status 0x821 data 0

0:00:02.522413668  7627 0x2e700798 INFO                  vpudec vpudec.c:1512:gst_vpudec_chain: Got drop information!!

0:00:02.525947001  7627 0x2e700798 LOG                   vpudec vpudec.c:1414:gst_vpudec_chain: Chain in with size = 30698

0:00:02.526490335  7627 0x2e700798 LOG                   vpudec vpudec.c:1467:gst_vpudec_chain: buf status 0x1 data 30698

0:00:02.527030001  7627 0x2e700798 LOG                   vpudec vpudec.c:1467:gst_vpudec_chain: buf status 0x821 data 0

0:00:02.527148001  7627 0x2e700798 INFO                  vpudec vpudec.c:1512:gst_vpudec_chain: Got drop information!!

[holgerweber]

If I use this command and I switch of low latency encoding of my external hardware encoder.

gst-launch-0.10 --gst-debug=vpudec:5 rtspsrc location=rtsp://192.168.25.60/stream1.sdp ! rtpmp2tdepay ! mpegtsdemux ! h264parse  ! vpudec ! mfw_v4lsink

It works... but with much to high latency! I have to use the low latency mode so has anyone an idea what I have to do?

>> Got drop information!!

How can I ignore that information or how can I solve that problem?

[LeonardoSandova]

do you observe the same issue with 'gst-launch playbin2 uri=rtsp://192.168.25.60/stream1.sdp'? Not sure if it works but lets try playbin2 to create the pipeline for you.

[holgerweber]

If I use playbin2, gstreamer uses the aiurdemux element.

No video appears and I get this messages (indipendent from low latency encode or normal encode):

Setting pipeline to PAUSED ...

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

0:00:02.246670002 12472  0x17ea60 LOG    aiurdemux aiurdemux.c:1836:gst_aiurdemux_handle_sink_event:<aiurdemux0> handling new segment from0

Aiur: 3.0.5
Core: BLN_MAD-MMLAYER_MPG2PARSER_04.04.04 build on Dec 11 2012 13:39:16
  mime: video/mpeg, mpegversion=(int)[1,2]; video/mpegts, systemstream=true; video/x-cdxa
  file: /usr/lib/imx-mm/parser/lib_mpg2_parser_arm11_elinux.so.3.1
Content Info:
Seekable : No
Size(byte): -1

0:00:02.278108661 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.278418995 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.278510328 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.278571328 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.278826328 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.278893661 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.278945661 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.278995995 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.279043661 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.279093328 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.279141328 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.279188995 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.279237328 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.279284328 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

0:00:02.279331661 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:2335:aiurdemux_add_user_tags: API[getMetaData] failed, ret=-6

Mpeg2CreateParser:parser created successfully

Movie Info:
Seekable : No
Live : Yes
Duration : 0:00:00.000000000
ReadMode : File
AutoRetimestamp: 2000ms
programs : 1
Track : 1

0:00:02.279613328 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:3195:aiurdemux_parse_streams: Warning: API[getLanguage] not implement!

0:00:02.279757661 12472  0x2e4da0 INFO    aiurdemux aiurdemux.c:2744:aiurdemux_parse_video: Create video pad video_00112c

Track 00 [video_00112c] Enabled
ppid: 1, ppid 300
Duration: 0:00:00.000000000
Language:
Mime:
video/x-h264, parsed=(boolean)true, width=(int)1280, height=(int)720, framerate=
(fraction)30/1, codec_data=(buffer)00000127640028ad840d218e634e30851514420a934ac
899a81e805005bb011000007d00001d4c084000
[INFO] Product Info: i.MX6Q/D/S

vpudec versions :smileyhappy:
plugin: 3.0.5
wrapper: 1.0.28(VPUWRAPPER_ARM_LINUX Build on Dec 14 2012 12:23:57)
vpulib: 5.4.10
firmware: 2.1.8.34588
0:00:02.310173000 12472  0x2e4da0 LOG    vpudec vpudec.c:392:vpudec_core_mem_alloc_dma_buffer: Call VPU_DecGetMem return 0x0

0:00:02.311043333 12472  0x2e4da0 ERROR    aiurdemux aiurstreamcache.c:335:gst_aiur_stream_cache_seek: Unexpect backward seek addr 0, cachestart 815798, offset 200010

0:00:02.311145333 12472  0x2e4da0 INFO    aiurdemux aiurstreamcache.c:360:gst_aiur_stream_cache_seek: stream cache try seek to 0

!!!! Pressing Strg+C !!!

Caught interrupt -- handling interrupt.

Interrupt: Stopping pipeline ...

Execution ended after 9027289665 ns.

Setting pipeline to PAUSED ...

Setting pipeline to READY ...

0:00:09.206715331 12472  0x17050 ERROR    vpudec vpudec.c:558:vpudec_core_deinit: Func VPU_DecFlushAll failed!! with ret 3

0:00:09.209800998 12472  0x17050 LOG    vpudec vpudec.c:373:vpudec_core_mem_free_dma_buffer: Call VPU_DecFreeMem return 0x0

0:00:09.210019331 12472  0x17050 INFO    vpudec vpudec.c:577:vpudec_core_deinit: Stat:

in : 0

out : 0

show: 0

0:00:09.313042333 12472  0x2e4da0 WARN    aiurdemux aiurdemux.c:1325:aiurdemux_send_stream_eos: Pad video: Send eos.

0:00:09.313309999 12472  0x2e4da0 LOG    aiurdemux aiurdemux.c:4247:aiurdemux_push_task:<aiurdemux0> pausing task, reason unexpected

0:00:09.313476333 12472  0x2e4da0 LOG    aiurdemux aiurdemux.c:4256:aiurdemux_push_task:<aiurdemux0> Sending EOS at end of segment

0:00:09.314449666 12472  0x17050 INFO    aiurdemux aiurdemux.c:2007:aiurdemux_release_resource: stream video[0] interleave queue max size 0

Setting pipeline to NULL ...

[--->FINALIZE aiurdemux
Freeing pipeline ...