i.MX6 H.264 AVC accessing SEI data

[davidthompson]

I was wondering if anyone more familiar with the code could point me in the right direction. I'd like access to specific SEI messages embedded within the h.264 stream, without having to reparse the whole stream a second time. I realize I'll need to write this code myself, I was just looking for advice on where the best place to make my modifications would be. Here are some details.

I am playing an h.264 video on the i.MX6, and the video is decoding/displaying correctly. However, my application needs access to additional data embedded in the h.264 stream, not related to decoding the video. Specifically, I am trying to access Closed Captioning data (Subtitles on American television). This data is embedded in "ITU-T T.35 registered User data SEI" messages, which can typically just be parsed along with the rest of the h.264 stream.

My problem is, that the actual H.264 stream parsing is done within the closed-source Freescale library, that is only distributed (I believe) in precompiled format. I'm pretty sure the data is even being parsed, I'm just not familiar with any API that would let me access it. Is there any standard mechanism for me to request additional data from this library?

I believe this is the current flow of data:
GStreamer (vpudec)  >>  libfslvpuwrap  >>  libvpu  >>  lib_H264_dec_arm11_elinux.so (Freescale precompiled library)
I haven't yet found any place, particularly in libVPU, where I can get access to SEI messages. Is there someplace else I should be looking?

I can always pre-parse the h.264 stream, before I hand it off to the VPU, but since the entire thing would then get parsed again a second time, that would be inefficient, and something I'd like to avoid. I understand if this is not possible, I was just hoping someone had done something similar before, and could advise me on what changes needed to be made.

Thanks for the help,
David

[davidthompson]

Also, I forgot to mention this in the original post. It seems like the Freescale decoder is already parsing the data I want even. I ran objdump and found the following methods, which is exactly what I need.

# objdump -t /usr/lib/lib_H264_dec_arm11_elinux.so | grep User

00005594 g     F .text 00000004              AVCD_DecodeUserdataUnregisteredInfo

00005590 g     F .text 00000004              AVCD_DecodeUserdata_ITU_T_T35_Info

I just need to figure out what it does with that data, if it sticks it into some structure I could read it back from, or if the data is simply thrown away as unused.

[davidthompson]

I believe I was misunderstanding a few of the GStreamer plugins. While I'm not any closer to finding where to make the changes I need, I wanted to verify I wasn't looking in the completely wrong place. If anyone familiar with the gstreamer plugins could confirm this, I'd appreciate it.

There are, or at least have been, three different Freescale h.264 decoder plugins for gstreamer:

mfw_h264decoder- Software only decoder. Not hardware accelerated. Don't use it unless you don't have hardware accel.

mfw_vpudecoder- VPU, hardware accelerated decoder. Similar to vpudec (see below) but has been either discontinued, or replaced by a new plugin doing similar work on the i.MX6.

vpudec- VPU, hardware accelerated decoder. Replaces mfw_vpudecoder, at least in newer (i.MX6) boards, maybe others. Use this one if it is available to you.

My mistake was, looking for connections to the decoders libfslcodec places in /usr/lib. For example, /usr/lib/lib_H264_dec_arm11_elinux.so is a precompiled library for the SOFTWARE only decoder, mfw_h264decoder and has nothing to do with the vpudec decoder. It seems like for vpudec, the decoders are loaded from a binary file in /lib/firmware/vpu.

Can anyone confirm my understanding of the decoders is now more or less correct? I'm still looking for a way to get some extra data back from the parsing process, but I'm hoping at least now I'm looking in the correct place. Any suggestions are appreciated as always.

Thanks,
David

[eaglezhou]

Hi, David

    Let me make simple explaination for the three decoders:

       - mfw_h264decoder: it is h264 software plugin
       - mfw_vpudecoder: it is vpu plugin for iMX5
       - vpudec: it is vpu plugin for iMX6DQ

       For software path:
           mfw_h264decoder -> lib_H264_dec_arm11_elinux.so
       For vpu path:
           mfw_vpudecoder -> libvpu -> /lib/firmware/vpu*.bin
           vpudec -> libfslvpuwrap -> libvpu -> /lib/firmware/vpu*.bin

    The source code for plugins/libfslvpuwrap/libvpu are all open.
    As far as I know, there are no related parser operation or interface exported from vpu.
    So if you are using vpu decoder on iMX6DQ, you can try to add your own parser code in vpudec or libfslvpuwrap.
  

Eagle

[benhenricksen]

Similarly does the encoder have an APi for inserting SEI data now?

For comparison / example:

DM36x H.264 encoder FAQ - Texas Instruments Wiki