How to decode frame-based video stream by MX53 VPU

[frankchenmh]

I'am developing an application which receives H264 bitstream from net and decodes the stream frame by frame. The platform is MX53 WINCE. I develop the application based on the VPU test WINCE700\support\APP\VPU\DECTEST. Now I get a problem. The encoded H264 frame does not have enough byte to feed the VPU bitstream buffer, which results in VPU hang after calling vpu_DecStartOneFrame(). Is there any known solution for this issue?

BTW,  is there a link for download MX53 WINCE VPU API doc?

Thanks.

[eaglezhou · Accepted solution]

Hi, Frank

     For iMX5 platform, In general, there are three kinds of selection provided to user

     (1) stream mode with blocking

          filePlayEnable=0, prescanEnable=0

          in such mode, vpu will always wait enough data for decoding

          user need to create one additonal thread responsible to feed the data to avoid application hang.

     (2) stream mode without blocking

          filePlayEnable=0, prescanEnable=1

          in such mode, vpu will always check whether the frame is integrated, it will start decode only when the frame is integrated.

          otherwise, it will return without decoding. this mode will introduce some performance penalty

     (3) file mode without blocking

          filePlayEnable=1

          in such mode, it is user's responsibility to make sure the frame is integrated.

          if the frame isn't enough, vpu will return failure.

Eagle.

[frankchenmh]

Hi Eagle,

Thanks for your reply.

For my case the third selection "file mode without blocking" will resolve the issue because my application can guarantee the frame is integrated.

I tested with filePlayEnable=1. It worked for the very first H264 bitstream (the H264 Header SPS+PPS which is 21 bytes). By default (filePlayEnable=0, prescanEnable=0) the vpu_DecGetInitialInfo() failed if we feed only 21-bytes H264 header to the VPU bitstream buffer, but vpu_DecGetInitialInfo() succeeded when filePlayEnable is set to 1.

But when decoding real frames the VPU API has strange result. The application calles vpu_DecGetBitstreamBuffer() before feed data to VPU buffer. The last parameter of vpu_DecGetBitstreamBuffer() will be filled by the API and indicates the free space of VPU buffer. When the buffer is empty (ie. no un-decoded bitstream in the buffer), it should be 1048575 (it equals 1024*1024 - 1). This can be verified for the case filePlayEnable=0 and prescanEnable=0.

However for the case filePlayEnable=1, the free space is reduced to 490 (512 - 1 - 21) even for a empty buffer. It is very small and so it is hard to feed bit stream to the buffer. Is this a normal VPU behaviour, or I have made some mistakes in calling VPU API?

Is there any VPU example that decodes H264 with filePlayEnable=1 for MX53?

Thanks.

[eaglezhou]

Hi, Frank

     In general, there are some alignment limitation in vpu, such as 512 bytes, this is why vpu_DecGetInitialInfo() may return failure sometimes.

     For stream mode, you need to feed more bytes to push vpu work.

     For file mode, you needn't call vpu_DecGetBitstreamBuffer(). what you need to do is just feed the frame data into start address of bitstream buffer

     and set the correct frame size through parameter 'chunkSize'. Another way, you had better fill the SPS/PPS/FrameData for the first decoding.

     There isn't file mode sample codes avaliable in our unit test so far.

     For more details, you can refer to our vpu api document,

     or you can read our source codes in framework (gstreamer plugin for linux, openmax component for android).

Regard,

Eagle

[frankchenmh]

Hi Eagle,

Basically I can now decode frames with filePlayEnable flag set based on your info about the bitstream buffer management.

However there are a little mosaics on the right-bottom corner of the output video. The warning messages like "Num of Error Mbs : 296, in Frame : 67" are printed out. There is no such problem when filePlay mode is disabled. I guess there still may be mistake when filling the bitstream buffer.

I tried align the bitstream to 512byte-aligned when filling the buffer. For frames that are not 512B-aligned, they are appended with padding of zero data. The chunkSize is also the size with padding accordingly. But with this change there are more mosaics.

I searched the MX53 Android source tree and didn't find the code about filePlay mode. Perhaps the VPU openmax code is released as binary only. I do not have MX53 pure Linux development environment for now.

If you have any comment about the mosaic issue, please let me know. Thanks.

[eaglezhou]

Hi, Frank

     The warning "Num of Error Mbs : 296, in Frame : 67" means vpu find some error when decoding current frame.

     I have no much idea now

     (1) the mosaic occur in every frame or not ?

          can you try another clip to make sure your logic are ok ?

     (2) you need to make sure the frame size is less than the bitstream buffer size.

     (3) you can compare the output frame numbers between stream mode and file mode

          maybe some error frames are dropped by stream mode.

     (4) in fact, you needn't fill the 512byte-aligned size for file mode.

Regard,

Eagle 

[frankchenmh]

Hi Eagle,

Please see my comments below.

(1) Not every frame has mosaic, but most of them have mosaic.

(2) I am sure the frame size is less than the buffer size. The frames are only several KB or less.

(3) There is no frame drop for either stream mode or file play mode.

Currently I set the chunkSize to exactly the frame size (without 512-alignment). Is this right?

(Sorry for asking the API-specific question. I am in the process requesting the VPU API doc. The doc is NDA-required as I was told.)

Thanks.

[eaglezhou]

Hi, Frank

    chunkSize is exactly the frame size.

    (1) please feed SPS/PPS/FrameData but not FrameData only for the first frame.

    (2) if the clip is interlaced, please feed two fields(top+bottom) instead of one field.

Regard

Eagle.

[frankchenmh]

Hi Eagle,

The mosaic issue has been resolved. The issue is due to the mistake that I made in the code accidently that for some frames the bitstream is not filled at the beginning of the VPU bitstream buffer. Now the result is perfect. Thanks for your help for achieving all of this!