decode H.264 stream with pps sps on vpu

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

decode H.264 stream with pps sps on vpu

3,362 Views
jorenbultheel
Contributor I

Hi,

I try to use the vpu to decode a H264 stream but vpu_DecGetInitialInfo() returns RETCODE_FAILURE. I think this is due to bad sps pps information propagation to the vpu.

Some answers on the following questions should help me out:

1) what is spSaveBuffer for and how big should I make it? What to put in it or what to read out of it? (I know it must be physical memory)

2) I extract PPS and SPS information from the extradata available in my H264 stream (I already found out how to do it in the case of annex-b format and also in the avcc format).

     Why am I the only one in the (google) world using the methods vpu_DecGiveCommand with DEC_SET_SPS_RBSP and DEC_SET_PPS_RBSP as parameters? Is it the right way to do it using the DecParamSet structure?

3) If the previous method was not the way to do it, how should I give the SPS PPS information to the VPU? Should I prepend the stream with this extradata?

4) Where can I retrieve more information from the RETCODE_FAILURE that tells me nothing. (I read already the VPU_API_RM_L3.0.35_1.1.0.pdf)

5) what happens when there are more than 1 pps data? Doesn't the vpu support more than 1? In the gst-fsl-plugins-3.5.7-1.0.0 example, they use only the first one, if any.

Note: I managed to decode a stream from a camera that did have only SPS information and no PPS information in the stream. But when trying it from a H264 file in avcc format with pps information, it fails, or when receiving a H264 stream from another camera with pps information it fails again.

Labels (4)
0 Kudos
4 Replies

1,542 Views
jeehwanpark
Contributor I

Hi, I'm working on the same problem and there are a couple of things I've got so far, so I'd like to share them.

To tell you the conclusion first, I don't think those APIs work (at least on i.MX6 dual-lite).

I looked into the vpu source codes but couldn't find where and how they are used.

Anyway, let me answer to your questions (those are my opinion, so they might be wrong. If any wrong, please let me know):

1) what is spSaveBuffer for and how big should I make it? What to put in it or what to read out of it? (I know it must be physical memory)

According to the vpu manual(L3.0.35_4.1.0, 09/2013), the original purpose on psSaveBuffer is to save SPS and PPS RBSP (later, PS RBSP) data.

They should big enough to store SPS and PPS data and the doc says 5K is enough in general.

This memory space is not for user, but vpu decoder. "buffer which the decoder saves PS (SPS/PPS) RBSP", the manual says.

In the imx_vpu_test, it doesn't map virtual memory for psSaveBuffer that means it doesn't intend for user.

However, what I've learned is vpu decoder never use psSaveBuffer.

Even though I opened vpu decoder without psSaveBuffer (I set 0), it worked! It didn't even report "notSufficientPsBuffer".

So I looked into source codes and found that vpu writes the address and size of psSaveBuffer passed by user to registers only when cpu is NOT mx6, in vpu_DecGetInitialInfo(). I concluded vpu didn't use psSaveBuffer by them.

Not only that, in vpu_DecOpen(), it adds PS_SAVE_SIZE (defined as 0x080000, vpu_util.h) to buffer size when it allocates physical memory for context buffer.

2) I extract PPS and SPS information from the extradata available in my H264 stream (I already found out how to do it in the case of annex-b format and also in  the avcc format).

    Why am I the only one in the (google) world using the methods vpu_DecGiveCommand with DEC_SET_SPS_RBSP and DEC_SET_PPS_RBSP as parameters? Is it the right way to do it using the DecParamSet structure?

Yes, it is the right way according to the manual, but It doesn't seem to work.

The manual doesn't give any further information such as what the previous and the next steps user should take are.

So I've done several tests; I used H264 raw bitstream data created by vpu encoder (using mxc_vpu_test) as input. It contains PS RBSP once in the start of the file.

First, I put PS RBSP(including start code 0x00000001 for each) to bitstream buffer using memcpy() and called vpu_DecGetInitialInfo(). Then, I fed bitstream data.

This worked.

Second, I put PS RBSP(including start code) to bitstream buffer using memcpy(). Then, I fed first bitstream frame data and called vpu_DecGetInitialInfo(). And then, I fed the rest bitstream data.

This worked, too.

Third, I called vpu_DecGiveCommand() to give PS RBSP(without start code) and vpu_DecGetInitialInfo(). Then, I fed bitstream data.

This case failed when I called vpu_DecGetInitialInfo(). It returned RETCODE_WRONG_CALL_SEQUENCE(-10). vpu also printed this error message, "rd 0x~~, rd reg 0x~~, wr 0x~~, wr reg 0x~~, idx ~~, idx reg ~~", which refers to empty bitstream buffer.

Plus, I never got a chance to feed bitstream data to vpu.

Fourth, I called vpu_DecGiveCommand() to give PS_RBSP(without start code). Then, I fed first bitstream frame data and vpu_DecGetInitialInfo(). And then, I fed the rest bitstream data.

This failed, too. In this case, however, it returned RETCODE_FAILURE and I don't know why it failed. I only assume it was because VpuReadReg(RET_DEC_SEQ_SUCCESS) in vpu_DecGetInitialInfo() returns failure.

By comparison between first, second and third, fourth tests, I concluded vpu_DecGiveCommand() with DEC_SET_SPS_RBSP and DEC_SET_PPS_RBSP doesn't work.

3) If the previous method was not the way to do it, how should I give the SPS PPS information to the VPU? Should I prepend the stream with this extradata?

According to my test, I say yes.

4) Where can I retrieve more information from the RETCODE_FAILURE that tells me nothing. (I read already the VPU_API_RM_L3.0.35_1.1.0.pdf)

I don't know, either. T.T

5) what happens when there are more than 1 pps data? Doesn't the vpu support more than 1? In the gst-fsl-plugins-3.5.7-1.0.0 example, they use only the first one, if any.

I don't know this, too. I've never looked in gst-fsl-plugins. And my requirement doesn't need to support more than 1 pps. So I didn't work on that.

But I'm curious about what case you have more than 1 pps.

Note: I managed to decode a stream from a camera that did have only SPS information and no PPS information in the stream. But when trying it from a H264 file in avcc format with pps information, it fails, or when receiving a H264 stream from another camera with pps information it fails again.

I think, SPS / PPS is unique for settings(size, etc.) so that you cannot use one's SPS / PPS with another's bitstream data.

Moreover, it doesn't seem that vpu supports avcc format.

I hope any of these could help you and anybody who has further info share it.

0 Kudos

1,542 Views
jorenbultheel
Contributor I

Thank you Jeehwan Park, for your answer, I read it the first day but I was still waiting for some extra responses on this post.

Several months later now...

Anyone an idea?

Is this the final answer that it doesn't work?

:smileyshocked:

0 Kudos

1,542 Views
joanxie
NXP TechSupport
NXP TechSupport

since you have the vpu api, you can find the "psSaveBufferSize":

is a size in bytes of a buffer pointed to by psSaveBuffer. This value must be a multiple of 1024. The

maximum size is 65565x1024 bytes.

pls check the parameter "notSufficientPsBuffer":

is a flag that represents whether PS (SPS/PPS) save buffer is sufficient to decode the current

picture. VPU does not get the last part of the current picture stream because of the buffer overflow. The host

VPU requires buffers for saving SPS/PPS and SLICE RBSP when decoding a H.264

stream. In general, 5 Kbytes is sufficient for the SPS/PPS save buffer and a quarter of the raw YUV image size is sufficient

for the SLICE save buffer. If VPU requires more buffer space to decode a H.264 stream, VPU reports a buffer overflow.

0 Kudos

1,542 Views
jorenbultheel
Contributor I

Thank you for the information. Now I understand what those buffers serve for and I do not suspect them anymore for causing my errors since I made them big enough and put no information in them.

My error happens earlier before I could check the "notSufficientPsBuffer" parameter:

my flow is roughly:

OK -vpu_DecOpen(parameters containing psSaveBuffer)

OK -vpu_DecGiveCommand DEC_SET_SPS_RBSP, DEC_SET_PPS_RBSP

OK -vpu_DecSetEscSeqInit(m_decoder_handle, 1);

NOK -vpu_DecGetInitialInfo(info) ---> error happens here RETCODE_FAILURE and info->errorcode = 0x80000000

- vpu_DecSetEscSeqInit(m_decoder_handle, 0);

-vpu_DecRegisterFrameBuffer(slice buffer stuff)

- vpu_DecStartOneFrame

- vpu_DecGetOutputInfo --> here i would be able to check if psSaveBuffer and slice buffers are big enough

The error happens in vpu_DecGetInitialInfo.

It is dependent on the stream I put in:

OK: decode stream from a sony camera with sps and pps information in annex-b format

NOK: decode stream from an axis camera with sps and pps information in annex-b format.

NOK: decode stream form a file with sps and pps information in avcc format. (1080p H264 AAC 25fps 720K big bug bunny movie.)

What does the errorcode 0x80000000 mean? and why do I have RETCODE_FAILURE dependent on the stream I am feeding to the vpu?

0 Kudos