h.264 decoder holding last few frames

Moderators: mdrjr, odroid

h.264 decoder holding last few frames

Unread postby Jeremy » Sat Jan 27, 2018 10:17 am

I am working on a project that utilizes the hardware h.264 decoding and rendering full-screen on an Odroid-C2. This project receives h.264 NAL units in real-time from an external encoder over the network, but if there is no delta change in the source frame (example, a slide show), no frames are sent after the slide settles. This is normally OK, except that I am noticing that the decoder pipeline appears to be keeping a few frames in the queue and not rendering them.

As soon as the slide changes, and more frames get pushed into the pipeline, it continues as expected, until the next slide settles. The net effect though, is that the user is basically looking at the second to last slide, and not the most current frame that was sent into the decoder. I have been able to verify this by storing the frames to disk, and observing the frames in an external decoder.

I have checked the aml_codec library for a "flush" command, and have come up short. I have also tried to enable the "FreeRun" mode, and that does not "appear" to have any effect on the problem.

Does anyone know of a way around this problem?
Jeremy
 
Posts: 2
Joined: Sat Jan 27, 2018 10:04 am
languages_spoken: english
ODROIDs: C2, XU4

Re: h.264 decoder holding last few frames

Unread postby phaseshifter » Sat Jan 27, 2018 11:56 am

@ odroid pls move this thread to c-2 issues sub forum
Odroids and an amd 64 bit x86 machine..
phaseshifter
 
Posts: 2915
Joined: Fri May 08, 2015 9:12 am
languages_spoken: english
ODROIDs: N-1..c1+ ..c-2..xu3 lite,xu4...u-3..
u-2...other odroid acc`s as well

Re: h.264 decoder holding last few frames

Unread postby crashoverride » Sat Jan 27, 2018 1:45 pm

This is an issue with the encoder, not the decoder. Whether the source frame data changes or not, the encoder should be sending a continuous stream of data (real time). The decoder can not continue because it does not know what the next frame (I, P, B) is. In other words, this is correct operation for the decoder.

[edit]
Sending a "key" frame (I) in the encoder will cause the decoder to "flush".
crashoverride
 
Posts: 3926
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1

Re: h.264 decoder holding last few frames

Unread postby odroid » Sun Jan 28, 2018 10:37 am

phaseshifter wrote:@ odroid pls move this thread to c-2 issues sub forum


Moved.
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: h.264 decoder holding last few frames

Unread postby _BG_ » Sun Feb 04, 2018 9:45 am

Hi @ crashoverride,

Thanks for the response. I am working on the same issue and would like to get an idea for possible workarounds for it.

In this case, actually we do not have the option to modify encoder behaviour, however we know that the encoder only sends I and P frames. So theoretically it shall be possible to decode each frame without waiting for a future reference.

But from other topics like viewtopic.php?f=135&t=22418 I understand that 1in-1out decoding is not supported by Odroid C2. So we would still like to explore our options here.

One option that comes to mind, is for us to mimic a continuous-like stream ourselves i.e. filling in the gaps with dummy / manually generated P frames (maybe with zero motion vectors or using a previous P frame with updating necessary references / frame_nums also in the future frames etc) .

Do you think this could be a viable option? Have you ever tried anything similar?

Another option I think is, since you have mentioned the decoder is holding frames since it does not know about the next frames, I assume they are being stored in a Decoded Picture Buffer, as possible short-term or long-term reference frames.

If this is the case, then they should be reached with the vfm frame grabber that you have mentioned in other posts. From what I understood , I believe that is not a supported tool but do you think that it is worth to try?

Thanks a lot for your input
_BG_
 
Posts: 4
Joined: Fri Feb 02, 2018 12:16 pm
languages_spoken: english
ODROIDs: C2

Re: h.264 decoder holding last few frames

Unread postby crashoverride » Sun Feb 04, 2018 2:46 pm

If you are only sending I and P frames, they should be decoded as soon as received. It is B frames that break the "1 in, 1 out"' decoder operation.

I would recommend using a H264 protocol analyzer to verify that only I and P frames are present. If this is the case, then its likely there is data that is sitting in an application buffer somewhere that does not make it to the decoder. Check the application strategy for feeding the codec to ensure its sending all the data it has received.
crashoverride
 
Posts: 3926
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1

Re: h.264 decoder holding last few frames

Unread postby _BG_ » Mon Feb 05, 2018 12:48 am

Yes, already done that with both ffprobe and h264 bit stream analyzer, definitely only I and P frames like seen below.

--------------------------------------------------------------------------
Frame POC Pic# QP SnrY SnrU SnrV Y:U:V Time(ms)
--------------------------------------------------------------------------
00000(IDR) 0 0 28 4:2:0 21
00001( P ) 2 1 30 4:2:0 9
00002( P ) 4 2 27 4:2:0 14
00003( P ) 6 3 21 4:2:0 15
00004( P ) 8 4 21 4:2:0 12
00005( P ) 10 5 17 4:2:0 10
...

I guess the fact that POC increments by 2 consistenly means each frame is a reference frame so each P frame is only referencing the previous P frame as you would expect in a realtime stream.

We have verified the application is not buffering the frames, the analysis is done by recording the frames that are definitely sent to codec_write() .

I can see by the codec_get_vbuf_state() that the decoder buffer is being consumed (if I interpret it correctly. The data_len value returned by vbuf is changing)

However the frame is not shown on the screen.

One also interesting fact is that the codec_get_vpts() also does not increment, so either that value is not the last decoded picture pts or it is not incemented until shown on the screen. Not much info there.

I have seen in one of your example applications (I guess it was c2play) that you utilize a different API codec_checkin_pts() before feeding the frames. I tried using the packet PTS we have from container demuxer with that API but without any improvement.

By the way thanks a ton for writing these applications, they are almost the only documentation on how the HW decoder works.
_BG_
 
Posts: 4
Joined: Fri Feb 02, 2018 12:16 pm
languages_spoken: english
ODROIDs: C2

Re: h.264 decoder holding last few frames

Unread postby crashoverride » Mon Feb 05, 2018 12:46 pm

_BG_ wrote:you utilize a different API codec_checkin_pts() before feeding the frames.

C2play directly calls the codec IOCTLS, bypassing amlibs. This is done because amlibs also sets display properties causing side effects in c2play. If "freerun" mode is used, as indicated in the opening post, then PTS (presentation time stamp) is not relevant since its ignored.

If the application has been verified to send all data to the decoder, then it should also be verified that 1) the encoder has sent all data and is not buffer it, 2) the NAL units are well formed (start codes, etc), and 3) the data represents frames and not "slices". Does the problem occur when using a PC or other device for reception?

[edit]
It is abnormal for a "real time" encoder to stop sending data. This suggests the encoder may be buffering its output to satisfy its GOP (group of pictures) settings.
crashoverride
 
Posts: 3926
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1

Re: h.264 decoder holding last few frames

Unread postby _BG_ » Tue Feb 06, 2018 7:05 pm

Hi,

Yes freerun mode is set. I agree this is not a usual encoder behaviour but I believe encoder is implemented in a way to save the amount of transmitted data when there is a stationary image i.e. a slide show going on.

I do not have a PC playback test setup but the encoder with another device works fine so encoder seems to be sending enough data for the frame to be rendered on the screen.
(The application SW on that device is also different but I believe this test at least ensures we have enough data at h264 frame level to render the frame)

Also in our test setups, when we record the received frames as raw h264 and then manually export the frames , we can actually obtain the last frames that were not shown on the screen. And we are sure these very same frames were written to HW decoder by codec_write()

It is almost as if there is a 4-5 frame buffered somewhere in the decoder, and I cannot find any setting/API to access or manage that behaviour.
_BG_
 
Posts: 4
Joined: Fri Feb 02, 2018 12:16 pm
languages_spoken: english
ODROIDs: C2

Re: h.264 decoder holding last few frames

Unread postby crashoverride » Tue Feb 06, 2018 8:24 pm

Hardware decoders typically have limited resources. Because of this, they tend to not hold on to any buffers longer than required. I have not observed the C1/C2 hold on to any buffers. Particularly relevant is the experiment I did with the C2 hardware encoder (real time I, P) with a web cam. I did not encounter any "stuck" frames.

A possible experiment may be to "filter" the incoming bitstream through FFMPEG. Ensure you are specifying "Annex B" output. I have used this method in the past to "correct" bitstreams from problematic sources. FFMPEG will also report errors in the bitstream in verbose mode.
crashoverride
 
Posts: 3926
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1


Return to Issues

Who is online

Users browsing this forum: No registered users and 2 guests