Hardware h264 encoding (S805)

[crashoverride]

Experimental support for C0/C1 H264 video encoding is available here:
https://github.com/OtherCrashOverride/c2_vpcodec/tree/S805

Note that /dev/amvenc_avc will need to have permissions adjusted or "sudo" will need to be used:

Code: Select all

sudo chmod a+rw /dev/amvenc_avc

I have tested it with c2enc and noticed there is some occasional glitching in the bitstream. I have not tested it with c2cap yet.
https://github.com/OtherCrashOverride/c2enc

[edit]
Testing has shown that using all I frames (GOP=0) produces a correct bitstream.

[crashoverride]

I have been investigating the corruption issue. My understanding of things thus far is as follows. There are three components that must work together for encoding: firmware, kernel driver, and user space. Amlogic changes the user space to kernel driver API over time.

The original kernel driver in the repo apparently used an old API. There was a recent commit that updated it to a new API:
https://github.com/hardkernel/linux/commit/b2d20160196c75996ec7da15db23af6518ac495c
However, it appears this updates it to an API that is too new. Somewhere between the driver version we had and that commit is the version of the API the Amlogic provided libraries target.

It would be helpful to know the origin of the code in that commit.

[svpcom]

This is a driver from latest android kernel (3.10) for odroid-c1 provided by Hardkernel. Original driver was outdated and didn't work with userspace API for S805 provided by AMLogic.
But new driver has a problem with corrupted x264 stream in case of fast motion. I've done some debugging and found that corruption happens when we go into "reencode" mode. I've compare API fro S905 and S805 and found that
S905 doesn't use reencode. I've backport changes from S905 to S805 API and it became works.

Ported userspace API for S805:
https://github.com/svpcom/codec_c1

Capture client (i've rewrite it from scratch for my purposes):
https://github.com/svpcom/c1cap

My forum post about c1 port.
viewtopic.php?f=116&t=24136&p=165851

I have been investigating the corruption issue. My understanding of things thus far is as follows. There are three components that must work together for encoding: firmware, kernel driver, and user space. Amlogic changes the user space to kernel driver API over time.

The original kernel driver in the repo apparently used an old API. There was a recent commit that updated it to a new API:
https://github.com/hardkernel/linux/commit/b2d20160196c75996ec7da15db23af6518ac495c
However, it appears this updates it to an API that is too new. Somewhere between the driver version we had and that commit is the version of the API the Amlogic provided libraries target.

It would be helpful to know the origin of the code in that commit.

[crashoverride]

I've backport changes from S905 to S805 API and it became works.

The changes were not published before I started looking into this. If the API is now the same for S905 and S805, we should only need a single code path for both. I will investigate further.

[crashoverride]

I tested that using the master branch of c2_vpcodec instead of the S805 branch produces a good bitstream. Bitrate settings did affect the size of the output; however, I did not confirm they are correct. If someone has the time, I would appreciate their confirming these findings.

TL;DR = for S805 just use the regular c2_vpcodec. No changes are required.

[crashoverride]

A branch of c2cap is now available for the c1:
https://github.com/OtherCrashOverride/c2cap/tree/c1

It requires two kernel patches to work properly:
https://github.com/OtherCrashOverride/l ... 3d56246584
https://github.com/OtherCrashOverride/l ... 4dc5f642e8

I tested it with the master branch of c2_vpcodec compiled on the c1:
https://github.com/OtherCrashOverride/c2_vpcodec

Note that the C1 Cortex-A5 is not as fast as the Cortex-A53 in the C2. This affects MJPEG performance since software libjpeg-turbo is used. Only 10-13 fps were achieved with 1280x720.

I should also note that the Odroid camera exhibits the same problems on C1 as it does on C2.

[edit]
Pull request for the patches is here:
https://github.com/hardkernel/linux/pull/247

[MartinRi]

The reason for the corrupted stream with the s805 branch is the enabled reencoding in combination with prepending of SPS/PPS data to each IDR-Frame.

When using S905 encoder code (gx_enc_fast) only every gop+1 frames will be an IDR-Frame, all the frames inbetween will be P-Frames only - even if an IDR-Frame would be smaller.
With the S805 encoder code (m8_enc_fast) every gop+1 frames will be an IDR-Frame, the other frames will be encoded to P-Frames or - when they take 1.2 times more space than the last IDR-Frame - they are reencoded to IDR-Frames.

Reencoding reduces the size of the data produced by the encoder but takes additional time - so it probably depends on the use case (online/offline encoding) whether size or throughput is the priority.

The corruption occurs because of the way the SPS/PPS gets prepended to each IDR-Frame:
In https://github.com/OtherCrashOverride/c ... c.cpp#L159 the needed space gets reserved on the buffer and in https://github.com/OtherCrashOverride/c ... c.cpp#L181 the data is copied.
The first part depends on the return value of AML_HWSetInput == AMVENC_NEW_IDR, the latter on the type of the NAL produced by AML_HWEncNAL being equal to AVC_NALTYPE_IDR.
AML_HWSetInput only returns AMVENC_NEW_IDR for a) the first frame, b) for every gop+1 frames or c) when forced by some flag value.
But the type of the NAL produced by AML_HWEncNAL is (because of the reencoding) often not as predicted by AML_HWSetInput.
Therefore no space gets reserved on the buffer but the SPS/PPS data is copied anyway which overwrites the first bits of the IDR-Frame.

ffmpeg now reports "Truncating likely oversized PPS" because it can't find the start of the following IDR-Frame and expect it to be still PPS data.
Following P-Frames are now missing their real reference frame and produce bogus values.


A quick and dirty fix to be able to use m8_enc_fast (with reencoding) could be something like https://github.com/MartinRi/c2_vpcodec/ ... 5770f5ee19.
(When repeating the SPS/PPS data is not needed - e.g. when not streaming - one could just try to revert https://github.com/svpcom/codec_c1/blob ... achI.patch.)

[crashoverride]

Thanks for analysis of the issue. The question now is are there any advantages to maintaining the "m8_enc_fast" code path? The only thing I saw in the code that was actually m8 specific was inline asm used for color space conversion:
https://github.com/OtherCrashOverride/c ... p#L41-L118

Since the libvpcodec API only accepts NV12, those code paths are never used. This affords the advantage of having a single code path because "gx_enc_fast" does not use inline asm:
https://github.com/OtherCrashOverride/c ... p#L42-L114

We still need to run some tests comparing the bitstream output using controlled input with c2enc. Ideally, the same input should produce the same output on C1 and C2 when parameters (gop, bitrate) are identical. I would like to test against the "m8_enc_fast" code path too.

[MartinRi]

It looks like the image format is currently hardcoded to AMVENC_NV21 in vl_video_encoder_encode - the img_format argument of vl_video_encoder_init is never used.
But after changing https://github.com/OtherCrashOverride/c ... c.cpp#L139 to AMVENC_RGBA8888 and modifying https://github.com/OtherCrashOverride/c ... #L108-L109 to

Code: Select all

int bufferSize = width * height * 4;

I was able to successfully execute (using the libvpcodec from s805 branch with my fix):

Code: Select all

ffmpeg -i ~/trailer_1080p.mov -an -f rawvideo -pix_fmt rgba - | ./c2enc -w 1920 -h 1080 -f 25 -b 2000 > trailer.h264

[crashoverride]

The preferred method for color space conversion, particularly on Cortex-A5 (S805), is to use GE2D hardware acceleration instead of the CPU:
https://github.com/OtherCrashOverride/c ... #L607-L657

[edit]
The code is also portable between S805 and S905 and does not require re-writing as NEON asm does.

[edit 2]
Fix broken link

Added time stamp support to c2cap c1 branch. New command line option "-t" will overlay date/time text on captured video.

[edit 3]
c2cap c1 branch merged with beta1. Both C1 and C2 will now share the same code.

[odroid]

Pull request for the patches is here:
https://github.com/hardkernel/linux/pull/247

It has been merged and released (-174)
https://github.com/hardkernel/linux/com ... 91e0b6f312

[Pitrson]

Great ! Thansk a million, will give it a try and let you know. Was waiting for this as I have C1 and XU4 only, so no reason to buy C2
Thanks.

[svpcom]

Hi,
I've found that current rate control code (rate_control_gx_fast.cpp) doesn't drop frames if unable to increase QP (maximum supported 51).
For example bitrate became too high ( > 10Mbp/s) if I try to encode PAL (720x576) without input signal (i.e. show screen). According to amlogic sources
last working rate control code with frame drop exists only for m6 (m6_enc/rate_control.cpp) but code for S805/S905 has broken frame drop support.
Do you have any ideas how to fix this?

The preferred method for color space conversion, particularly on Cortex-A5 (S805), is to use GE2D hardware acceleration instead of the CPU:
https://github.com/OtherCrashOverride/c ... #L607-L657

[edit]
The code is also portable between S805 and S905 and does not require re-writing as NEON asm does.

[edit 2]
Fix broken link

Added time stamp support to c2cap c1 branch. New command line option "-t" will overlay date/time text on captured video.

[edit 3]
c2cap c1 branch merged with beta1. Both C1 and C2 will now share the same code.

[nmadrane]

Hi,

I am trying to integrate this hardware encoder into mediastreamer (which is the media framework of linphone). MediaStreamer already has 2 filters for software H264 encoding : one based on libX264 and one based on OpenH264. I just took one of them, deleted all software encoding stuff and called the Amlogic hardware encoder by using the library from :

https://github.com/OtherCrashOverride/c2_vpcodec

The ultimate goal is to have linphone integrated into the box (libreelec), so that I can start video conferences between the box and any other SIP phone. Everything seems to be ok. The encoding starts, it generates a H264 bitstream. I am continuously encoding the same image, just for testing :

test1.jpg

The image used as the input for the encoder

(23.19 KiB) Downloaded 977 times

I am absolutely sure that the H264 bitstream from the hardware encoder is good. I converted it to MOV with ffmpeg and the result is perfect.

The problem is with the RTP transmission. The bitstream has to be split into NAL units. MediaStreamer already has a function for that.

Code: Select all

int outCnt = vl_video_encoder_encode(handle, type, in, in_size, &out);   // perform hardware encoding (i am sure that "out" contains valid H264 data)
ms_h264_bitstream_to_nalus((unsigned char*)out, outCnt, &nalus);         // split into NAL units and store in a queue
rfc3984_pack(packetizer, &nalus, f->outputs[0], timestamp);                 // pack


But the produced NALs are too big for the max UDP MTU size. The RFC3984 packetizer included in MediaStreamer supports mode 0 (no split) and mode 1 (split).

When I use mode 0, it gives warnings about the size of NALs. This is normal, since my MTU is about 1300 and the NAL sizes are about 18 KB to 22 KB for I-Frames, and something like 3 or 4 KB for the others frames. Playing at the other end with a softphone gives a strange result. The top half of the image is ok. The bottom one is corrupt :

output.rar

The video as it is received at the other end, on a softphone

(31.56 KiB) Downloaded 81 times

(please note that the image is flipped down and the color are interpreted as RGB instead of BGR but this is not a big deal, I just have to adjust my RGB->NV12 conversion code).

When I use mode 1, I have a black screen.

I am testing on a LAN, there is only one router/hub between the box and the softphone. I tried several softphones at the receiver end :

With eyebeam, I have the half-frame corruption problem
With jitsi, no video at all (just black screen)
With linphone, no video at all (just black screen)

Do you think my problem is due to fragmentation ? Maybe the softphones I am using do not support packet-mode 1.
Is it possible to tell the hardware encoder what is the max NAL unit size ? That way I could give him the size of my MTU (which is I thing 1300) and force him to give me small NAL units.

I really don't understand why the bottom half of the movie is moving from left to right. It repeats over and over after 10 frames. 10 is the GOP size that I used for encoding. The input image is always the same (I am continuously encoding the same picture), so the output should be a fixed image...

[crashoverride]

Make sure the bitstream format matches what your library expects. If I recall correctly, the bitstream format output by the hardware H264 encoder is Annex B format. You can use a H264 bitstream analyzer to gain more insight.

[schmidtbag]

I don't meant to necropost, but I'm very interested in trying to get this to work. I have the S805 build for c2enc and c2_vpcodec installed, but I don't really understand how to use it [properly].

I'm trying to do live 720p or 1080p 30FPS recording to a file with as little CPU usage as possible, while still compressing the video. I can't figure out how to use c2enc properly.

[mad_ady]

Here's how I'm creating a hw encoded video from a series of pictures:

Code: Select all

/usr/bin/ffmpeg -r $FPS -i ./${BASE}-%*.jpeg -an -c:v rawvideo -r $FPS -pix_fmt nv21 -f rawvideo - | /usr/local/bin/c2enc -w $WIDTH -h $HEIGHT -f $FPS -b $BANDWIDTH | /usr/bin/ffmpeg -r $FPS -thread_queue_size 256 -f h264 -i - -an -c:v copy "$NFS/Webcam2-movies/$DATE/${BASE}-${DATE}.mp4"


[titody]

What setup works in reduce size of video (not resolution)? I try with bitrate but I dont understand bitrate in c2enc ... in ffmpeg , I use v:b 250k and the video lowers to that.

[mad_ady]

c2enc uses -b 250000 to do the same thing

[titody]

c2enc uses -b 250000 to do the same thing

Test it , but always gives 1200kbs and the output same size . Where its the problem? I want to make the video lower megabytes than the original.

[crashoverride]

I have the S805 build for c2enc and c2_vpcodec installed

Use the master branch:

I tested that using the master branch of c2_vpcodec instead of the S805 branch produces a good bitstream. Bitrate settings did affect the size of the output; however, I did not confirm they are correct. If someone has the time, I would appreciate their confirming these findings.

TL;DR = for S805 just use the regular c2_vpcodec. No changes are required.

[titody]

Use the master branch:

I use master branch but I cant reduce bitrate of videos only goes higher than the original.
I input a video of 657kbs bitrate and output goes to 2000...kbs , and the output video is bigger that the orginal .

Where I have to look for get my objective of lower the bitrates?

[crashoverride]

I input a video of 657kbs bitrate and output goes to 2000...kbs , and the output video is bigger that the orginal .

What is the exact command line you are using?

[titody]

Code: Select all

What is the exact command line you are using?

ffmpeg -v warning -i "$r" -an -f rawvideo -pix_fmt nv21 - | c2enc -w $a -h $b -f 25 -b 250000 -g 0 | ffmpeg -r 25 -i - -i "$r" -c:v copy -map 0:v:0 -map 1:a:0 -c:a copy $nn.mkv

variable a and b I enter manually for match the same as input.

If original video have 600..kbs then suppose to have 250kbs aprox..?

[crashoverride]

Increase the value of the "-g 0" argument. Using a "0" value means that all frames are I-Frames which are full frames. Try values of 10 or higher.

[titody]

Increase the value of the "-g 0" argument. Using a "0" value means that all frames are I-Frames which are full frames. Try values of 10 or higher.

Reduce a little bit . I want 250k, not goes down to that .

I found that it works fine if I put higher bitrates and goes up to the number .

What are the uses of this prog , in how many form I can use it??

[crashoverride]

The c2enc options are shown here:
https://github.com/OtherCrashOverride/c ... pp#L20-L24

Code: Select all

   { "width",         required_argument,   NULL,   'w' },
   { "height",         required_argument,   NULL,   'h' },
   { "fps",         required_argument,   NULL,   'f' },
   { "bitrate",      required_argument,   NULL,   'b' },
   { "gop",         required_argument, NULL,   'g' },


width and height must match the image being used including any stride.
fps, bitrate, and gop (group of pictures) are used internally by the encoder library (not c2enc) to adjust the bitstream to maintain the desired bitrate.

[titody]

width and height must match the image being used including any stride.
fps, bitrate, and gop (group of pictures) are used internally by the encoder library (not c2enc) to adjust the bitstream to maintain the desired bitrate.

Why q= or quality always on -1 ?

I still trying to get a low bitrate but its impossible. I had used the rpi omxtx concept tool and give fast results of reduce size but only works with some codecs . But the c2enc you made its works with any codec as input , but not do the job well.

[crashoverride]

Why q= or quality always on -1 ?

There is no "q" or "quality" parameter to c2enc. I am uncertain what is being asked.

c2enc is a "front end" to the hardware codec library. c2enc does not perform any compression itself. It only passes data to the hardware codec API. The amount of compressions depends on the source data and parameters provided and the hardware codec. For a lower bitrate, reduce the video frame size, frame rate, or both.

The hardware codec is a "realtime" encoder. This means it only produces "I" and "P" frames. Because it does not produce "B" frames, it can not compress as aggressively as an 'non realtime' encoder.

[titody]

c2enc is a "front end" to the hardware codec library. c2enc does not perform any compression itself. It only passes data to the hardware codec API. The amount of compressions depends on the source data and parameters provided and the hardware codec. For a lower bitrate, reduce the video frame size, frame rate, or both..

What will be the command line if I want to reduse frame size , all in the same c2enc pipe ?

[crashoverride]

c2enc does not perform any scaling. However, ffmpeg does and is described here:
https://trac.ffmpeg.org/wiki/Scaling

[titody]

c2enc does not perform any scaling. However, ffmpeg does and is described here:
https://trac.ffmpeg.org/wiki/Scaling

Ok , what I want to know is how I use the hw accelerated c2enc inside the ffmpeg pipe for make process more faster?

I am just trying to find some use for c2enc ...

[mad_ady]

There are two examples of ffmpeg + c2enc/c2cap in this tutorial: viewtopic.php?t=23503. It doesn't do scaling, but it could be added in the first ffmpeg command.

[titody]

Someone have used c2enc and make a video of less than 1000kb/s bitrate??