Hardware Acceleration for video processing with FFMPEG?

Post Reply
ophirr
Posts: 6
Joined: Sun Jun 05, 2022 10:24 am
languages_spoken: english
ODROIDs: odroid-c4
Has thanked: 2 times
Been thanked: 0
Contact:

Hardware Acceleration for video processing with FFMPEG?

Post by ophirr »

Hello, I'm attempting to capture 1920x1080 footage from a usb webcam on an ODROID-C4 but I'm running into high cpu usage when doing anything except for -c:v copy. Is it possible to enable ffmpeg hardware acceleration on the ODROID-C4? If so, any information on building ffmpeg with the necessary support would be greatly appreciated. Would love to be able to capture 1920x1080 footage without burning through the CPU. Dumping some example commands / os information below.

OS:

Code: Select all

$ uname -a
Linux odroid1 4.9.277-83 #1 SMP PREEMPT Mon Feb 28 15:16:26 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Camera information:

Code: Select all

$ v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'MJPG' (Motion-JPEG, compressed)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.017s (60.000 fps)
                [snip]
        [1]: 'YUYV' (YUYV 4:2:2)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.167s (6.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.111s (9.000 fps)
                [snip]
Example of ffmpeg commands I'm running:

Code: Select all

# reaches 20 fps, ~5% CPU usage
$ ffmpeg -f v4l2 -re -video_size 1920x1080 -input_format mjpeg -i /dev/video0 -c:v copy raw.mkv

# writing to a local rtsp server using default conversion, can't do more than 6fps without slowing down, 200% CPU usage
$ ffmpeg -f v4l2 -r 6 -video_size 1920x1080 -input_format mjpeg -i /dev/video0 -b:v 4M -maxrate 8M -f rtsp rtsp://127.0.0.1:8554/webcam

# writing to a local rtsp server doing very little compression, reaches 15fps with some latency but has 400% CPU usage
# ffmpeg -f v4l2 -r 15 -video_size 1920x1080 -input_format mjpeg -i /dev/video0 -preset superfast -c:v libx264 -b:v 4M -maxrate 8M -f rtsp rtsp://127.0.0.1:8554/webcam

User avatar
mad_ady
Posts: 10649
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4 (HC1, HC2), C1+, C2, C4 (HC4), N1, N2, H2, Go, Go Advance, M1
Location: Bucharest, Romania
Has thanked: 645 times
Been thanked: 916 times
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by mad_ady »

Currently you can't do hardware encoding with ffmpeg on the C4, but crashoverride's c2cap might work on the c4 and could produce a h264 stream of the video (see https://github.com/OtherCrashOverride/c2cap).

Back on the C2 I used it with ffmpeg, but there was a 5s buffering delay that caused sound to be out of sync. Here are some examples/ideas: https://github.com/mad-ady/odroid-webcam-scripts

Crashoverride stated recently in a different thread that he was working on bringing ffmpeg support to Amlogic, but I don't think it's ready yet.

ophirr
Posts: 6
Joined: Sun Jun 05, 2022 10:24 am
languages_spoken: english
ODROIDs: odroid-c4
Has thanked: 2 times
Been thanked: 0
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by ophirr »

Thanks for the tip. I did my best to get c2cap built along with c2_vpcodec, but running it (and uncommenting some of the pritntfs) shows that there's an error happening within vl_video_encoder_encode in libvpcodec:

Code: Select all

Got a buffer: index = 3
vl_video_encoder_encode = -1
Got a buffer: index = 4
vl_video_encoder_encode = -1
Got a buffer: index = 5
vl_video_encoder_encode = -1
Got a buffer: index = 6
vl_video_encoder_encode = -1
Got a buffer: index = 7
vl_video_encoder_encode = -1
Got a buffer: index = 0
vl_video_encoder_encode = -1
Got a buffer: index = 1
vl_video_encoder_encode = -1
Got a buffer: index = 2
vl_video_encoder_encode = -1
Got a buffer: index = 3
vl_video_encoder_encode = -1
Got a buffer: index = 4
vl_video_encoder_encode = -1
Got a buffer: index = 5
vl_video_encoder_encode = -1
Got a buffer: index = 6
vl_video_encoder_encode = -1
Got a buffer: index = 7
vl_video_encoder_encode = -1
Got a buffer: index = 0
vl_video_encoder_encode = -1
Got a buffer: index = 1
vl_video_encoder_encode = -1
Got a buffer: index = 2
vl_video_encoder_encode = -1
# etc.
I'll try poking around at AML_HWEncoder::AML_HWEncNAL to see if I can find the root cause error. I may be missing something obvious in the build process as well.

EDIT:

I replaced some of the ALOG statements with printfs and got something that looks more like a root error, UNINITIALIZED. Does this mean it won't work with the C4?

Code: Select all

v4l2_format: width=640, height=480, pixelformat=0x56595559
capture.timeperframe: numerator=1, denominator=30
vl_video_encoder_init: width=640, height=480, frame_rate=30, bit_rate=5000000, gop=10
bit_rate:5000000
Video frame size is 640 x 480
AML_HWEncInitialize Fail, error=0. handle: 0x559331d2c0
handle = 367541736128
nv12Size = 1228800
ENCODEC_BUFFER_SIZE = 32768
Got a buffer: index = 0
AML_HWEncNAL Fail: UNINITIALIZED. handle: 0x559331d2c0
Encode SPS and PPS error, encoderStatus = -4. handle: 0x559331d2c0
vl_video_encoder_encode = -1
Got a buffer: index = 1
AML_HWEncNAL Fail: UNINITIALIZED. handle: 0x559331d2c0
Encode SPS and PPS error, encoderStatus = -4. handle: 0x559331d2c0
vl_video_encoder_encode = -1
Got a buffer: index = 2
AML_HWEncNAL Fail: UNINITIALIZED. handle: 0x559331d2c0
Encode SPS and PPS error, encoderStatus = -4. handle: 0x559331d2c0
vl_video_encoder_encode = -1
Got a buffer: index = 3
AML_HWEncNAL Fail: UNINITIALIZED. handle: 0x559331d2c0
Encode SPS and PPS error, encoderStatus = -4. handle: 0x559331d2c0
vl_video_encoder_encode = -1
Got a buffer: index = 4
AML_HWEncNAL Fail: UNINITIALIZED. handle: 0x559331d2c0
Encode SPS and PPS error, encoderStatus = -4. handle: 0x559331d2c0
vl_video_encoder_encode = -1
Got a buffer: index = 5
AML_HWEncNAL Fail: UNINITIALIZED. handle: 0x559331d2c0
Encode SPS and PPS error, encoderStatus = -4. handle: 0x559331d2c0

User avatar
rooted
Posts: 9509
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 761 times
Been thanked: 488 times
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by rooted »

@crashoverride would be the best person to answer this, perhaps he will see this post.

ophirr
Posts: 6
Joined: Sun Jun 05, 2022 10:24 am
languages_spoken: english
ODROIDs: odroid-c4
Has thanked: 2 times
Been thanked: 0
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by ophirr »

One last bit of sleuthing, I see that `InitAMVEncode` is on the call path, and eventually it tries to open up `/dev/amvenc_avc`. That path doesn't seem to exist on the o4, and I confirmed that trying to read it is the cause of failure:

Code: Select all

diff --git a/enc_api.cpp b/enc_api.cpp
index e880f97..8dea911 100644
--- a/enc_api.cpp
+++ b/enc_api.cpp
@@ -197,6 +197,7 @@ AMVEnc_Status InitAMVEncode(amvenc_hw_t* hw_info, int force_mode)
     hw_info->dev_data = NULL;
 
     fd = open(ENCODER_PATH, O_RDWR);
+    printf("TRIED TO OPEN ENCODER_PATH: fd=%d\n", fd);
     if(fd<0){
         return AMVENC_FAIL;
     }

Code: Select all

v4l2_format: width=640, height=480, pixelformat=0x56595559
capture.timeperframe: numerator=1, denominator=30
vl_video_encoder_init: width=640, height=480, frame_rate=30, bit_rate=5000000, gop=10
bit_rate:5000000
Video frame size is 640 x 480
TRIED TO OPEN ENCODER_PATH: fd=-1
Any idea what `/dev/amvenc_avc` is?

crashoverride
Posts: 5672
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 545 times
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by crashoverride »

ophirr wrote:
Tue Jun 07, 2022 7:22 am
Any idea what `/dev/amvenc_avc` is?
Ensure the kernel encoder drivers are loaded:

Code: Select all

$ sudo modprobe encoder
$ sudo modprobe vpu
The "encoder" driver provides /dev/amvenc_avc. The "vpu" driver provides /dev/HevcEnc.

Note you will also need to ensure you have the correct permissions set to access the device:

Code: Select all

sudo chown root:video /dev/amvenc_avc
sudo chown root:video /dev/HevcEnc

ophirr
Posts: 6
Joined: Sun Jun 05, 2022 10:24 am
languages_spoken: english
ODROIDs: odroid-c4
Has thanked: 2 times
Been thanked: 0
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by ophirr »

Further progress! After adding encoder/vpu and confirming that the permissions are set as expected, `c2cap` makes it all the way into `GxFastEncodeSPS_PPS`.

Output:

Code: Select all

-snip-
v4l2_format: width=640, height=480, pixelformat=0x56595559
capture.timeperframe: numerator=1, denominator=30
vl_video_encoder_init: width=640, height=480, frame_rate=30, bit_rate=5000000, gop=10
bit_rate:5000000
Video frame size is 640 x 480
TRIED TO OPEN ENCODER_PATH: fd=5
Should call fast Initialize?
->Calling fast Initialize
ioctl(FASTGX_AVC_IOC_CONFIG_INIT) ret = 0
FAST INIT SETTING p->output_buff.size = 0
AML_HWEncInitialize success, handle: 0x55962012c0, fd:5
handle = 367590904512
nv12Size = 1228800
ENCODEC_BUFFER_SIZE = 32768
Got a buffer: index = 0
AML_HWEncNAL state: 1. handle: 0x55962012c0
FastEncodeSPS_PPS status:8, fd:5
p->sps_len = 13 p->pps_len = 8 p->output_buf.size = 0
->AMVENC_FAIL
gdev[hw_info->dev_id]->EncodeSPS_PPS ret: 0
AML_HWEncNAL state: 1, err=0, handle: 0x55962012c0
Encode SPS and PPS error, encoderStatus = 0. handle: 0x55962012c0
I noticed that `output_buf.size` being equal to 0 within `GxFastEncodeSPS_PPS` was the cause of failure, so I traced that back to `GxInitFastEncode` where it looks like `output_buf.size` is initialized. It looks like buff_info[12] is set to 0 despite the ioctl call to FASTGX_AVC_IOC_CONFIG_INIT. That's about as far as I can debug it tonight, my initial thoughts are that this either means:
- The init params being passed in are incorrect / missing something
- Something is going wrong in the FASTGX_AVC_IOC_CONFIG_INIT request
- Or something obvious I missed

I'd appreciate any tips, and regardless thank you for your time

crashoverride
Posts: 5672
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 545 times
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by crashoverride »

ophirr wrote:
Tue Jun 07, 2022 3:54 pm
I'd appreciate any tips, and regardless thank you for your time
I have tested in the past that encoding for h264 and HEVC both work. This implies the issue resides in the client software. The 'c2cap' program was written for a previous version of the encoder libraries. It needs to be updated for the newer c4/n2 libraries.

Note the library API signature change:
https://github.com/OtherCrashOverride/c ... _1_0.h#L68

Code: Select all

	extern int vl_video_encoder_encode(vl_codec_handle_t handle, vl_frame_type_t type, char *in, int in_size, char **out);
https://github.com/OtherCrashOverride/e ... _1_0.h#L69

Code: Select all

    int vl_video_encoder_encode(vl_codec_handle_t handle, vl_frame_type_t type, unsigned char *in, int in_size, unsigned char *out, int format);
The "format" parameter is processed here:
https://github.com/OtherCrashOverride/e ... #L155-L170

I seem to recall that without the GE2D library enabled, the formats are limited to NV12 or NV21.

crashoverride
Posts: 5672
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 545 times
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by crashoverride »

I found my notes regarding this subject in my archive (2020-08). A lot of research I do never sees publication for various reasons. The blocking issue with my "c4cap" project was audio related. Regarding CPU usage, my notes indicate that hardware MJPEG decoding is too slow for realtime 1080p@60 USB webcam. Software decoding is required and maxes out the CPU. Since the video was of no use without associated audio for my project, I attempted to add capture and compression for that also. This also required using a container format (mkv) rather than a raw bitstream. Ultimately, I was never able to reliably synchronize the captured audio and video. This meant adopting a new audio API that provided timestamps. Work never progressed passed this point due to other projects taking priority.

I am currently modifying FFMPEG for use on ODROID-M1. After that work is complete, I intend to add support for C4/N2. This will obsolete "c4cap" as it will provide both audio synchronization and container support.

[edit]
There is also one other point of technical interest. It is currently not possible to support H264 and HEVC libraries at the same time. This is due to the APIs having identical functions names while residing is separate libraries. This causes name collisions when linking. The encoder libraries will need to be modified to change the API symbol names so they do not collide.
These users thanked the author crashoverride for the post (total 2):
odroid (Wed Jun 08, 2022 11:36 am) • ophirr (Sat Jun 11, 2022 6:08 am)

User avatar
mad_ady
Posts: 10649
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4 (HC1, HC2), C1+, C2, C4 (HC4), N1, N2, H2, Go, Go Advance, M1
Location: Bucharest, Romania
Has thanked: 645 times
Been thanked: 916 times
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by mad_ady »

There is also one other point of technical interest. It is currently not possible to support H264 and HEVC libraries at the same time. This is due to the APIs having identical functions names while residing is separate libraries. This causes name collisions when linking. The encoder libraries will need to be modified to change the API symbol names so they do not collide.
Would LD_PRELOAD allow you to force one encoder versus the other seamlessly, or would that cause missing functions?

crashoverride
Posts: 5672
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 545 times
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by crashoverride »

mad_ady wrote:
Wed Jun 08, 2022 1:36 pm
Would LD_PRELOAD allow you to force one encoder versus the other seamlessly, or would that cause missing functions?
It would amount to an "ugly hack". I believe its better to solve the issue than work around it.

ophirr
Posts: 6
Joined: Sun Jun 05, 2022 10:24 am
languages_spoken: english
ODROIDs: odroid-c4
Has thanked: 2 times
Been thanked: 0
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by ophirr »

Thank you for the responses. When I have a free weekend I'll see if I can tinker with c4cap to get it running and report back the results.

> I am currently modifying FFMPEG for use on ODROID-M1. After that work is complete, I intend to add support for C4/N2.

If I wanted to follow along with that work, what's the best place to monitor? The ffmpeg-devel mailing list?

crashoverride
Posts: 5672
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 545 times
Contact:

Re: Hardware Acceleration for video processing with FFMPEG?

Post by crashoverride »

ophirr wrote:
Sat Jun 11, 2022 6:12 am
If I wanted to follow along with that work, what's the best place to monitor? The ffmpeg-devel mailing list?
All relevant information will be posted on this forum. The code will be hosted on my github.

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 0 guests