FFMPEG HEVC x265 h265 hardware encoding

[unt]

Rockchip RK3568B2, Mali-G52 MP2 GPU supports x265 hardware encoding. See: https://www.96rocks.com/blog/2020/11/28 ... hip-rk3568
'ffmpeg -encoders | grep hevc' returns:
ffmpeg version 89daac5-1ubuntu2+202101121912~focal Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
configuration: --prefix=/usr --extra-version='1ubuntu2+202101121912~focal' --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=
/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --disable-avisynth -
-enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-l
ibcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --e
nable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-libru
bberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enabl
e-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enabl
e-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-l
ibdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 51.100 / 56. 51.100
libavcodec 58. 91.100 / 58. 91.100
libavformat 58. 45.100 / 58. 45.100
libavdevice 58. 10.100 / 58. 10.100
libavfilter 7. 85.100 / 7. 85.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 7.100 / 5. 7.100
libswresample 3. 7.100 / 3. 7.100
libpostproc 55. 7.100 / 55. 7.100
V..... libx265 libx265 H.265 / HEVC (codec hevc)
V..... nvenc_hevc NVIDIA NVENC hevc encoder (codec hevc)
V..... hevc_nvenc NVIDIA NVENC hevc encoder (codec hevc)
V..... hevc_v4l2m2m V4L2 mem2mem HEVC encoder wrapper (codec hevc)
V..... hevc_vaapi H.265/HEVC (VAAPI) (codec hevc)

the only one that worked was libx265 (software encoding), which expectedly is too slow... From what I understand hevc_v4l2m2m is the hardware method to encode in M1, but the test
ffmpeg -i vid0.avi -c:v hevc_v4l2m2m vid1.mp4
returned:
[hevc_v4l2m2m @ 0x5592eaac10] Could not find a valid device
[hevc_v4l2m2m @ 0x5592eaac10] can't configure encoder
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate,
width or height

h264_v4l2m2m also didn't work...

Can someone please help me with that?

thanks

[unt]

I'm using the ubuntu server image from HK.
'sudo lshw -c video' returns nothing... why would that be?

[crashoverride]

Rockchip does not provide a V4L2 M2M interface for their codecs. They use RK MPP for both encoding and decoding.
https://github.com/hardkernel/rk3568_rk_mpp

[unt]

yes I noticed that. I could build and install RockChip MPP (dev) from github rockchip-linux, ffmpeg with options --enable-rkmpp --enable-version3 --enable-libdrm, but ~/bin/ffmpeg -hwaccels:
Hardware acceleration methods:
drm
The encoders x264 and x265:
V....D libx264 libx264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (codec h264)
V....D libx264rgb libx264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 RGB (codec h264)
V..... h264_v4l2m2m V4L2 mem2mem H.264 encoder wrapper (codec h264)
V....D libx265 libx265 H.265 / HEVC (codec hevc)
V..... hevc_v4l2m2m V4L2 mem2mem HEVC encoder wrapper (codec hevc)
What you're saying is hevc_v4l2m2m (which didn't work), is not the right encoder for rk3568? Is it hevc_vaapi? I coudn't really find precise information on it. But I see you sugested the hardkernel repository. FFMPEG from apt wasn't compiled with --enable-rkmpp...

[mad_ady]

See if you get a rkmpp decoder. I could only get decoder support when I tried the same thing for N1.

[crashoverride]

What you're saying is hevc_v4l2m2m (which didn't work), is not the right encoder for rk3568? Is it hevc_vaapi?

Codecs ending with "_v4l2m2m" will not work. Codecs ending with "_vaapi" will not work either. Codecs ending with "_rkmpp" will work.

But I see you sugested the hardkernel repository.

I linked the HardKernel repository because Rockchip has a history of removing repositories from their Github.

FFMPEG from apt wasn't compiled with --enable-rkmpp...

The FFMPEG package is supplied by Ubuntu which focuses on PC hardware. A custom compiled version of FFMPEG is likely required.
https://github.com/JeffyCN/FFmpeg

I can see the Rockchip MPP decoder is supported, but Github truncates the file list so I can not tell whether the encoder is supported or not.

It is also worth mentioning that to use MPP, the correct permissions need to be set on it as detailed in this post.
viewtopic.php?p=347298#p347298

[unt]

The FFMPEG package is supplied by Ubuntu which focuses on PC hardware. A custom compiled version of FFMPEG is likely required.

Yes I wandered about that too, it's logical since rkmpp is very specific to RockChip hardware.

I compiled again using HK's repo, the result was the same, no rkmpp encoders, only decoders:
V..... h264_rkmpp h264 (rkmpp) (codec h264)
V..... hevc_rkmpp hevc (rkmpp) (codec hevc)

Sad since RK3568 supports hevc encoder, which is important to reduce videos sizes while maintaining reasonable quality... So a bit of a misleading advertisement, since the harware supports it, but nobody can use it as there are no drivers available...

[crashoverride]

So a bit of a misleading advertisement, since the harware supports it, but nobody can use it as there are no drivers available...

There are drivers and hardware encode/decode is supported. It is FFMPEG that does not support it. Anybody can use it through the RK MPP library, GStreamer, or Android.

https://github.com/JeffyCN/mirrors/tree ... r-rockchip
https://github.com/JeffyCN/mirrors/blob ... stmppenc.c

[rooted]

Hardware encoding, quality, and small file size don't generally go together in my experience.

[unt]

crashoverride:
There are drivers and hardware encode/decode is supported. It is FFMPEG that does not support it. Anybody can use it through the RK MPP library, GStreamer, or Android.

Ah ok... the limitation is FFMPEG's, thanks for the infos. I'll check GStreamer. The executables generated by MPP like mpi_enc_test seem to be quite limited for someone used fo ffmpeg, maybe it's possible to use it in a pipe mode with ffmpeg.

rooted:
Hardware encoding, quality, and small file size don't generally go together in my experience.

NVidia's nvenc is quick and has done a good job so far, but I imagine different hdw/drivers may be actually quite different in compression quality.

thanks

[rooted]

Yeah I was referring to hardware encoding on SBCs, it's certainly possible as it's done well on my various phones but that's vastly different since the amount of development dollars spent far exceeds what is put into these SBC devices.

[mad_ady]

From my tests on xu4 and c2 the encoder is suitable for transcoding when streaming. Not for long term storage. The encoder is used when shooting video from Android and produces a higher bitrate stream for the same quality level compared to the software encoder (which can do two passes). So I gave up on hardware encoding (on sbcs) for my home video collection (I transcode all my videos to reduce size by up to 10 times for the same perceived quality level).

[martinet]

The FFMPEG package is supplied by Ubuntu which focuses on PC hardware. A custom compiled version of FFMPEG is likely required.

Yes I wandered about that too, it's logical since rkmpp is very specific to RockChip hardware.

I compiled again using HK's repo, the result was the same, no rkmpp encoders, only decoders:
V..... h264_rkmpp h264 (rkmpp) (codec h264)
V..... hevc_rkmpp hevc (rkmpp) (codec hevc)

Sad since RK3568 supports hevc encoder, which is important to reduce videos sizes while maintaining reasonable quality... So a bit of a misleading advertisement, since the harware supports it, but nobody can use it as there are no drivers available...

Hi. Any chance you could point me to instructions how to compile ffmpeg with this 2 decoders? For my use case I need decoders only.

[unt]

martinet wrote:
Hi. Any chance you could point me to instructions how to compile ffmpeg with this 2 decoders? For my use case I need decoders only.

Hi,
I got the basic instructions from https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu
The RKMPP library source code, to be used with ffmpeg .configure with enable-rkmpp, I got from https://github.com/hardkernel/rk3568_rk_mpp, but also tested https://github.com/rockchip-linux/mpp, see comments above. You'll have to adjust paths and variables to your liking:

Code: Select all

FFMPEG_BUILD_BASE=$HOME/inst/ffmpeg
export FFMPEG_BUILD_PATH=$FFMPEG_BUILD_BASE/build
export PKG_CONFIG_PATH=$FFMPEG_BUILD_PATH/lib/pkgconfig
export CMAKE_INSTALL_PREFIX=$FFMPEG_BUILD_PATH

mkdir -p $HOME/bin
mkdir -p $FFMPEG_BUILD_PATH

#Compile RKMPP
MPP_VER=develop
MPP_BASE_DIR=$FFMPEG_BUILD_BASE/rk3568_rk_mpp-rk3568-odroid
MPP_SRC_DIR=$MPP_BASE_DIR/build/linux/aarch64

#or use git
aria2c https://github.com/hardkernel/rk3568_rk_mpp/archive/refs/heads/rk3568/odroid.zip
unzip mpp-$MPP_VER.zip
cd $MPP_SRC_DIR

export CMAKE_INSTALL_PREFIX=$FFMPEG_BUILD_PATH
#Generate build system
./make-Makefiles.bash
#Build using 4 cores
cmake --build . -j4
#Install to $FFMPEG_BUILD_PATH
cmake --install $MPP_SRC_DIR --prefix $FFMPEG_BUILD_PATH
#CMake install didn't change prefix in pkg_config dir, so sedded it
sed -i "s|prefix=/usr/local|prefix=${FFMPEG_BUILD_PATH}|g" $PKG_CONFIG_PATH/rockchip_mpp.pc
sed -i "s|prefix=/usr/local|prefix=${FFMPEG_BUILD_PATH}|g" $PKG_CONFIG_PATH/rockchip_vpu.pc
#Test binaries generated at $FFMPEG_BUILD_PATH/bin
export LD_LIBRARY_PATH=~/inst/ffmpeg/build/lib:$LD_LIBRARY_PATH
#Check utilities, etc
#cd $FFMPEG_BUILD_PATH/bin
#./mpi_dec_test -i ./4k.h264 -o 1080.yuv -w 1920 -h 1080 -t 7

#FFMEG build
cd $FFMPEG_SRC_DIR

#Configure so 'make install' installs to $HOME/bin. If you also want ffplay remove corresponding line below.
./configure \
  --prefix=$FFMPEG_BUILD_PATH \
  --bindir=$HOME/bin \
  --extra-libs="-lpthread -lm" \
  --ld="g++" \
  --disable-ffplay \
  --enable-gpl \
  --enable-nonfree \
  --arch=aarch64 \
  --enable-static --disable-shared \
  --enable-libass \
  --enable-libfdk-aac \
  --enable-libfreetype \
  --enable-libmp3lame \
  --enable-libopus \
  --enable-libvorbis \
  --enable-libvpx \
  --enable-libx264 \
  --enable-libx265 \
  --enable-libvmaf \
  --enable-neon \
  --enable-rkmpp --enable-version3 --enable-libdrm 

#Compile ffmpeg
make -j4
#Install ffmpeg to ~/bin
make install
#You may need to adjust LD_LIBRARY_PATH to include libraries

hope that helps.

[martinet]

Thanks!

[crashoverride]

I started work on adding hardware encoder support for rkmpp to FFMPEG. My proof-of-concept that encodes frames with fixed parameters is working. A lot still remains to be done, but it seems viable.

[mad_ady]

I thought you weren't into ffmpeg...

[crashoverride]

I thought you weren't into ffmpeg...

I did it for the greetz from my fellow developers. V4L2 is a crime against humanity! No developer should ever have to endure it.


So, its not about FFMPEG, its about libavcodec/libavformat. Of all the linux media APIs i have evaluated, it is the winner. After completing the rkmpp implementation, I plan to write an amcodec implementation for N2/C4.

[mad_ady]

Big hugs from me, then, though I'm not a fellow developer...

But wasn't there the issue that the N2/C4 encoder was stateful and v4l2 was stateless (most likely I'm confusing terms)?
Anyway, ffmpeg support would be awesome!
But - it would be limited to stock kernel, right? It wouldn't be useful on mainline...

[crashoverride]

But wasn't there the issue that the N2/C4 encoder was stateful and v4l2 was stateless (most likely I'm confusing terms)?

That is a V4L2-ism. If you are not using V4L2, its irrelevant.

But - it would be limited to stock kernel, right? It wouldn't be useful on mainline...

ffmpeg/libavcodec/libavformat are kernel version independent. If "mainstream" (actually BSP) supports rkmpp/amcodec, it will work. If it supports V4L2, it will work. See the beauty?

[martinet]

martinet wrote:
Hi. Any chance you could point me to instructions how to compile ffmpeg with this 2 decoders? For my use case I need decoders only.

Hi,
I got the basic instructions from https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu
The RKMPP library source code, to be used with ffmpeg .configure with enable-rkmpp, I got from https://github.com/hardkernel/rk3568_rk_mpp, but also tested https://github.com/rockchip-linux/mpp, see comments above. You'll have to adjust paths and variables to your liking:

Code: Select all

FFMPEG_BUILD_BASE=$HOME/inst/ffmpeg
export FFMPEG_BUILD_PATH=$FFMPEG_BUILD_BASE/build
export PKG_CONFIG_PATH=$FFMPEG_BUILD_PATH/lib/pkgconfig
export CMAKE_INSTALL_PREFIX=$FFMPEG_BUILD_PATH

mkdir -p $HOME/bin
mkdir -p $FFMPEG_BUILD_PATH

#Compile RKMPP
MPP_VER=develop
MPP_BASE_DIR=$FFMPEG_BUILD_BASE/rk3568_rk_mpp-rk3568-odroid
MPP_SRC_DIR=$MPP_BASE_DIR/build/linux/aarch64

#or use git
aria2c https://github.com/hardkernel/rk3568_rk_mpp/archive/refs/heads/rk3568/odroid.zip
unzip mpp-$MPP_VER.zip
cd $MPP_SRC_DIR

export CMAKE_INSTALL_PREFIX=$FFMPEG_BUILD_PATH
#Generate build system
./make-Makefiles.bash
#Build using 4 cores
cmake --build . -j4
#Install to $FFMPEG_BUILD_PATH
cmake --install $MPP_SRC_DIR --prefix $FFMPEG_BUILD_PATH
#CMake install didn't change prefix in pkg_config dir, so sedded it
sed -i "s|prefix=/usr/local|prefix=${FFMPEG_BUILD_PATH}|g" $PKG_CONFIG_PATH/rockchip_mpp.pc
sed -i "s|prefix=/usr/local|prefix=${FFMPEG_BUILD_PATH}|g" $PKG_CONFIG_PATH/rockchip_vpu.pc
#Test binaries generated at $FFMPEG_BUILD_PATH/bin
export LD_LIBRARY_PATH=~/inst/ffmpeg/build/lib:$LD_LIBRARY_PATH
#Check utilities, etc
#cd $FFMPEG_BUILD_PATH/bin
#./mpi_dec_test -i ./4k.h264 -o 1080.yuv -w 1920 -h 1080 -t 7

#FFMEG build
cd $FFMPEG_SRC_DIR

#Configure so 'make install' installs to $HOME/bin. If you also want ffplay remove corresponding line below.
./configure \
  --prefix=$FFMPEG_BUILD_PATH \
  --bindir=$HOME/bin \
  --extra-libs="-lpthread -lm" \
  --ld="g++" \
  --disable-ffplay \
  --enable-gpl \
  --enable-nonfree \
  --arch=aarch64 \
  --enable-static --disable-shared \
  --enable-libass \
  --enable-libfdk-aac \
  --enable-libfreetype \
  --enable-libmp3lame \
  --enable-libopus \
  --enable-libvorbis \
  --enable-libvpx \
  --enable-libx264 \
  --enable-libx265 \
  --enable-libvmaf \
  --enable-neon \
  --enable-rkmpp --enable-version3 --enable-libdrm 

#Compile ffmpeg
make -j4
#Install ffmpeg to ~/bin
make install
#You may need to adjust LD_LIBRARY_PATH to include libraries

hope that helps.

One noob question - where did you get libdrm from? I'm trying to build ffmpeg for frigate inside docker container. Original docker file in master seems to be fuc*ed up: https://github.com/blakeblackshear/frig ... rch64#L400 as it points to not existing repository: https://github.com/rockchip-linux/libdrm-rockchip
I removed this entry and after running ffmpeg i get: ffmpeg: error while loading shared libraries: libdrm.so.2: cannot open shared object file: No such file or directory

[crashoverride]

I removed this entry and after running ffmpeg i get: ffmpeg: error while loading shared libraries: libdrm.so.2: cannot open shared object file: No such file or directory

You can install the distribution provided package "libdrm2" on Ubuntu:

Code: Select all

$ apt-cache search libdrm2
libdrm2 - Userspace interface to kernel DRM services -- runtime

[crashoverride]

The rkmpp encoder is now in a usable state. I have tested realtime compression from a webcam with the following:

Code: Select all

./ffmpeg -hide_banner -f video4linux2 -input_format mjpeg -s 1280x720 -framerate 30 -i /dev/video9  -vcodec h264_rkmpp_enc -b:v 1M -maxrate 1M -g 24 output.mp4

I should point out that there is an issue with FFMPEG and the CSI camera making it unusable at the current time:

Code: Select all

$ ffmpeg -hide_banner -f video4linux2 -list_formats all -i /dev/video0
[video4linux2,v4l2 @ 0x55a83cf360] Not a video capture device.
/dev/video0: No such device

FFMPEG checks for the capability V4L2_CAP_VIDEO_CAPTURE but the rkisp driver only has V4L2_CAP_VIDEO_CAPTURE_MPLANE
https://github.com/JeffyCN/FFmpeg/blob/ ... 4l2.c#L174
https://github.com/hardkernel/linux/blo ... re.c#L1423

It then goes on to check the "channel" of the capture device, but I could not find either VIDIOC_G_INPUT or VIDIOC_S_INPUT implemented in the rkisp driver:
https://github.com/JeffyCN/FFmpeg/blob/ ... #L853-L868

After hacking FFMPEG to ignore both of the above, it still would not capture any video and simply faults.

[rooted]

I'm grateful to hear this, too bad it wouldn't just work with the camera as well.

Thank you for taking on the task.

[unt]

martinet wrote:
Hi. Any chance you could point me to instructions how to compile ffmpeg with this 2 decoders? For my use case I need decoders only.

Hi,
I got the basic instructions from https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu
The RKMPP library source code, to be used with ffmpeg .configure with enable-rkmpp, I got from https://github.com/hardkernel/rk3568_rk_mpp, but also tested https://github.com/rockchip-linux/mpp, see comments above. You'll have to adjust paths and variables to your liking:

Code: Select all

FFMPEG_BUILD_BASE=$HOME/inst/ffmpeg
export FFMPEG_BUILD_PATH=$FFMPEG_BUILD_BASE/build
export PKG_CONFIG_PATH=$FFMPEG_BUILD_PATH/lib/pkgconfig
export CMAKE_INSTALL_PREFIX=$FFMPEG_BUILD_PATH

mkdir -p $HOME/bin
mkdir -p $FFMPEG_BUILD_PATH

#Compile RKMPP
MPP_VER=develop
MPP_BASE_DIR=$FFMPEG_BUILD_BASE/rk3568_rk_mpp-rk3568-odroid
MPP_SRC_DIR=$MPP_BASE_DIR/build/linux/aarch64

#or use git
aria2c https://github.com/hardkernel/rk3568_rk_mpp/archive/refs/heads/rk3568/odroid.zip
unzip mpp-$MPP_VER.zip
cd $MPP_SRC_DIR

export CMAKE_INSTALL_PREFIX=$FFMPEG_BUILD_PATH
#Generate build system
./make-Makefiles.bash
#Build using 4 cores
cmake --build . -j4
#Install to $FFMPEG_BUILD_PATH
cmake --install $MPP_SRC_DIR --prefix $FFMPEG_BUILD_PATH
#CMake install didn't change prefix in pkg_config dir, so sedded it
sed -i "s|prefix=/usr/local|prefix=${FFMPEG_BUILD_PATH}|g" $PKG_CONFIG_PATH/rockchip_mpp.pc
sed -i "s|prefix=/usr/local|prefix=${FFMPEG_BUILD_PATH}|g" $PKG_CONFIG_PATH/rockchip_vpu.pc
#Test binaries generated at $FFMPEG_BUILD_PATH/bin
export LD_LIBRARY_PATH=~/inst/ffmpeg/build/lib:$LD_LIBRARY_PATH
#Check utilities, etc
#cd $FFMPEG_BUILD_PATH/bin
#./mpi_dec_test -i ./4k.h264 -o 1080.yuv -w 1920 -h 1080 -t 7

#FFMEG build
cd $FFMPEG_SRC_DIR

#Configure so 'make install' installs to $HOME/bin. If you also want ffplay remove corresponding line below.
./configure \
  --prefix=$FFMPEG_BUILD_PATH \
  --bindir=$HOME/bin \
  --extra-libs="-lpthread -lm" \
  --ld="g++" \
  --disable-ffplay \
  --enable-gpl \
  --enable-nonfree \
  --arch=aarch64 \
  --enable-static --disable-shared \
  --enable-libass \
  --enable-libfdk-aac \
  --enable-libfreetype \
  --enable-libmp3lame \
  --enable-libopus \
  --enable-libvorbis \
  --enable-libvpx \
  --enable-libx264 \
  --enable-libx265 \
  --enable-libvmaf \
  --enable-neon \
  --enable-rkmpp --enable-version3 --enable-libdrm 

#Compile ffmpeg
make -j4
#Install ffmpeg to ~/bin
make install
#You may need to adjust LD_LIBRARY_PATH to include libraries

hope that helps.

One noob question - where did you get libdrm from? I'm trying to build ffmpeg for frigate inside docker container. Original docker file in master seems to be fuc*ed up: https://github.com/blakeblackshear/frig ... rch64#L400 as it points to not existing repository: https://github.com/rockchip-linux/libdrm-rockchip
I removed this entry and after running ffmpeg i get: ffmpeg: error while loading shared libraries: libdrm.so.2: cannot open shared object file: No such file or directory

Hi martinet,
As a general rule when building from source, the .configure script will need -dev libraries, there are libs without -dev for many reasons, some of them are used as tools, not necessarily to be linked to. So libdrm needed won't be satisfied by libdrm2 (I tested it), but libdrm-dev works fine.
Cheers

[unt]

The rkmpp encoder is now in a usable state. I have tested realtime compression from a webcam with the following:

Code: Select all

./ffmpeg -hide_banner -f video4linux2 -input_format mjpeg -s 1280x720 -framerate 30 -i /dev/video9  -vcodec h264_rkmpp_enc -b:v 1M -maxrate 1M -g 24 output.mp4

I should point out that there is an issue with FFMPEG and the CSI camera making it unusable at the current time:

Code: Select all

$ ffmpeg -hide_banner -f video4linux2 -list_formats all -i /dev/video0
[video4linux2,v4l2 @ 0x55a83cf360] Not a video capture device.
/dev/video0: No such device

FFMPEG checks for the capability V4L2_CAP_VIDEO_CAPTURE but the rkisp driver only has V4L2_CAP_VIDEO_CAPTURE_MPLANE
https://github.com/JeffyCN/FFmpeg/blob/ ... 4l2.c#L174
https://github.com/hardkernel/linux/blo ... re.c#L1423

It then goes on to check the "channel" of the capture device, but I could not find either VIDIOC_G_INPUT or VIDIOC_S_INPUT implemented in the rkisp driver:
https://github.com/JeffyCN/FFmpeg/blob/ ... #L853-L868

After hacking FFMPEG to ignore both of the above, it still would not capture any video and simply faults.

Hi crashoverride,
I couldn't get any ffmpeg x264, hevc rkmpp mpp encoders made available. I tested 2 sources for ffmpeg (githubs ffmpeg.org and JeffyCN_FFmpeg), and 2 rkmpp (githubs rockchip-linux/mpp, hardkernel/rk3568_rk_mpp). I looked into the rkmpp code and saw x264 abd x265 encoders are implemented, but still no luck, they don't show when .configure shows the options and after making also not. Would you please tell us which have you used, settings and so on?
thanks

[martinet]

martinet wrote:
Hi. Any chance you could point me to instructions how to compile ffmpeg with this 2 decoders? For my use case I need decoders only.

Hi,
I got the basic instructions from https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu
The RKMPP library source code, to be used with ffmpeg .configure with enable-rkmpp, I got from https://github.com/hardkernel/rk3568_rk_mpp, but also tested https://github.com/rockchip-linux/mpp, see comments above. You'll have to adjust paths and variables to your liking:

Code: Select all

FFMPEG_BUILD_BASE=$HOME/inst/ffmpeg
export FFMPEG_BUILD_PATH=$FFMPEG_BUILD_BASE/build
export PKG_CONFIG_PATH=$FFMPEG_BUILD_PATH/lib/pkgconfig
export CMAKE_INSTALL_PREFIX=$FFMPEG_BUILD_PATH

mkdir -p $HOME/bin
mkdir -p $FFMPEG_BUILD_PATH

#Compile RKMPP
MPP_VER=develop
MPP_BASE_DIR=$FFMPEG_BUILD_BASE/rk3568_rk_mpp-rk3568-odroid
MPP_SRC_DIR=$MPP_BASE_DIR/build/linux/aarch64

#or use git
aria2c https://github.com/hardkernel/rk3568_rk_mpp/archive/refs/heads/rk3568/odroid.zip
unzip mpp-$MPP_VER.zip
cd $MPP_SRC_DIR

export CMAKE_INSTALL_PREFIX=$FFMPEG_BUILD_PATH
#Generate build system
./make-Makefiles.bash
#Build using 4 cores
cmake --build . -j4
#Install to $FFMPEG_BUILD_PATH
cmake --install $MPP_SRC_DIR --prefix $FFMPEG_BUILD_PATH
#CMake install didn't change prefix in pkg_config dir, so sedded it
sed -i "s|prefix=/usr/local|prefix=${FFMPEG_BUILD_PATH}|g" $PKG_CONFIG_PATH/rockchip_mpp.pc
sed -i "s|prefix=/usr/local|prefix=${FFMPEG_BUILD_PATH}|g" $PKG_CONFIG_PATH/rockchip_vpu.pc
#Test binaries generated at $FFMPEG_BUILD_PATH/bin
export LD_LIBRARY_PATH=~/inst/ffmpeg/build/lib:$LD_LIBRARY_PATH
#Check utilities, etc
#cd $FFMPEG_BUILD_PATH/bin
#./mpi_dec_test -i ./4k.h264 -o 1080.yuv -w 1920 -h 1080 -t 7

#FFMEG build
cd $FFMPEG_SRC_DIR

#Configure so 'make install' installs to $HOME/bin. If you also want ffplay remove corresponding line below.
./configure \
  --prefix=$FFMPEG_BUILD_PATH \
  --bindir=$HOME/bin \
  --extra-libs="-lpthread -lm" \
  --ld="g++" \
  --disable-ffplay \
  --enable-gpl \
  --enable-nonfree \
  --arch=aarch64 \
  --enable-static --disable-shared \
  --enable-libass \
  --enable-libfdk-aac \
  --enable-libfreetype \
  --enable-libmp3lame \
  --enable-libopus \
  --enable-libvorbis \
  --enable-libvpx \
  --enable-libx264 \
  --enable-libx265 \
  --enable-libvmaf \
  --enable-neon \
  --enable-rkmpp --enable-version3 --enable-libdrm 

#Compile ffmpeg
make -j4
#Install ffmpeg to ~/bin
make install
#You may need to adjust LD_LIBRARY_PATH to include libraries

hope that helps.

One noob question - where did you get libdrm from? I'm trying to build ffmpeg for frigate inside docker container. Original docker file in master seems to be fuc*ed up: https://github.com/blakeblackshear/frig ... rch64#L400 as it points to not existing repository: https://github.com/rockchip-linux/libdrm-rockchip
I removed this entry and after running ffmpeg i get: ffmpeg: error while loading shared libraries: libdrm.so.2: cannot open shared object file: No such file or directory

Hi martinet,
As a general rule when building from source, the .configure script will need -dev libraries, there are libs without -dev for many reasons, some of them are used as tools, not necessarily to be linked to. So libdrm needed won't be satisfied by libdrm2 (I tested it), but libdrm-dev works fine.
Cheers

Hi unt.

Thank you for your reply. I managed to build it already inside frigates container, but have no luck running it with my camera stream.

Code: Select all

frigate-custom  | [2022-05-09 14:47:58] watchdog.office                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   : [flv @ 0x5599967ec0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   : [h264_rkmpp @ 0x55999cbfb0] Failed to send packet to decoder (code = -11)
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   :     Last message repeated 183 times
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   : Finishing stream 1:0 without any data written to it.
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   : [swscaler @ 0x55999f2c90] [swscaler @ 0x5599a01740] deprecated pixel format used, make sure you did set range correctly
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   : [swscaler @ 0x55999f2c90] [swscaler @ 0x5599990450] deprecated pixel format used, make sure you did set range correctly
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   : [swscaler @ 0x55999f2c90] [swscaler @ 0x5599a1dde0] deprecated pixel format used, make sure you did set range correctly
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   : [swscaler @ 0x55999f2c90] [swscaler @ 0x5599a2c630] deprecated pixel format used, make sure you did set range correctly
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   : [swscaler @ 0x55999f2c90] [swscaler @ 0x5599969900] deprecated pixel format used, make sure you did set range correctly
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   : [flv @ 0x5599967ec0] Failed to update header with correct duration.
frigate-custom  | [2022-05-09 14:47:58] ffmpeg.office.detect           ERROR   : [flv @ 0x5599967ec0] Failed to update header with correct filesize.

I'm wondering, if any device from /dev/ should be made available for docker container for rkmpp decoder to run properly?

Also, when installing libdrm-dev the following additional packages were installed:

Code: Select all

Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libdrm-amdgpu1 libdrm-etnaviv1 libdrm-freedreno1 libdrm-nouveau2
  libdrm-radeon1 libdrm-tegra0 libpciaccess-dev libpciaccess0
Suggested packages:
  pciutils
The following NEW packages will be installed:
  libdrm-amdgpu1 libdrm-dev libdrm-etnaviv1 libdrm-freedreno1 libdrm-nouveau2
  libdrm-radeon1 libdrm-tegra0 libpciaccess-dev libpciaccess0
0 upgraded, 9 newly installed, 0 to remove and 1 not upgraded.
Need to get 261 kB of archives.

I'm wondering if libdrm-rockchip shouldn't be installed? like the one from: https://oph.mdrjr.net/madady/odroidn1/d ... -rockchip/

I checked out commit from this ticket: https://github.com/rockchip-linux/mpp/issues/272 and also no luck with the same error.

[crashoverride]

I tested 2 sources for ffmpeg (githubs ffmpeg.org and JeffyCN_FFmpeg), and 2 rkmpp (githubs rockchip-linux/mpp, hardkernel/rk3568_rk_mpp).

The source I have tested is JeffyCN_FFmpeg:
https://github.com/JeffyCN/FFmpeg

The rkmpp I am using is the one provided by HardKernel in the official image:

Code: Select all

$ apt-cache search librockchip-mpp-dev
librockchip-mpp-dev - Media Process Platform

I couldn't get any ffmpeg x264, hevc rkmpp mpp encoders made available.

Make sure you are using the version you compiled and not the distribution provided one:
System version

Code: Select all

$ ffmpeg
ffmpeg version 89daac5-1ubuntu2+202101121912~focal Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/usr --extra-version='1ubuntu2+202101121912~focal' --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --disable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

Compiled version

Code: Select all

$ ./ffmpeg
ffmpeg version N-104475-g39576c97bc Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --bindir=./bin --extra-libs='-lpthread -lm' --ld=g++ --disable-ffplay --enable-gpl --enable-nonfree --arch=aarch64 --enable-static --disable-shared --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libx264 --enable-libx265 --enable-neon --enable-rkmpp --enable-version3 --enable-libdrm
  libavutil      57.  7.100 / 57.  7.100
  libavcodec     59. 12.101 / 59. 12.101
  libavformat    59.  8.100 / 59.  8.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8. 16.100 /  8. 16.100
  libswscale      6.  1.100 /  6.  1.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

[crashoverride]

I'm wondering, if any device from /dev/ should be made available for docker container for rkmpp decoder to run properly?

The hardware codec device needs to be available for rkmpp to work:

Code: Select all

$ ls -l /dev/mpp_service 
crw-rw---- 1 root video 244, 0 May  6 04:17 /dev/mpp_service

[crashoverride]

Since many USB webcams use MJPEG, I added support for hardware decoding of it. In RKMPP, MJPEG uses an entirely different method than the other codecs, so I had to author a new FFMPEG decoder for it. Also of interest is that it would not output YUV420P like the other codecs. It only seems to support NV12 as a YUV format in addition to several RGB formats.

I tested using a webcam with hardware MJPEG decoding and hardware H264 encoding with the following command:

Code: Select all

$ ./ffmpeg -hide_banner -f video4linux2 -input_format mjpeg -s 1280x720 -framerate 60 -vcodec mjpeg_rkmpp -i /dev/video9  -vcodec h264_rkmpp_enc -b:v 1M -minrate 500K -maxrate 5M -g 24 output.mp4

It appears to work as expected, and a significant difference in CPU load can be observed versus using the software MJPEG decoder built into FFMPEG.

[mad_ady]

Awesome! Great work!

[crashoverride]

Some notes of interest:

I attempted to add MPEG2 hardware decoding. However, it fails in rkmpp. It uses the same decoder code already available, so a future version of the library may allow it work.

I attempted to add HEVC hardware encoding. However, it also also fails due to issues with rkmpp. While "AnnexB" can be disabled for the H264 encoder, it is hard-coded into the HEVC encoder. FFMPEG requires the use of "Extra Data" instead of AnnexB. The header issue will either need to be fixed in rkmpp or bitstream processing to extract it will need to be added to my code.

Buffers will always require an additional copy. I spent a lot of time trying to optimize buffer usage. However, there is a bug in the kernel MPP service driver that panics the kernel every time (I really love having a RESET button!). Since it should not be possible to panic the system from user space, this is considered a driver bug. There may be additional issues with RKMPP when attempting to setup a "Normal" buffer group.

The inability to use the CSI camera with FFMPEG is not a driver bug. From all the research I have done on the issue, the driver appears to implement a V4L2 MPLANE API only. This is even the case with the mainline kernel ISP driver. User space applications (like FFMPEG) need to be updated with MPLANE support. It appears that FFMPEG has never worked on any Rockchip SoC that has a CSI interface because of this. Additionally, it will not work with the mainline kernel. All the documentation for CSI use on Rockchip platforms have been with GStreamer instead of FFMPEG likely for this reason.