[Howto] Odroid XU3/4 using the hardware encoder

by **jwrona** » Fri May 19, 2017 7:38 pm

memeka wrote:how about when using clutterautovideosink ?

EDIT: you don't need v4l2video5convert

Thanks, clutterautovideosink works, but its laggy and probably uses Open GL:

Code: Select all: libGL error: unable to load driver: exynos_dri.so libGL error: driver pointer missing libGL error: failed to load driver: exynos

by **allanm84** » Tue May 23, 2017 5:32 am

Has anyone been able to stream JPEG from the oCam and convert it? I'm attempting to take in 1080p mjpeg from the ocam, run it through v4l2video3videodec, then encode it h.264 with v4l2video11h264enc. Unfortunately the videodec fails with a memory error, but jpegdec works.

The following pipeline works just fine:

Code: Select all: gst-launch-1.0 v4l2src device=/dev/video6 ! \ image/jpeg,width=1920 ,height=1080 ,framerate=30/1 ! \ jpegdec ! videoconvert ! \ video/x-raw,format=NV12 ! \ v4l2video11h264enc extra-controls="encode,h264_level=10,h264_profile=1,frame_level_rate_control_enable=1,video_bitrate=$bitrate" ! \ rtph264pay ! \ udpsink port=$port host=$host_ip sync=false

But this fails:

Code: Select all: gst-launch-1.0 v4l2src device=/dev/video6 ! \ image/jpeg,width=1920 ,height=1080 ,framerate=30/1 ! \ v4l2video3videodec ! \ video/x-raw,format=NV12 ! \ v4l2video11h264enc extra-controls="encode,h264_level=10,h264_profile=1,frame_level_rate_control_enable=1,video_bitrate=$bitrate" ! \ rtph264pay ! \ udpsink port=$port host=$host_ip sync=false

With the following error:

Code: Select all: ERROR: from element /GstPipeline:pipeline0/v4l2video3videodec:v4l2video3videodec0: Failed to allocate required memory. Additional debug info: gstv4l2videodec.c(711): gst_v4l2_video_dec_handle_frame (): /GstPipeline:pipeline0/v4l2video3videodec:v4l2video3videodec0: Buffer pool activation failed Execution ended after 0:00:00.205629164

I played around with the ouput-io-mode and capture-io-mode for the decoder and encoder, but that didn't seem to do anything. I also threw some queues in there to no avail. Any help?

by **mad_ady** » Wed May 24, 2017 5:29 pm

Ok, I'm doing something wrong, and I can't figure out what.
I'm using the hardware encoder with the following command to produce an mp4 file:

Code: Select all: ffmpeg -y -i input.ts -map 0:0 -codec:v h264 -b:v 1500k -pix_fmt nv21 -vf 'yadif, scale=trunc(iw/64)*64:trunc(ih/64)*64' -acodec copy A.mp4

The resulting file plays with smplayer and vlc under linux, but can't be opened with gstreamer-based players. Once playback has started I can jump around in the video without issues. One strange thing is that when I open the file again with SMplayer or VLC and they try to resume from where I left off, I get a black image and sound plays fine. The image never recovers (like when receiving an I frame). If I try to play this file on Android with MXPlayer - I can play the file with software decoding only. Other players display a black image with audio only.

I also tried to cut a part of the video (strip out commercials) with:

Code: Select all: ffmpeg -i A.mp4 -vcodec copy -acodec copy -ss 4:20 -t 20:00 B.mp4

The resulting file does not render video in any player - just a black screen with sound.

Here are some mediainfo/ffmpeg outputs of A.mp4:

Code: Select all: mediainfo /DataVolume/TVRecordings/prelucrate/20170520072503_.mp4 General Complete name : /DataVolume/TVRecordings/prelucrate/20170520072503_.mp4 Format : MPEG-4 Format profile : Base Media Codec ID : isom (isom/iso2/avc1/mp41) File size : 404 MiB Duration : 35mn 1s Overall bit rate : 1 611 Kbps Encoded date : UTC 1904-01-01 00:00:00 Tagged date : UTC 1904-01-01 00:00:00 Writing application : Lavf57.25.100 Video ID : 1 Format : AVC Format/Info : Advanced Video Codec Format profile : High@L3.2 Format settings, CABAC : No Format settings, ReFrames : 1 frame Format settings, GOP : M=1, N=12 Codec ID : avc1 Codec ID/Info : Advanced Video Coding Duration : 35mn 1s Bit rate : 1 477 Kbps Width : 704 pixels Height : 576 pixels Display aspect ratio : 16:9 Original display aspect ratio : 1.222 Frame rate mode : Constant Frame rate : 25.000 fps Standard : PAL Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 8 bits Scan type : Progressive Bits/(Pixel*Frame) : 0.146 Stream size : 370 MiB (92%) Encoded date : UTC 1904-01-01 00:00:00 Tagged date : UTC 1904-01-01 00:00:00 Audio ID : 2 Format : MPEG Audio Format version : Version 1 Format profile : Layer 2 Codec ID : 6B Duration : 35mn 1s Bit rate mode : Constant Bit rate : 128 Kbps Channel(s) : 2 channels Sampling rate : 48.0 KHz Compression mode : Lossy Stream size : 32.1 MiB (8%) Language : Romanian Default : Yes Alternate group : 1 Encoded date : UTC 1904-01-01 00:00:00 Tagged date : UTC 1904-01-01 00:00:00 ffmpeg -i /DataVolume/TVRecordings/prelucrate/20170520072503_.mp4 ffmpeg version 3.0.2-1ubuntu4 Copyright (c) 2000-2016 the FFmpeg developers built with gcc 5.4.0 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) 20160609 configuration: --prefix=/usr --extra-version=1ubuntu4 --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --cc=cc --cxx=g++ --enable-gpl --enable-shared --d isable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --en able-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg -- enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora -- enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc139 4 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-chromaprint --enable-libx264 WARNING: library configuration mismatch avcodec configuration: --prefix=/usr --extra-version=1ubuntu4 --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --cc=cc --cxx=g++ --enable-gpl --enabl e-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable- libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-li bopenjpeg --enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable- libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --ena ble-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-chromaprint --enable-libx264 --enable-version3 --disable-doc --disable-programs --disable-avdevice --disable-avfilter --disable-avf ormat --disable-avresample --disable-postproc --disable-swscale --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libvo_amrwbenc libavutil 55. 17.103 / 55. 17.103 libavcodec 57. 24.102 / 57. 24.102 libavformat 57. 25.100 / 57. 25.100 libavdevice 57. 0.101 / 57. 0.101 libavfilter 6. 31.100 / 6. 31.100 libavresample 3. 0. 0 / 3. 0. 0 libswscale 4. 0.100 / 4. 0.100 libswresample 2. 0.101 / 2. 0.101 libpostproc 54. 0.100 / 54. 0.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/DataVolume/TVRecordings/prelucrate/20170520072503_.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf57.25.100 Duration: 00:35:01.56, start: 0.000000, bitrate: 1610 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 704x576, 1477 kb/s, SAR 16:11 DAR 16:9, 25 fps, 25 tbr, 12800 tbn, 25600 tbc (default) Metadata: handler_name : VideoHandler Stream #0:1(rum): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, s16p, 127 kb/s (default) Metadata: handler_name : SoundHandler At least one output file must be specified

Any ideas what I'm doing wrong?

by **memeka** » Wed May 24, 2017 7:35 pm

@mad_ady

I noticed with ffmpeg, if you comment out

Code: Select all: SET_V4L_EXT_CTRL(value, V4L2_CID_MPEG_VIDEO_HEADER_MODE, V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME

or if you set it to the default V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE, the resulting video is not correct (players don't recognize the codec)...

i have no idea why tho`

gstreamer encoder by default uses V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE and all is good...

so maybe your issue has something to do with this setting?

EDIT: with ffmpeg encoder transcoding in emby (with V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME), I can resume video in emby player (ffmpeg to rstp streaming) without issues.

by **mad_ady** » Wed May 24, 2017 7:40 pm

I'll take a look, thanks!

Edit: I've checked the code and it seems to be there:

Code: Select all: SET_V4L_EXT_CTRL(value, V4L2_CID_MPEG_VIDEO_HEADER_MODE, V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME, V4L2_CTRL_CLASS_MPEG, "header mode");

I'll recompile ffmpeg from your Git just in case I'm running something older...

by **memeka** » Wed May 24, 2017 8:57 pm

don't think it will work.
i think that since header mode is "joined with 1st frame", cutting the beginning will lose the header...
so you need to recompile ffmpeg with header mode "separate" ... the issue is that for some reason, in this case, ffmpeg will not create the header ... no idea why...

i would recommend to "play" and "re-encode" the video w/o the commercials. such that the 1st frame will be remade with header joined...

by **mad_ady** » Wed May 24, 2017 9:28 pm

I understand - but the thing is - even if the uncut clip which has the first frame intact has problems playing back. I'll need to scratch my head further...

I could trim the file before encoding it. I'll give that a try as well.

by **memeka** » Wed May 24, 2017 9:31 pm

you can always use gstreamer encoder

by **mad_ady** » Wed May 24, 2017 9:36 pm

I'm not at my wit's end yet

by **memeka** » Thu Jun 01, 2017 8:59 am

emby released with support for exynos mfc encoding: viewtopic.php?f=98&t=26390&p=191996#p191996

by **tweekzilla** » Sun Jul 23, 2017 7:12 am

I'm trying to motion to work with hardware encoding using ffmpeg. How do you enable the v4l2_m2m in the master head of the ffmpeg git repository? I can't seem to find the configuration option to do that

by **mad_ady** » Sun Jul 23, 2017 1:28 pm

It's not on the master branch, but on a subbranch.

by **ArraY80** » Mon Jul 24, 2017 1:32 am

I have Ubuntu mate 16.04.02 LTS
With this kernel:

Last login: Sun Jul 23 15:39:25 2017 from 84.78.24.207
root@odroid:~# uname -a
Linux odroid 4.9.35-48 #1 SMP PREEMPT Thu Jul 6 14:35:57 UTC 2017 armv7l armv7l armv7l GNU/Linux
root@odroid:~# apt-get upgrade
Leyendo lista de paquetes... Hecho
Creando árbol de dependencias
Leyendo la información de estado... Hecho
Calculando la actualización... Hecho
Los siguientes paquetes se han retenido:
linux-image-xu3
0 actualizados, 0 nuevos se instalarán, 0 para eliminar y 1 no actualizados.

1) Do I have to compile the kernel again or with that version I do not have to do anything and just install ffmpeg or gstreamer?

2) Is the installation of ffmpeg correct with apt-get install ffmpeg or do I have to manually compile and install ffmpeg?

3)And for gstreamer os equal instalación?

4) Do I have to install the retained package linux-image-xu3 before?

mad_ady wrote:Regular users on the forum might already know that memeka has managed to use the hardware video encoder on the XU4 by using a mainline kernel (4.8.x) . Here are the steps on how you can achieve the same thing:

Before you begin - this is a tutorial designed for medium/advanced users and if things go wrong you might have an unbootable system. It might be wise to make a backup before beginning: http://forum.odroid.com/viewtopic.php?t=22930 (shameless plug)

1. Install a mainline kernel that supports the MFC encoder. The instructions here use @elatllat's branch (https://github.com/Dmole/linux.git), but also mdrjr is working on a branch of his own (https://github.com/mdrjr/linux) and so is memeka (https://github.com/mihailescu2m/linux/c ... dxu3-4.9.y). You can use my kernel config (which tries to add all of HardKernel's modules), but you can modify it to fit your needs http://pastebin.com/7YnakKmP, or you can use whatever defconfig comes with the kernel you chose.
*** NOTE: the kernel does not support HMP (big.LITTLE extensions) and treats all cores as equal. This is because HMP patches are unstable and can lock up the system. Consequently, overall system performance will be lower than when using Kernel 3.10 ***
*** NOTE2: Also, for XU4 sound over HDMI will no longer work with this kernel! ***

I have added the kernel compilation procedure here for convenience, but you should read/discuss the official kernel compilation thread here: http://forum.odroid.com/viewtopic.php?f=95&t=23163 or this guide: https://github.com/umiddelb/armhf/wiki/ ... ARM-device

a) Make a copy of your existing kernel, initrd, dtb and boot.ini:
Code: Select all
$ cd /media/boot $ sudo -i # cp zImage zImage-3 # cp uInitrd uInitrd-3 # cp exynos5422-odroidxu3.dtb exynos5422-odroidxu3-3.dtb # cp boot.ini boot3.ini

Edit boot3.ini to point to the newly copied files by appending -3 to the name of the zImage, uInitrd and dtb. By replacing the stock boot.ini with this modified boot.ini you will be able to boot your old kernel in case of problems.

b) Download the new kernel
Code: Select all
$ sudo apt-get -y install bc curl gcc git libncurses5-dev lzop make u-boot-tools dos2unix $ git clone --depth 1 -b odroidxu4-mihailescu2m-4.9 https://github.com/Dmole/linux.git linux $ cd linux/ $ make odroidxu4_defconfig

Optionally get my config which supports most of the USB peripherals (TV tuners, sound cards, wifi cards) and a lot of networking modules (LXC support, VLANs, iptables).
Code: Select all
$ wget http://pastebin.com/raw/7YnakKmP -O .config $ dos2unix .config

c) Next compile the kernel
Code: Select all
$ make menuconfig $ make -j 8 zImage dtbs modules $ kver=`make kernelrelease` $ sudo cp arch/arm/boot/zImage arch/arm/boot/dts/exynos5422-odroidxu[34].dtb /media/boot $ sudo cp .config /media/boot/config $ sudo make INSTALL_MOD_STRIP=1 modules_install $ sudo make firmware_install $ sudo cp .config /boot/config-${kver} $ cd /boot $ sudo update-initramfs -c -k ${kver} $ sudo mkimage -A arm -O linux -T ramdisk -a 0x0 -e 0x0 -n initrd.img-${kver} -d initrd.img-${kver} uInitrd-${kver} $ sudo cp uInitrd-${kver} /media/boot/uInitrd

Next modify /media/boot/boot.ini and load exynos5422-odroidxu4.dtb instead of exynos5422-odroidxu3.dtb and save boot.ini.

d) Shut down and unplug HDMI and power. Without this step when you boot with the new kernel you will not have USB3 bus and onboard networking! This is needed only when switching between 3.x and 4.x kernels as far as I've seen.

Optionally - if you intend to run with a custom kernel from now on it's best to mark linux-image-xu3, linux-headers-xu3 and bootini as manual so that they won't be overwritten on subsequent package updates:
Code: Select all
sudo apt-mark hold linux-image-xu3 linux-headers-xu3 bootini

Using FFMPEG
2. Now that the kernel is ready, compile/install a custom ffmpeg which supports the hardware encoder
a) Compile your own ffmpeg
Code: Select all
#debian build tools $ sudo apt-get install build-essential fakeroot devscripts libchromaprint-dev librubberband-dev libjs-bootstrap #get the patched ffmpeg version $ git clone -b v4l2_m2m-3.0.2 --depth=1 https://github.com/mihailescu2m/FFmpeg.git # install ffmpeg's build dependencies (~190 packages) $ sudo apt-get build-dep ffmpeg #build ffmpeg as deb packages with no checks (some checks fail) $ cd FFmpeg/ $ DEB_BUILD_OPTIONS="nocheck" debuild -b -uc -us

When building is done you should have 23 deb packages one level up (in the same directory as FFmpeg). You can download these packages pre-compiled from here: http://www.mediafire.com/?l39zeyy4r172m

b) Install the packages:
Code: Select all
$ cd ../ $ sudo dpkg -i *.deb

This should replace any system-installed ffmpeg build with this ffmpeg which supports the hardware encoder.

If you plan on keeping using these packages you should mark them as manual so that they are not overwritten by future system updates:
Code: Select all
sudo apt-mark hold ffmpeg-doc ffmpeg libav-tools libavcodec-dev libavcodec-extra57 libavcodec-extra libavcodec57 libavdevice-dev libavdevice57 libavfilter-dev libavfilter6 libavformat-dev libavformat57 libavresample-dev libavresample3 libavutil-dev libavutil55 libpostproc-dev libpostproc54 libswresample-dev libswresample2 libswscale-dev libswscale4

3. Use the hardware encoder when transcoding (examples):
Note, depending on what you want to transform with ffmpeg you might get better or worse framerate. For example, changing the output fps of a video cuts off about 20-30fps from encoding time. The examples below do not change the fps.
Code: Select all
# encode video only, about 50fps (max) $ ffmpeg -i big_buck_bunny_720p_h264.mov -acodec aac -vcodec h264 -b:v 2M -pix_fmt nv21 bbb.mp4 # encode video only, about 110fps (max) $ ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -r 24 -i - -an -vcodec h264 -b:v 2M -profile:v 10 -pix_fmt nv21 bbb.mp4 #encode video and audio, about 75fps (max) $ ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -r 24 -i - -i big_buck_bunny_720p_h264.mov -map 0:v:0 -vcodec h264 -b:v 2M -profile:v 10 -pix_fmt nv21 -map 1:a:0 -c:a:1 aac bbb.mp4

The encoder automatically selects h264_v4l2m2m which does the hardware encoding:
Code: Select all
Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_v4l2m2m)) Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))

If you get the error:
Code: Select all
[h264_v4l2m2m @ 0xf3fa0] H264 codec detected, init annexb converter [h264_v4l2m2m @ 0xf3fa0] Device path not set, probing /dev/video* [h264_v4l2m2m @ 0xf3fa0] exynos-gsc.1:m2m is not the one we want [h264_v4l2m2m @ 0xf3fa0] exynos-gsc.0:m2m is not the one we want [h264_v4l2m2m @ 0xf3fa0] s5p-mfc-dec is not the one we want [h264_v4l2m2m @ 0xf3fa0] Could not find a valid device

Make sure the user you are running ffmpeg with is part of the video group:
Code: Select all
odroid@odroid:~$ id uid=1000(odroid) gid=1000(odroid) groups=1000(odroid),4(adm),20(dialout),24(cdrom),27(sudo),30(dip),44(video),46(plugdev),115(lpadmin),116(lightdm)

Sample output:
Code: Select all
adrianp@odroid:~> ffmpeg -i Sintel.2010.720p.mkv -acodec aac -vcodec h264 -b:v 2M -pix_fmt nv21 sintel-encoded.mp4 ffmpeg version 3.0.2-1ubuntu4 Copyright (c) 2000-2016 the FFmpeg developers built with gcc 5.4.0 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.2) 20160609 configuration: --prefix=/usr --extra-version=1ubuntu4 --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --cc=cc --cxx=g++ --enable-gpl --enable-shared --d isable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg -- enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora -- enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc139 4 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-chromaprint --enable-libx264 WARNING: library configuration mismatch avcodec configuration: --prefix=/usr --extra-version=1ubuntu4 --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --cc=cc --cxx=g++ --enable-gpl --enabl e-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable- libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-li bopenjpeg --enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable- libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --ena ble-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-chromaprint --enable-libx264 --enable-version3 --disable-doc --disable-programs --disable-avdevice --disable-avfilter --disable-avf ormat --disable-avresample --disable-postproc --disable-swscale --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libvo_amrwbenc libavutil 55. 17.103 / 55. 17.103 libavcodec 57. 24.102 / 57. 24.102 libavformat 57. 25.100 / 57. 25.100 libavdevice 57. 0.101 / 57. 0.101 libavfilter 6. 31.100 / 6. 31.100 libavresample 3. 0. 0 / 3. 0. 0 libswscale 4. 0.100 / 4. 0.100 libswresample 2. 0.101 / 2. 0.101 libpostproc 54. 0.100 / 54. 0.100 Input #0, matroska,webm, from 'Sintel.2010.720p.mkv': Metadata: encoder : libebml v1.0.0 + libmatroska v1.0.0 creation_time : 2011-04-24 17:20:33 Duration: 00:14:48.03, start: 0.000000, bitrate: 6071 kb/s Chapter #0:0: start 0.000000, end 103.125000 Metadata: title : Chapter 01 Chapter #0:1: start 103.125000, end 148.667000 Metadata: title : Chapter 02 Chapter #0:2: start 148.667000, end 349.792000 Metadata: title : Chapter 03 Chapter #0:3: start 349.792000, end 437.208000 Metadata: title : Chapter 04 Chapter #0:4: start 437.208000, end 472.075000 Metadata: title : Chapter 05 Chapter #0:5: start 472.075000, end 678.833000 Metadata: title : Chapter 06 Chapter #0:6: start 678.833000, end 744.083000 Metadata: title : Chapter 07 Chapter #0:7: start 744.083000, end 888.032000 Metadata: title : Chapter 08 Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709/unknown/unknown), 1280x544, SAR 1:1 DAR 40:17, 24 fps, 24 tbr, 1k tbn, 48 tbc Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s Metadata: title : AC3 5.1 @ 640 Kbps Stream #0:2(ger): Subtitle: subrip Stream #0:3(eng): Subtitle: subrip Stream #0:4(spa): Subtitle: subrip Stream #0:5(fre): Subtitle: subrip Stream #0:6(ita): Subtitle: subrip Stream #0:7(dut): Subtitle: subrip Stream #0:8(pol): Subtitle: subrip Stream #0:9(por): Subtitle: subrip Stream #0:10(rus): Subtitle: subrip Stream #0:11(vie): Subtitle: subrip Codec AVOption preset (Configuration preset) specified for output file #0 (sintel-encoded.mp4) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some encoder which was not actually used for any stream. File 'sintel-encoded.mp4' already exists. Overwrite ? [y/N] y [h264_v4l2m2m @ 0xf3fe0] H264 codec detected, init annexb converter [h264_v4l2m2m @ 0xf3fe0] Device path not set, probing /dev/video* [h264_v4l2m2m @ 0xf3fe0] exynos-gsc.1:m2m is not the one we want [h264_v4l2m2m @ 0xf3fe0] exynos-gsc.0:m2m is not the one we want [h264_v4l2m2m @ 0xf3fe0] Using device /dev/video1 Output #0, mp4, to 'sintel-encoded.mp4': Metadata: encoder : Lavf57.25.100 Chapter #0:0: start 0.000000, end 103.125000 Metadata: title : Chapter 01 Chapter #0:1: start 103.125000, end 148.667000 Metadata: title : Chapter 02 Chapter #0:2: start 148.667000, end 349.792000 Metadata: title : Chapter 03 Chapter #0:3: start 349.792000, end 437.208000 Metadata: title : Chapter 04 Chapter #0:4: start 437.208000, end 472.075000 Metadata: title : Chapter 05 Chapter #0:5: start 472.075000, end 678.833000 Metadata: title : Chapter 06 Chapter #0:6: start 678.833000, end 744.083000 Metadata: title : Chapter 07 Chapter #0:7: start 744.083000, end 888.032000 Metadata: title : Chapter 08 Stream #0:0(eng): Video: h264 (h264_v4l2m2m) ([33][0][0][0] / 0x0021), yuv420p, 1280x544 [SAR 1:1 DAR 40:17], q=2-31, 2000 kb/s, 24 fps, 12288 tbn, 24 tbc Metadata: encoder : Lavc57.24.102 h264_v4l2m2m Stream #0:1(eng): Audio: aac (LC) ([64][0][0][0] / 0x0040), 48000 Hz, 5.1(side), fltp, 341 kb/s Metadata: title : AC3 5.1 @ 640 Kbps encoder : Lavc57.24.102 aac Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_v4l2m2m)) Stream #0:1 -> #0:1 (ac3 (native) -> aac (native)) Press [q] to stop, [?] for help [h264_v4l2m2m @ 0xf3fe0] Performing useless memcpy() on output pool because buffers do not match [h264_v4l2m2m @ 0xf3fe0] This could be avoided by using av_v4l_buffer_pool_get_buffer*() or av_v4l_buffer_pool_make_pipe() [mp4 @ 0xb9c70] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly [mp4 @ 0xb9c70] Encoder did not produce proper pts, making some up. [h264_v4l2m2m @ 0xf3fe0] No event occurred while waiting.01 bitrate=36881.3kbits/s speed=1.84x frame= 2027 fps= 44 q=-0.0 Lsize= 383559kB time=00:01:25.20 bitrate=36877.0kbits/s speed=1.84x video:379964kB audio:3551kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.011593%

Known issues:

* All known issues have been solved. Yay!

Using gstreamer
2. Compile gstreamer from memeka's branch
Code: Select all
$ apt-get install gstreamer1.0-plugins-bad $ git clone https://github.com/mihailescu2m/gst-plugins-good $ cd gst-plugins-good/ $ sudo apt-get install libgstreamer1.0-dev libgudev-1.0-dev libgstreamer-plugins-base1.0-dev dh-autoreconf automake autoconf libtool autopoint cdbs gtk-doc-tools libshout3-dev libaa1-dev libflac-dev libdv4-dev libdv-dev libgtk-3-dev libtag1-dev libsoup2.4-dev gstreamer1.0-doc gstreamer1.0-plugins-base-doc $ dpkg-buildpackage -us -uc -b -j4 $ cd ../ $ sudo dpkg -i gstreamer*.deb

Alternatively you can get the pre-built deb packages from here: https://www.mediafire.com/folder/5lz2po ... -odroidxu4
If you intend to keep these packages installed, mark them as manual so that they are not updated by apt:
Code: Select all
sudo apt-mark hold gstreamer1.0-plugins-good-dbg gstreamer1.0-plugins-good-doc gstreamer1.0-plugins-good gstreamer1.0-pulseaudio libgstreamer-plugins-good1.0-0 libgstreamer-plugins-good1.0-dev

To transcode something here's what you can try:
Identify which is the encoding and the decoding interfaces (they change after boot)
Code: Select all
$ decoder=`gst-inspect-1.0 | grep v4l2 | grep videodec | cut -d ":" -f 2` $ encoder=`gst-inspect-1.0 | grep v4l2 | grep h264enc | cut -d ":" -f 2` $ gst-launch-1.0 filesrc location=big_buck_bunny_720p_h264.mov ! qtdemux ! h264parse ! $decoder ! $encoder extra-controls="encode,h264_level=10,h264_profile=4,frame_level_rate_control_enable=1,video_bitrate=2097152" ! h264parse ! matroskamux ! filesink location=bbb.mkv

by **mad_ady** » Mon Jul 24, 2017 3:03 am

The kernel should have everything you need for mfc encoding. No need for a custom kernel anymore. But you still need a custom gstreamer or ffmpeg. If you don't want to compile it try installing the packages I link to in the first post!

by **tweekzilla** » Thu Jul 27, 2017 10:50 am

mad_ady wrote:It's not on the master branch, but on a subbranch.

I guess what I was asking was is the code now folded into the ffmpeg main repository - i.e. I can compile the latest version of ffmpeg with some configure flags or is it only that specific branch in the forked repository

by **mad_ady** » Thu Jul 27, 2017 11:37 am

As far as I know it's still a specific branch

by **jongerenchaos** » Wed Aug 16, 2017 1:49 am

Is hardware encoding with VP8/9 also possible with these packages?

by **odroid** » Wed Aug 16, 2017 3:38 am

There is no VP8/9 hardware accelerator in Exynos-5422.

by **jongerenchaos** » Wed Aug 16, 2017 5:51 pm

odroid wrote:There is no VP8/9 hardware accelerator in Exynos-5422.

On the website http://wiki.webmproject.org/zz-obsolete/devices is the chipset also vp8/vp9 encoding ready. Is it only not supported yet in the software?

by **odroid** » Thu Aug 17, 2017 12:19 am

I can't find Exynos-5422 in the list of VP9 capable SoCs.
http://wiki.webmproject.org/hardware/socs

by **ard** » Tue Aug 29, 2017 7:40 pm

odroid wrote:I can't find Exynos-5422 in the list of VP9 capable SoCs.
http://wiki.webmproject.org/hardware/socs

But it is in the vp8 list.
The mfc @ exynos 5 can do vp8. Exynos 7 and higher also support vp9.
I also saw v4l2vp8enc in the gstreamer master branch.

by **wiley** » Tue Sep 05, 2017 2:30 pm

So I've been trying to get a Logitech BRIO webcam (USB 3.0 + 4K or 1080p60) to work with the hardware encoding in the XU4. I've been working with gstreamer, and so far the best I can do is this:

gst-launch-1.0 v4l2src device=/dev/video0 ! \
'video/x-raw,width=1280,height=720,framerate=30/1,format=NV12' ! \
v4l2video11h264enc extra-controls="encode,video_bitrate=15728640" ! \
h264parse ! \
mux. \
alsasrc device=hw:1 ! \
"audio/x-raw, format=S16LE, rate=48000,channels=2" ! \
voaacenc bitrate=128000 ! \
aacparse ! \
audio/mpeg,mpegversion=4,stream-format=raw ! \
queue ! flvmux streamable=true name=mux ! \
progressreport ! \
rtmpsink location="<youtube stream server> live=true"

This works, the result looks pretty nice, but it's just 720p30.

For reference, the gstreamer output looks like this:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstAudioSrcClock
Redistribute latency...
Redistribute latency...
WARNING: from element /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Can't record audio fast enough
Additional debug info:
gstaudiobasesrc.c(847): gst_audio_base_src_create (): /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
Dropped 33600 samples. This is most likely because downstream can't keep up and is consuming samples too slowly.
progressreport0 (00:00:05): 4 seconds
progressreport0 (00:00:10): 9 seconds
progressreport0 (00:00:15): 14 seconds
progressreport0 (00:00:20): 19 seconds
progressreport0 (00:00:25): 24 seconds
progressreport0 (00:00:30): 29 seconds
progressreport0 (00:00:35): 34 seconds
progressreport0 (00:00:40): 39 seconds
progressreport0 (00:00:45): 44 seconds
progressreport0 (00:00:50): 49 seconds
progressreport0 (00:00:55): 54 seconds
progressreport0 (00:01:00): 59 seconds
progressreport0 (00:01:05): 64 seconds
progressreport0 (00:01:10): 69 seconds
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:01:14.060786061
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

and the kernel output looks like this:

Sep 05 05:22:37 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000
Sep 05 05:22:37 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000
Sep 05 05:22:37 odroid kernel: vidioc_g_parm:1721: Setting FPS is only possible for the output queue
Sep 05 05:22:37 odroid kernel: s5p-mfc 11000000.codec:: Encoding not initialised
Sep 05 05:22:38 odroid kernel: vidioc_g_parm:1721: Setting FPS is only possible for the output queue
Sep 05 05:23:51 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000

If I try the same pipeline with 1080p30, I get this instead:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstAudioSrcClock
Redistribute latency...
Redistribute latency...
WARNING: from element /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Can't record audio fast enough
Additional debug info:
gstaudiobasesrc.c(847): gst_audio_base_src_create (): /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
Dropped 35520 samples. This is most likely because downstream can't keep up and is consuming samples too slowly.
ERROR: from element /GstPipeline:pipeline0/v4l2video11h264enc:v4l2video11h264enc0: Failed to process frame.
Additional debug info:
gstv4l2videoenc.c(577): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline0/v4l2video11h264enc:v4l2video11h264enc0:
Maybe be due to not enough memory or failing driver
Execution ended after 0:00:11.420726313
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

and the kernel output looks like:

Sep 05 05:27:13 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000
Sep 05 05:27:13 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000
Sep 05 05:27:13 odroid kernel: vidioc_g_parm:1721: Setting FPS is only possible for the output queue
Sep 05 05:27:13 odroid kernel: s5p-mfc 11000000.codec:: Encoding not initialised
Sep 05 05:27:14 odroid kernel: vidioc_g_parm:1721: Setting FPS is only possible for the output queue
Sep 05 05:27:24 odroid kernel: s5p_mfc_watchdog:160: Time out during waiting for HW
Sep 05 05:27:24 odroid kernel: s5p_mfc_watchdog_worker:178: Driver timeout error handling
Sep 05 05:27:24 odroid kernel: vidioc_dqbuf:1284: Call on DQBUF after unrecoverable error
Sep 05 05:27:24 odroid kernel: vidioc_dqbuf:1284: Call on DQBUF after unrecoverable error
Sep 05 05:27:24 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000
Sep 05 05:27:24 odroid kernel: s5p_mfc_handle_error:471: Interrupt Error: 0000001e
Sep 05 05:27:24 odroid kernel: s5p_mfc_close_mfc_inst:472: Err returning instance

Any idea what's going wrong here? The MFC codec is supposed to be capable of 1080p60, right?

by **memeka** » Tue Sep 05, 2017 2:47 pm

try first w/o audio
if that works, try another audio encoding or another audio format

by **wiley** » Tue Sep 05, 2017 3:02 pm

memeka wrote:try first w/o audio
if that works, try another audio encoding or another audio format

I went ahead and tried just running the video to a testsink:

gst-launch-1.0 v4l2src device=/dev/video0 ! \
'video/x-raw,width=1920,height=1080,framerate=30/1,format=NV12' ! \
v4l2video11h264enc extra-controls="encode,video_bitrate=15728640" ! \
h264parse ! \
testsink

Unfortunately, I see the same MFC timeout and error.

I just tried two other things that seem to conflict, though:
- If I cut out the encoder completely and just run raw camera frames to the testsink, it works and I get reasonable CPU usage, which seems to imply the USB bus bandwidth isn't the problem (which makes sense - at 12 bits per pixel, this is 1920*1080*30*12 = 747 megabits/sec over a 5 gigabit bus).
- If I replace the v4l2src with a videotestsrc, the entire pipeline works. This seems to imply that the camera is the problem after all.

Now I'm really confused...

by **wiley** » Tue Sep 05, 2017 4:36 pm

Well, if I assume the previous problem was, in fact, due to bandwidth constraints, the next logical step is to switch to MJPEG (which is needed for 1080p60 anyway). Looking into that, I can software-decode 60 jpegs per second, but the software decoder can't do NV12/NV21, so I have to do a colorspace conversion. This lowers the FPS down to ~12 (and switching to the hardware converter only gets ~14).

I should be able to get around this by using the hardware jpeg decoder to go straight to NV12, but I get this behavior

wiley@odroid:~$ gst-launch-1.0 videotestsrc ! \
> 'video/x-raw,width=1920,height=1080,framerate=60/1' ! \
> jpegenc ! \
> 'image/jpeg,width=1920,height=1080,framerate=60/1' ! \
> v4l2video21videodec ! \
> 'video/x-raw,width=1920,height=1080,framerate=60/1,format=NV12' ! \
> testsink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element /GstPipeline:pipeline0/v4l2video21videodec:v4l2video21videodec0: Failed to allocate required memory.
Additional debug info:
gstv4l2videodec.c(711): gst_v4l2_video_dec_handle_frame (): /GstPipeline:pipeline0/v4l2video21videodec:v4l2video21videodec0:
Buffer pool activation failed
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

I think the error is incorrect because the same thing occurs if I lower everything to 640x480@10, which should reduce the memory usage quite a bit. I do get this heap from the kernel:

Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3631564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3631564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3631564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3631564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3631564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3136564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3136564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3136564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3136564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3136564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3432564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3432564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3432564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3432564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3432564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x59455247) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x59455247) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x59455247) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x59455247) invalid.
Sep 05 07:35:54 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x59455247) invalid.

... but to me, that looks like it's just gstreamer probing the supported output formats from the jpeg decoder.

by **wiley** » Sun Sep 10, 2017 4:03 pm

Has anyone managed to get the hardware jpeg decoder to work in gstreamer or ffmpeg?

by **trungnmh90** » Thu Jan 11, 2018 2:10 pm

You can use jpeg encoder/decoder using V4L2 call, my problem is just don't know how to set jpeg compression on encoder. Both `VIDIOC_G_JPEGCOMP` and `VIDIOC_S_JPEGCOMP` return "Inappropriate ioctl for device".

Code: Select all: sudo v4l2-ctl -d /dev/video30 -D Driver Info (not using libv4l2): Driver name : s5p-jpeg Card type : s5p-jpeg encoder Bus info : platform:11f50000.jpeg Driver version: 4.14.11 Capabilities : 0x84208000 Video Memory-to-Memory Streaming Extended Pix Format Device Capabilities Device Caps : 0x04208000 Video Memory-to-Memory Streaming Extended Pix Format

Code: Select all: sudo v4l2-ctl -d /dev/video30 --get-jpeg-comp VIDIOC_G_JPEGCOMP: failed: Inappropriate ioctl for device

Does anybody know how to set compression value for jpeg encoder? And I am sure it is there.

Code: Select all: sudo v4l2-ctl -d /dev/video30 --all ... JPEG Compression Controls chroma_subsampling (menu) : min=0 max=5 default=1 value=1 restart_interval (int) : min=0 max=3 step=65535 default=0 value=0 compression_quality (int) : min=0 max=3 step=1 default=3 value=3

by **mad_ady** » Thu Jan 11, 2018 3:24 pm

It looks like there is a compression_quality attribute bound to /dev/video30 that you should be able to set with a similar command (I'll look it up)

by **elector** » Fri Jan 19, 2018 4:23 am

Hi guys

For the last half year or so (on and mostly off...) I've been trying to get my XU4 to encode a video/audio stream coming from a GoPro->HDMI->AJA UTAP->USB3.0->XU4 but have so far not been able to do so. Every time I seem to run into some (to me) weird issues. But now I'm trying again, and hope that you guys have some good ideas on how to do this.

I have installed ffmpeg and gst following the advices posted earlier in this thread.

root@odroid:~# uname -a
Linux odroid 4.14.12-99 #1 SMP PREEMPT Mon Jan 8 03:14:09 UTC 2018 armv7l armv7l armv7l GNU/Linux

root@odroid:~# ffmpeg -version
ffmpeg version 3.3.3-5 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) 20160609
[rest of output omitted, let me know if it's needed]

root@odroid:~# gst-launch-1.0 --version
gst-launch-1.0 version 1.11.91
GStreamer 1.11.91
https://launchpad.net/distros/ubuntu/+s ... treamer1.0

So... when I try a relative simple capture with gst I see the following, how come?

root@odroid:~# gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw,width=1280,height=720,framerate=25/1,format=NV12' ! v4l2video11h264enc extra-controls="encode,video_bitrate=15728640" ! h264parse ! testsink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(2939): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.059362369
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
root@odroid:~#

[EDIT]
Just as a test that the camera is working I can stream while encoding in MPEG2, at a low framerate though, but it works.

root@odroid:~# ffmpeg -re -f v4l2 -framerate 25 -video_size 1280x720 -i /dev/video0 -b:v 5000k -c:a aac -b:a 128k -loglevel info -f mpegts udp://192.168.0.1:1234
[not so interesting output omitted]
Input #0, video4linux2,v4l2, from '/dev/video0':
Duration: N/A, start: 7033.542778, bitrate: 368640 kb/s
Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 1280x720, 368640 kb/s, 25 fps, 25 tbr, 1000k tbn, 1000k tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg2video (native))
Press [q] to stop, [?] for help
No pixel format specified, yuv422p for MPEG-2 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
Output #0, mpegts, to 'udp://192.168.0.1:1234':
Metadata:
encoder : Lavf57.71.100
Stream #0:0: Video: mpeg2video (4:2:2), yuv422p, 1280x720, q=2-31, 5000 kb/s, 25 fps, 90k tbn, 25 tbc
Metadata:
encoder : Lavc57.89.100 mpeg2video
Side data:
cpb: bitrate max/min/avg: 0/0/5000000 buffer size: 0 vbv_delay: -1
frame= 326 fps=6.8 q=2.0 Lsize= 11450kB time=00:00:43.04 bitrate=2179.3kbits/s speed=0.892x x
[I hit CTRL+C here]
video:10580kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.219366%
Exiting normally, received signal 2.

Any advice greatly appreciated.

by **memeka** » Fri Jan 19, 2018 9:03 am

1. Not sure NV12 is accepted input. So try going first through v4l2video20convert (check that is the actual gst node) before the encoder.
2. Also use fakesink in your tests instead of testsink
3. Try puting some ! queue ! between elements, eg. before and after the encoder

by **elector** » Sat Jan 20, 2018 4:25 am

Thanks for your input. Unfortunately it didn't change much.

While tailing /var/log/syslog I got the folowing

root@odroid:~# gst-launch-1.0 v4l2src device=/dev/video0 ! v4l2video20convert ! queue ! 'video/x-raw,width=1280,height=720,framerate=25/1' ! queue ! v4l2video11h264enc extra-controls="encode,video_bitrate=15728640" ! queue ! h264parse ! queue ! fakesink
Setting pipeline to PAUSED ...
Jan 19 19:23:11 odroid kernel: [ 4418.152760] Wrong buffer/video queue type (1)
[cut 13 lines of the same as these two above and below]
Jan 19 19:23:11 odroid kernel: [ 4418.208589] Wrong buffer/video queue type (1)
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Jan 19 19:23:12 odroid kernel: [ 4418.213081] Wrong buffer/video queue type (1)
Jan 19 19:23:12 odroid kernel: [ 4418.462747] xhci-hcd xhci-hcd.3.auto: ERROR unknown event type 37
Jan 19 19:23:12 odroid kernel: [ 4418.555047] xhci-hcd xhci-hcd.3.auto: ERROR unknown event type 37
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(2939): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.327502941
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

I have also tried a few more ! queue ! statements, and v4l2video21convert but no go.

root@odroid:~# gst-inspect-1.0 | grep video4
video4linux2: v4l2src: Video (video4linux2) Source
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2: v4l2video10videodec: V4L2 Video Decoder
video4linux2: v4l2video11h264enc: V4L2 H.264 Encoder
video4linux2: v4l2video31videodec: V4L2 Video Decoder
video4linux2: v4l2video33videodec: V4L2 Video Decoder
video4linux2: v4l2video20convert: V4L2 Video Converter
video4linux2: v4l2video21convert: V4L2 Video Converter

Any good ideas on how to proceed?

by **SamVDB²** » Tue Feb 13, 2018 6:04 pm

Here is the pipeline I use on a 4.14.13 kernel with a XU4: https://github.com/ardje/xu4-gst-quicks ... est.c#L518

It grabs video from /dev/video0 and audio from alsa card hw:1.

by **mad_ady** » Tue Feb 20, 2018 10:01 pm

I'm reopening this thread because I've switched to kernel 4.14 and am now testing with Hardkernel's ffmpeg build. I'm on kernel 4.14.16-104 currently.

There are a few problems left. I've tried these three encoding tests that I tried initially (in the first thread) (not ok means playing with black screen):

Code: Select all: ffmpeg -i big_buck_bunny_720p_h264.mov -acodec aac -vcodec h264 -b:v 2M -pix_fmt nv21 bbb.mp4 frame= 518 fps=2.3 q=-0.0 Lsize= 29551kB time=00:09:56.48 bitrate= 405.9kbits/s dup=0 drop=13792 speed=2.61x video:4572kB audio:24861kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.401162% //ok in mplayer, not ok in vlc, gstreamer. Also gets stuck after ~22s (518 frames). Required a reboot of the XU4 (HC2 actually) to unstuck. Even when it encoded without getting stuck, the video plays until this frame and afterwards is black. Weird. ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -r 24 -i - -an -vcodec h264 -b:v 2M -profile:v 10 -pix_fmt nv21 bbb.mp4 //ok in mplayer, not ok in vlc, gstreamer. High fps (85). ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -r 24 -i - -i big_buck_bunny_720p_h264.mov -map 0:v:0 -vcodec h264 -b:v 2M -profile:v 10 -pix_fmt nv21 -map 1:a:0 -c:a:1 aac bbb.mp4 //locks every time.

I think the black screen problem is related to what memeka said about V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. Any idea how I can check how the header is set or how to set it "the compliant way"?

by **memeka** » Tue Feb 20, 2018 11:47 pm

You don’t need annexb conversion with latest ffmpeg

by **mad_ady** » Wed Feb 21, 2018 2:35 am

That's good to know, though it has much higher encode fps.
I will also test low bitrate encoding, but I was hoping to get universally playable files with ffmpeg.
Maybe I will look into gstreamer after all. I take it that the stock gstreamer won't do, right?

by **mad_ady** » Wed Mar 28, 2018 4:16 pm

I've restarted playing with ffmpeg before I give up on it for good

If I produce mkv output instead of mp4, it's played better on other players:

Code: Select all: ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -y -r 24 -i - -an -vcodec h264 -b:v 500k -profile:v 10 -pix_fmt nv21 bbb.mkv

* this plays fine in vlc and gstreamer based players. On Android it plays with MxPlayer, but the video is not seekable for some reason!

This pipeline encodes the first 518 frames only, and then hangs, but without any dmesg output:

Code: Select all: ffmpeg -y -i big_buck_bunny_720p_h264.mov -acodec aac -vcodec h264 -b:v 2M -pix_fmt nv21 bbb.mkv

I am still not able to encode sound and video at the same time using the first pipeline:
* this doesn't work

Code: Select all: ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -y -r 24 -i - -i big_buck_bunny_720p_h264.mov -map 0:v:0 -vcodec h264 -b:v 2M -profile:v 10 -pix_fmt nv21 -map 1:2 -c:a aac bbb.mkv

* this should work (audio muxed in in the second step):

Code: Select all: ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -y -r 24 -i - -an -vcodec h264 -b:v 500k -profile:v 10 -pix_fmt nv21 bbb-video.mkv ffmpeg -i bbb-video.mkv -i big_buck_bunny_720p_h264.mov -map 0:v:0 -vcodec copy -map 1:a:0 -c:a aac bbb.mkv

... but doesn't. Produced a file with unplayable video, but audio works

. When encoding the second step I do get this error:

Code: Select all: [h264 @ 0x4c6a30] missing picture in access unit with size 23 [h264 @ 0x4c6a30] no frame!

Does anyone have other suggestions about what I'm doing wrong? Or should I throw in the towel and start reading on gst instead?
It does seem that ffmpeg produces non-standard output, that even ffmpeg can't read:

Code: Select all: root@odroid:/media/home/home/temp# ffmpeg -i bbb-video.mkv -vcodec copy bbb2.mkv ffmpeg version 3.3.3-5ubuntu1 Copyright (c) 2000-2017 the FFmpeg developers built with gcc 5.4.1 (Ubuntu/Linaro 5.4.1-2ubuntu1~16.04) 20160904 configuration: --prefix=/usr --extra-version=5ubuntu1 --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --disable-d3d11va --disable-dxva2 --disable-nvenc --disable-vaapi --disable-vda --disable-vdpau --disable-videotoolbox --enable-v4l2_m2m --enable-libdc1394 --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared libavutil 55. 58.100 / 55. 58.100 libavcodec 57. 89.100 / 57. 89.100 libavformat 57. 71.100 / 57. 71.100 libavdevice 57. 6.100 / 57. 6.100 libavfilter 6. 82.100 / 6. 82.100 libavresample 3. 5. 0 / 3. 5. 0 libswscale 4. 6.100 / 4. 6.100 libswresample 2. 7.100 / 2. 7.100 libpostproc 54. 5.100 / 54. 5.100 [h264 @ 0x4c79c0] missing picture in access unit with size 23 [h264 @ 0x4c79c0] no frame! Input #0, matroska,webm, from 'bbb-video.mkv': Metadata: ENCODER : Lavf57.71.100 Duration: 00:09:56.25, start: 0.000000, bitrate: 477 kb/s Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 24 fps, 24 tbr, 1k tbn, 2k tbc (default) Metadata: ENCODER : Lavc57.89.100 h264_v4l2m2m DURATION : 00:09:56.250000000 Output #0, matroska, to 'bbb2.mkv': Metadata: encoder : Lavf57.71.100 Stream #0:0: Video: h264 (High) (H264 / 0x34363248), yuv420p(progressive), 1280x720, q=2-31, 24 fps, 24 tbr, 1k tbn, 1k tbc (default) Metadata: ENCODER : Lavc57.89.100 h264_v4l2m2m DURATION : 00:09:56.250000000 Stream mapping: Stream #0:0 -> #0:0 (copy) Press [q] to stop, [?] for help frame=14310 fps=0.0 q=-1.0 Lsize= 34733kB time=00:09:56.20 bitrate= 477.2kbits/s speed= 644x video:34585kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.428139%

by **meveric** » Wed Mar 28, 2018 5:19 pm

I just tried on my image as well:
first I tried using hardware decoder and hardware encoder:

Code: Select all: ffmpeg -vcodec h264_v4l2m2m -i big_buck_bunny_720p_h264.mov -vcodec h264_v4l2m2m -b:v 2M -profile:v 10 -pix_fmt nv21 bbb-video.mkv [h264_v4l2m2m @ 0x4c9000] driver 'exynos-gsc' on card 'exynos-gsc gscaler' Last message repeated 1 times [h264_v4l2m2m @ 0x4c9000] driver 's5p-mfc' on card 's5p-mfc-enc' [h264_v4l2m2m @ 0x4c9000] driver 's5p-mfc' on card 's5p-mfc-dec' [h264_v4l2m2m @ 0x4c9000] Using device /dev/video10 [h264_v4l2m2m @ 0x4c9000] driver 's5p-mfc' on card 's5p-mfc-dec' Stream mapping: Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> h264 (h264_v4l2m2m)) Stream #0:2 -> #0:1 (aac (native) -> vorbis (libvorbis)) Press [q] to stop, [?] for help [h264_v4l2m2m @ 0x4c9000] output POLLERR [h264_v4l2m2m @ 0x4bcb90] driver 'exynos-gsc' on card 'exynos-gsc gscaler' Last message repeated 1 times [h264_v4l2m2m @ 0x4bcb90] driver 's5p-mfc' on card 's5p-mfc-enc' [h264_v4l2m2m @ 0x4bcb90] Using device /dev/video11 [h264_v4l2m2m @ 0x4bcb90] driver 's5p-mfc' on card 's5p-mfc-enc' [h264_v4l2m2m @ 0x4bcb90] h264_v4l2m2m encoder: enabling bit rate control: 500000 [h264_v4l2m2m @ 0x4bcb90] h264 profile not found [h264_v4l2m2m @ 0x4bcb90] Encoder adjusted: qmin (0), qmax (51)

(as you can see it uses both /dev/video10 for decoding and /dev/video11 for encoding)
The result was not good, after about 25~30 seconds, the decoder produced an error and the encoder started dropping all frames. Also the speed was about 30 FPS for encoding.
Then I switched to software decoder:

Code: Select all: ffmpeg -i big_buck_bunny_720p_h264.mov -vcodec h264_v4l2m2m -b:v 2M -profile:v 10 -pix_fmt nv21 bbb-video.mkv [h264_v4l2m2m @ 0x49cfb0] driver 'exynos-gsc' on card 'exynos-gsc gscaler' Last message repeated 1 times [h264_v4l2m2m @ 0x49cfb0] driver 's5p-mfc' on card 's5p-mfc-enc' [h264_v4l2m2m @ 0x49cfb0] Using device /dev/video11 [h264_v4l2m2m @ 0x49cfb0] driver 's5p-mfc' on card 's5p-mfc-enc' [h264_v4l2m2m @ 0x49cfb0] h264_v4l2m2m encoder: enabling bit rate control: 500000 [h264_v4l2m2m @ 0x49cfb0] h264 profile not found [h264_v4l2m2m @ 0x49cfb0] Encoder adjusted: qmin (0), qmax (51)

The result was very good, in fact it was way better then using hardware decoder. First, the video was completely converted without dropping of frames. Second, it was encoded in 60 FPS, so way faster than using hardware decoder.
The result seemed to be fine on mpv (software decoding) ffplay (software decoding) ffplay (hardware decoding) and even on vlc and cvlc.

I only tested via VNC so although it looked fine to me, it could be that the colors are off, as I use remote control to look at it (but I think it should be fine).
Edit:
yeah colors seem fine, just tried x11grab again and it's still horrible and colors are broken, so the above should work.

by **mad_ady** » Wed Mar 28, 2018 5:39 pm

Thanks for that input. So, the difference in order to do hardware decoding is that you specify '-vcodec h264_v4l2m2m' before '-i'. The thing is - for me with the "stock" ffmpeg from HK I get the same behavior (hardware decoding) even when I don't specify '-vcodec h264_v4l2m2m' before '-i'.
But at least we know the lockup is an issue in the decoder, not encoder.

My ffmpeg version is '3.3.3-5ubuntu1', and lists these h264 decoders:

Code: Select all: V..... h264_v4l2m2m V4L2 mem2mem H.264 decoder wrapper (codec h264) VFS..D h264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10

I played with it and if I can force it to use the software decoder by using this instead:

Code: Select all: ffmpeg -vcodec h264 -i big_buck_bunny_720p_h264.mov -vcodec h264_v4l2m2m -b:v 2M -profile:v 10 -pix_fmt nv21 bbb-video.mkv

Thanks for the input again - I shall return to playing now.

by **mad_ady** » Wed Mar 28, 2018 6:28 pm

I've played with encoding a bit more and I'm happier with the results (not happiest though).

So - doing software decoding to avoid lockups (thanks @meveric). Also, encoding sound together with video works this time.
* Encoding to mp4 results in a file which is not playable on Android and in gstreamer based players. Booo

Code: Select all: fmpeg -vcodec h264 -i big_buck_bunny_720p_h264.mov -vcodec h264_v4l2m2m -b:v 500k -profile:v 10 -pix_fmt nv21 -acodec aac bbb.mp4

* Encoding to mkv + sound results in a file which is playable everywhere. Yay!

Code: Select all: ffmpeg -vcodec h264 -i big_buck_bunny_720p_h264.mov -vcodec h264_v4l2m2m -b:v 500k -profile:v 10 -pix_fmt nv21 -acodec aac bbb.mkv

I will extend my tests to other input formats and add truncating the input if necessary to respect the 64bit boundary.

by **elatllat** » Wed Mar 28, 2018 7:06 pm

Try "-movflags +faststart" when making the mp4.
also ffprobe a working and non-working mp4 to see what the diff.

by **mad_ady** » Wed Mar 28, 2018 8:28 pm

Thanks for the idea. I tried it but ffprobe fails to load both files (with and without those options):

Code: Select all: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x4ff010] st: 0 edit list: 1 Missing key frame while searching for timestamp: 0 [h264 @ 0x5002a0] no frame!

It's as if the stream starts without a keyframe? I'll need to see if I can force a keyframe. Strange that mkv does not complain, and that's just a different container - the video stream should be the same, right?

by **meveric** » Wed Mar 28, 2018 8:56 pm

have you tried using mpeg4_v4l2m2m instead when making mp4 file?
When I try with h264_v4l2m2m I can't open the .mp4 in vlc but if I use mpeg4_v4l2m2m instead it seems to be working.

Code: Select all: Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> mpeg4 (mpeg4_v4l2m2m)) Stream #0:2 -> #0:1 (aac (native) -> aac (native)) Press [q] to stop, [?] for help [mpeg4_v4l2m2m @ 0x4ddae0] driver 'exynos-gsc' on card 'exynos-gsc gscaler' Last message repeated 1 times [mpeg4_v4l2m2m @ 0x4ddae0] driver 's5p-mfc' on card 's5p-mfc-enc' [mpeg4_v4l2m2m @ 0x4ddae0] Using device /dev/video11 [mpeg4_v4l2m2m @ 0x4ddae0] driver 's5p-mfc' on card 's5p-mfc-enc' [mpeg4_v4l2m2m @ 0x4ddae0] h264_v4l2m2m encoder: enabling bit rate control: 2000000 [mpeg4_v4l2m2m @ 0x4ddae0] mpeg4 profile not found [mpeg4_v4l2m2m @ 0x4ddae0] Encoder adjusted: qmin (1), qmax (31)

by **mad_ady** » Wed Mar 28, 2018 10:15 pm

I had no idea that was a thing...
I thought it referred to the encoding codec (h264), not container(mpeg4).
I will try it, thanks!

by **elatllat** » Wed Mar 28, 2018 10:24 pm

maybe a 'muxing' difference vs 'encoding' or 'container' ~

by **mad_ady** » Thu Mar 29, 2018 10:16 pm

So, I have tried

Code: Select all: ffmpeg -vcodec h264 -i big_buck_bunny_720p_h264.mov -vcodec mpeg4_v4l2m2m -b:v 500k -profile:v 10 -movflags +faststart -pix_fmt nv21 -acodec aac bbb_mpeg4.mp4

But the output doesn't play in mplayer nor in Android players. VLC plays it though.

by **dumbledore** » Fri Apr 13, 2018 1:12 am

I'm having issues with the hardware encoder crashing as well. (see here: https://forum.odroid.com/viewtopic.php?t=30548). I'm able to encode using some of the pipelines recommended here, but the encoder seems to have many more issues than the software encoder (which works with seemingly anything you throw at it).

by **_om_** » Thu May 03, 2018 10:30 pm

Hi guys,
I've faced hw encoder locks as well with last 4.14, while decoder is mostly works ok.
So, what is the recommended Ubuntu image so far to drive both hw decoder and encoder (h264 is enough) with gstreamer at the same time?
Or if there is no stable one, may be a linux-kernel deb package exists with proper config.
Thank you.

by **memeka** » Sat Jun 30, 2018 11:36 am

I have recompiled ffmpeg 3.4.2 with some extra patches, and encoder seems to be more stable.
I get no issues using hw transcoding now fully.

test for bbb 1080 trailer:
sw dec + hw enc: speedup = 1.6x, 2 big cores used @ ~55% each
hw dec + hw enc: speedup = 1.2x, 1 big core used @ ~40%

it's a bit slower, but using less than half the CPU of sw dec.

by **odroid** » Sat Jun 30, 2018 1:19 pm

@memeka,
Great! Can you please share the patches?
We want to update the ffmpeg package in our PPA server.

by **memeka** » Sat Jun 30, 2018 1:47 pm

odroid wrote:@memeka,
Great! Can you please share the patches?
We want to update the ffmpeg package in our PPA server.

https://github.com/mihailescu2m/FFmpeg branch v4l2m2m-3.4.2
just clone and dpkg-buildpackage