[Howto] Odroid XU3/4 using the hardware encoder

[jwrona]

how about when using clutterautovideosink ?

EDIT: you don't need v4l2video5convert

Thanks, clutterautovideosink works, but its laggy and probably uses Open GL:

Code: Select all

libGL error: unable to load driver: exynos_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: exynos

[allanm84]

Has anyone been able to stream JPEG from the oCam and convert it? I'm attempting to take in 1080p mjpeg from the ocam, run it through v4l2video3videodec, then encode it h.264 with v4l2video11h264enc. Unfortunately the videodec fails with a memory error, but jpegdec works.

The following pipeline works just fine:

Code: Select all

gst-launch-1.0 v4l2src device=/dev/video6 ! \
 image/jpeg,width=1920 ,height=1080 ,framerate=30/1 ! \
 jpegdec ! videoconvert ! \
 video/x-raw,format=NV12 ! \
 v4l2video11h264enc extra-controls="encode,h264_level=10,h264_profile=1,frame_level_rate_control_enable=1,video_bitrate=$bitrate" ! \
 rtph264pay ! \
 udpsink port=$port host=$host_ip sync=false

But this fails:

Code: Select all

gst-launch-1.0 v4l2src device=/dev/video6 ! \
 image/jpeg,width=1920 ,height=1080 ,framerate=30/1 ! \
 v4l2video3videodec ! \
 video/x-raw,format=NV12 ! \
 v4l2video11h264enc extra-controls="encode,h264_level=10,h264_profile=1,frame_level_rate_control_enable=1,video_bitrate=$bitrate" ! \
 rtph264pay ! \
 udpsink port=$port host=$host_ip sync=false

With the following error:

Code: Select all

ERROR: from element /GstPipeline:pipeline0/v4l2video3videodec:v4l2video3videodec0: Failed to allocate required memory.
Additional debug info:
gstv4l2videodec.c(711): gst_v4l2_video_dec_handle_frame (): /GstPipeline:pipeline0/v4l2video3videodec:v4l2video3videodec0:
Buffer pool activation failed
Execution ended after 0:00:00.205629164

I played around with the ouput-io-mode and capture-io-mode for the decoder and encoder, but that didn't seem to do anything. I also threw some queues in there to no avail. Any help?

[mad_ady]

Ok, I'm doing something wrong, and I can't figure out what.
I'm using the hardware encoder with the following command to produce an mp4 file:

Code: Select all

ffmpeg -y -i input.ts -map 0:0 -codec:v h264 -b:v 1500k -pix_fmt nv21 -vf 'yadif, scale=trunc(iw/64)*64:trunc(ih/64)*64' -acodec copy A.mp4

The resulting file plays with smplayer and vlc under linux, but can't be opened with gstreamer-based players. Once playback has started I can jump around in the video without issues. One strange thing is that when I open the file again with SMplayer or VLC and they try to resume from where I left off, I get a black image and sound plays fine. The image never recovers (like when receiving an I frame). If I try to play this file on Android with MXPlayer - I can play the file with software decoding only. Other players display a black image with audio only.

I also tried to cut a part of the video (strip out commercials) with:

Code: Select all

ffmpeg -i A.mp4 -vcodec copy -acodec copy -ss 4:20 -t 20:00 B.mp4

The resulting file does not render video in any player - just a black screen with sound.

Here are some mediainfo/ffmpeg outputs of A.mp4:

Code: Select all

mediainfo /DataVolume/TVRecordings/prelucrate/20170520072503_.mp4 
General
Complete name                            : /DataVolume/TVRecordings/prelucrate/20170520072503_.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 404 MiB
Duration                                 : 35mn 1s
Overall bit rate                         : 1 611 Kbps
Encoded date                             : UTC 1904-01-01 00:00:00
Tagged date                              : UTC 1904-01-01 00:00:00
Writing application                      : Lavf57.25.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.2
Format settings, CABAC                   : No
Format settings, ReFrames                : 1 frame
Format settings, GOP                     : M=1, N=12
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 35mn 1s
Bit rate                                 : 1 477 Kbps
Width                                    : 704 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 16:9
Original display aspect ratio            : 1.222
Frame rate mode                          : Constant
Frame rate                               : 25.000 fps
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.146
Stream size                              : 370 MiB (92%)
Encoded date                             : UTC 1904-01-01 00:00:00
Tagged date                              : UTC 1904-01-01 00:00:00

Audio
ID                                       : 2
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 2
Codec ID                                 : 6B
Duration                                 : 35mn 1s
Bit rate mode                            : Constant
Bit rate                                 : 128 Kbps
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 KHz
Compression mode                         : Lossy
Stream size                              : 32.1 MiB (8%)
Language                                 : Romanian
Default                                  : Yes
Alternate group                          : 1
Encoded date                             : UTC 1904-01-01 00:00:00
Tagged date                              : UTC 1904-01-01 00:00:00

ffmpeg -i /DataVolume/TVRecordings/prelucrate/20170520072503_.mp4
ffmpeg version 3.0.2-1ubuntu4 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) 20160609
  configuration: --prefix=/usr --extra-version=1ubuntu4 --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --cc=cc --cxx=g++ --enable-gpl --enable-shared --d
isable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --en
able-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --
enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --
enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc139
4 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-chromaprint --enable-libx264
  WARNING: library configuration mismatch
  avcodec     configuration: --prefix=/usr --extra-version=1ubuntu4 --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --cc=cc --cxx=g++ --enable-gpl --enabl
e-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-
libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-li
bopenjpeg --enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-
libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --ena
ble-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-chromaprint --enable-libx264 --enable-version3 --disable-doc --disable-programs --disable-avdevice --disable-avfilter --disable-avf
ormat --disable-avresample --disable-postproc --disable-swscale --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libvo_amrwbenc
  libavutil      55. 17.103 / 55. 17.103
  libavcodec     57. 24.102 / 57. 24.102
  libavformat    57. 25.100 / 57. 25.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 31.100 /  6. 31.100
  libavresample   3.  0.  0 /  3.  0.  0
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/DataVolume/TVRecordings/prelucrate/20170520072503_.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.25.100
  Duration: 00:35:01.56, start: 0.000000, bitrate: 1610 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 704x576, 1477 kb/s, SAR 16:11 DAR 16:9, 25 fps, 25 tbr, 12800 tbn, 25600 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(rum): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, s16p, 127 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
At least one output file must be specified

Any ideas what I'm doing wrong?

[memeka]

@mad_ady

I noticed with ffmpeg, if you comment out

Code: Select all

SET_V4L_EXT_CTRL(value, V4L2_CID_MPEG_VIDEO_HEADER_MODE, V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME 

or if you set it to the default V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE, the resulting video is not correct (players don't recognize the codec)...

i have no idea why tho` gstreamer encoder by default uses V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE and all is good...

so maybe your issue has something to do with this setting?

EDIT: with ffmpeg encoder transcoding in emby (with V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME), I can resume video in emby player (ffmpeg to rstp streaming) without issues.

[mad_ady]

I'll take a look, thanks!

Edit: I've checked the code and it seems to be there:

Code: Select all

SET_V4L_EXT_CTRL(value, V4L2_CID_MPEG_VIDEO_HEADER_MODE, V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME, V4L2_CTRL_CLASS_MPEG, "header mode");

I'll recompile ffmpeg from your Git just in case I'm running something older...

[memeka]

don't think it will work.
i think that since header mode is "joined with 1st frame", cutting the beginning will lose the header...
so you need to recompile ffmpeg with header mode "separate" ... the issue is that for some reason, in this case, ffmpeg will not create the header ... no idea why...
i would recommend to "play" and "re-encode" the video w/o the commercials. such that the 1st frame will be remade with header joined...

[mad_ady]

I understand - but the thing is - even if the uncut clip which has the first frame intact has problems playing back. I'll need to scratch my head further...

I could trim the file before encoding it. I'll give that a try as well.

[memeka]

you can always use gstreamer encoder

[mad_ady]

I'm not at my wit's end yet

[memeka]

emby released with support for exynos mfc encoding: viewtopic.php?f=98&t=26390&p=191996#p191996

[tweekzilla]

I'm trying to motion to work with hardware encoding using ffmpeg. How do you enable the v4l2_m2m in the master head of the ffmpeg git repository? I can't seem to find the configuration option to do that

[mad_ady]

It's not on the master branch, but on a subbranch.

[ArraY80]

I have Ubuntu mate 16.04.02 LTS
With this kernel:



Last login: Sun Jul 23 15:39:25 2017 from 84.78.24.207
root@odroid:~# uname -a
Linux odroid 4.9.35-48 #1 SMP PREEMPT Thu Jul 6 14:35:57 UTC 2017 armv7l armv7l armv7l GNU/Linux
root@odroid:~# apt-get upgrade
Leyendo lista de paquetes... Hecho
Creando árbol de dependencias
Leyendo la información de estado... Hecho
Calculando la actualización... Hecho
Los siguientes paquetes se han retenido:
linux-image-xu3
0 actualizados, 0 nuevos se instalarán, 0 para eliminar y 1 no actualizados.

1) Do I have to compile the kernel again or with that version I do not have to do anything and just install ffmpeg or gstreamer?

2) Is the installation of ffmpeg correct with apt-get install ffmpeg or do I have to manually compile and install ffmpeg?

3)And for gstreamer os equal instalación?


4) Do I have to install the retained package linux-image-xu3 before?

Regular users on the forum might already know that memeka has managed to use the hardware video encoder on the XU4 by using a mainline kernel (4.8.x) . Here are the steps on how you can achieve the same thing:

Before you begin - this is a tutorial designed for medium/advanced users and if things go wrong you might have an unbootable system. It might be wise to make a backup before beginning: http://forum.odroid.com/viewtopic.php?t=22930 (shameless plug)

1. Install a mainline kernel that supports the MFC encoder. The instructions here use @elatllat's branch (https://github.com/Dmole/linux.git), but also mdrjr is working on a branch of his own (https://github.com/mdrjr/linux) and so is memeka (https://github.com/mihailescu2m/linux/c ... dxu3-4.9.y). You can use my kernel config (which tries to add all of HardKernel's modules), but you can modify it to fit your needs http://pastebin.com/7YnakKmP, or you can use whatever defconfig comes with the kernel you chose.
*** NOTE: the kernel does not support HMP (big.LITTLE extensions) and treats all cores as equal. This is because HMP patches are unstable and can lock up the system. Consequently, overall system performance will be lower than when using Kernel 3.10 ***
*** NOTE2: Also, for XU4 sound over HDMI will no longer work with this kernel! ***

I have added the kernel compilation procedure here for convenience, but you should read/discuss the official kernel compilation thread here: http://forum.odroid.com/viewtopic.php?f=95&t=23163 or this guide: https://github.com/umiddelb/armhf/wiki/ ... ARM-device

a) Make a copy of your existing kernel, initrd, dtb and boot.ini:

Code: Select all

$ cd /media/boot
$ sudo -i
# cp zImage zImage-3
# cp uInitrd uInitrd-3
# cp exynos5422-odroidxu3.dtb exynos5422-odroidxu3-3.dtb
# cp boot.ini boot3.ini

Edit boot3.ini to point to the newly copied files by appending -3 to the name of the zImage, uInitrd and dtb. By replacing the stock boot.ini with this modified boot.ini you will be able to boot your old kernel in case of problems.

b) Download the new kernel

Code: Select all

$ sudo apt-get -y install bc curl gcc git libncurses5-dev lzop make u-boot-tools dos2unix
$ git clone --depth 1 -b odroidxu4-mihailescu2m-4.9 https://github.com/Dmole/linux.git linux
$ cd linux/
$ make odroidxu4_defconfig

Optionally get my config which supports most of the USB peripherals (TV tuners, sound cards, wifi cards) and a lot of networking modules (LXC support, VLANs, iptables).

Code: Select all

$ wget http://pastebin.com/raw/7YnakKmP -O .config
$ dos2unix .config

c) Next compile the kernel

Code: Select all

$ make menuconfig
$ make -j 8 zImage dtbs modules
$ kver=`make kernelrelease`
$ sudo cp arch/arm/boot/zImage arch/arm/boot/dts/exynos5422-odroidxu[34].dtb /media/boot
$ sudo cp .config /media/boot/config
$ sudo make INSTALL_MOD_STRIP=1 modules_install
$ sudo make firmware_install
$ sudo cp .config /boot/config-${kver}
$ cd /boot
$ sudo update-initramfs -c -k ${kver}
$ sudo mkimage -A arm -O linux -T ramdisk -a 0x0 -e 0x0 -n initrd.img-${kver} -d initrd.img-${kver} uInitrd-${kver}
$ sudo cp uInitrd-${kver} /media/boot/uInitrd

Next modify /media/boot/boot.ini and load exynos5422-odroidxu4.dtb instead of exynos5422-odroidxu3.dtb and save boot.ini.

d) Shut down and unplug HDMI and power. Without this step when you boot with the new kernel you will not have USB3 bus and onboard networking! This is needed only when switching between 3.x and 4.x kernels as far as I've seen.

Optionally - if you intend to run with a custom kernel from now on it's best to mark linux-image-xu3, linux-headers-xu3 and bootini as manual so that they won't be overwritten on subsequent package updates:

Code: Select all

sudo apt-mark hold linux-image-xu3 linux-headers-xu3 bootini

Using FFMPEG
2. Now that the kernel is ready, compile/install a custom ffmpeg which supports the hardware encoder
a) Compile your own ffmpeg

Code: Select all

#debian build tools
$ sudo apt-get install build-essential fakeroot devscripts libchromaprint-dev librubberband-dev libjs-bootstrap
#get the patched ffmpeg version
$ git clone -b v4l2_m2m-3.0.2 --depth=1 https://github.com/mihailescu2m/FFmpeg.git
# install ffmpeg's build dependencies (~190 packages)
$ sudo apt-get build-dep ffmpeg
#build ffmpeg as deb packages with no checks (some checks fail)
$ cd FFmpeg/
$ DEB_BUILD_OPTIONS="nocheck" debuild -b -uc -us

When building is done you should have 23 deb packages one level up (in the same directory as FFmpeg). You can download these packages pre-compiled from here: http://www.mediafire.com/?l39zeyy4r172m

b) Install the packages:

Code: Select all

$ cd ../
$ sudo dpkg -i *.deb

This should replace any system-installed ffmpeg build with this ffmpeg which supports the hardware encoder.

If you plan on keeping using these packages you should mark them as manual so that they are not overwritten by future system updates:

Code: Select all

sudo apt-mark hold ffmpeg-doc ffmpeg libav-tools libavcodec-dev libavcodec-extra57 libavcodec-extra libavcodec57 libavdevice-dev libavdevice57 libavfilter-dev libavfilter6 libavformat-dev libavformat57 libavresample-dev libavresample3 libavutil-dev libavutil55 libpostproc-dev libpostproc54 libswresample-dev libswresample2 libswscale-dev libswscale4

3. Use the hardware encoder when transcoding (examples):
Note, depending on what you want to transform with ffmpeg you might get better or worse framerate. For example, changing the output fps of a video cuts off about 20-30fps from encoding time. The examples below do not change the fps.

Code: Select all

# encode video only, about 50fps (max)
$ ffmpeg -i big_buck_bunny_720p_h264.mov -acodec aac -vcodec h264 -b:v 2M -pix_fmt nv21 bbb.mp4
# encode video only, about 110fps (max)
$ ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -r 24 -i - -an  -vcodec h264 -b:v 2M -profile:v 10 -pix_fmt nv21 bbb.mp4
#encode video and audio, about 75fps (max)
$ ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -r 24 -i - -i big_buck_bunny_720p_h264.mov -map 0:v:0 -vcodec h264 -b:v 2M -profile:v 10 -pix_fmt nv21  -map 1:a:0 -c:a:1 aac bbb.mp4

The encoder automatically selects h264_v4l2m2m which does the hardware encoding:

Code: Select all

  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_v4l2m2m))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))

If you get the error:

Code: Select all

[h264_v4l2m2m @ 0xf3fa0] H264 codec detected, init annexb converter
[h264_v4l2m2m @ 0xf3fa0] Device path not set, probing /dev/video*
[h264_v4l2m2m @ 0xf3fa0] exynos-gsc.1:m2m is not the one we want
[h264_v4l2m2m @ 0xf3fa0] exynos-gsc.0:m2m is not the one we want
[h264_v4l2m2m @ 0xf3fa0] s5p-mfc-dec is not the one we want
[h264_v4l2m2m @ 0xf3fa0] Could not find a valid device

Make sure the user you are running ffmpeg with is part of the video group:

Code: Select all

odroid@odroid:~$ id
uid=1000(odroid) gid=1000(odroid) groups=1000(odroid),4(adm),20(dialout),24(cdrom),27(sudo),30(dip),44(video),46(plugdev),115(lpadmin),116(lightdm)

Sample output:

Code: Select all

adrianp@odroid:~> ffmpeg -i Sintel.2010.720p.mkv -acodec aac -vcodec h264 -b:v 2M  -pix_fmt nv21 sintel-encoded.mp4                                                                                        
ffmpeg version 3.0.2-1ubuntu4 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.2) 20160609
  configuration: --prefix=/usr --extra-version=1ubuntu4 --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --cc=cc --cxx=g++ --enable-gpl --enable-shared --d
isable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --
enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --
enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc139
4 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-chromaprint --enable-libx264
  WARNING: library configuration mismatch
  avcodec     configuration: --prefix=/usr --extra-version=1ubuntu4 --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --cc=cc --cxx=g++ --enable-gpl --enabl
e-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-
libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-li
bopenjpeg --enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-
libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --ena
ble-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-chromaprint --enable-libx264 --enable-version3 --disable-doc --disable-programs --disable-avdevice --disable-avfilter --disable-avf
ormat --disable-avresample --disable-postproc --disable-swscale --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libvo_amrwbenc
  libavutil      55. 17.103 / 55. 17.103
  libavcodec     57. 24.102 / 57. 24.102
  libavformat    57. 25.100 / 57. 25.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 31.100 /  6. 31.100
  libavresample   3.  0.  0 /  3.  0.  0
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Input #0, matroska,webm, from 'Sintel.2010.720p.mkv':
  Metadata:
    encoder         : libebml v1.0.0 + libmatroska v1.0.0
    creation_time   : 2011-04-24 17:20:33
  Duration: 00:14:48.03, start: 0.000000, bitrate: 6071 kb/s
    Chapter #0:0: start 0.000000, end 103.125000
    Metadata:
      title           : Chapter 01
    Chapter #0:1: start 103.125000, end 148.667000
    Metadata:
      title           : Chapter 02
    Chapter #0:2: start 148.667000, end 349.792000
    Metadata:
      title           : Chapter 03
    Chapter #0:3: start 349.792000, end 437.208000
    Metadata:
      title           : Chapter 04
    Chapter #0:4: start 437.208000, end 472.075000
    Metadata:
      title           : Chapter 05
    Chapter #0:5: start 472.075000, end 678.833000
    Metadata:
      title           : Chapter 06
    Chapter #0:6: start 678.833000, end 744.083000
    Metadata:
      title           : Chapter 07
    Chapter #0:7: start 744.083000, end 888.032000
    Metadata:
      title           : Chapter 08
    Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709/unknown/unknown), 1280x544, SAR 1:1 DAR 40:17, 24 fps, 24 tbr, 1k tbn, 48 tbc
    Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s
    Metadata:
      title           : AC3 5.1 @ 640 Kbps
    Stream #0:2(ger): Subtitle: subrip
    Stream #0:3(eng): Subtitle: subrip
    Stream #0:4(spa): Subtitle: subrip
    Stream #0:5(fre): Subtitle: subrip
    Stream #0:6(ita): Subtitle: subrip
    Stream #0:7(dut): Subtitle: subrip
    Stream #0:8(pol): Subtitle: subrip
    Stream #0:9(por): Subtitle: subrip
    Stream #0:10(rus): Subtitle: subrip
    Stream #0:11(vie): Subtitle: subrip
Codec AVOption preset (Configuration preset) specified for output file #0 (sintel-encoded.mp4) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video
 streams) or that it is a private option of some encoder which was not actually used for any stream.
File 'sintel-encoded.mp4' already exists. Overwrite ? [y/N] y
[h264_v4l2m2m @ 0xf3fe0] H264 codec detected, init annexb converter
[h264_v4l2m2m @ 0xf3fe0] Device path not set, probing /dev/video*
[h264_v4l2m2m @ 0xf3fe0] exynos-gsc.1:m2m is not the one we want
[h264_v4l2m2m @ 0xf3fe0] exynos-gsc.0:m2m is not the one we want
[h264_v4l2m2m @ 0xf3fe0] Using device /dev/video1
Output #0, mp4, to 'sintel-encoded.mp4':
  Metadata:
    encoder         : Lavf57.25.100
    Chapter #0:0: start 0.000000, end 103.125000
    Metadata:
      title           : Chapter 01
    Chapter #0:1: start 103.125000, end 148.667000
    Metadata:
      title           : Chapter 02
    Chapter #0:2: start 148.667000, end 349.792000
    Metadata:
      title           : Chapter 03
    Chapter #0:3: start 349.792000, end 437.208000
    Metadata:
      title           : Chapter 04
    Chapter #0:4: start 437.208000, end 472.075000
    Metadata:
      title           : Chapter 05
    Chapter #0:5: start 472.075000, end 678.833000
    Metadata:
      title           : Chapter 06
    Chapter #0:6: start 678.833000, end 744.083000
    Metadata:
      title           : Chapter 07
    Chapter #0:7: start 744.083000, end 888.032000
    Metadata:
      title           : Chapter 08
    Stream #0:0(eng): Video: h264 (h264_v4l2m2m) ([33][0][0][0] / 0x0021), yuv420p, 1280x544 [SAR 1:1 DAR 40:17], q=2-31, 2000 kb/s, 24 fps, 12288 tbn, 24 tbc
    Metadata:
      encoder         : Lavc57.24.102 h264_v4l2m2m
    Stream #0:1(eng): Audio: aac (LC) ([64][0][0][0] / 0x0040), 48000 Hz, 5.1(side), fltp, 341 kb/s
    Metadata:
      title           : AC3 5.1 @ 640 Kbps
      encoder         : Lavc57.24.102 aac
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_v4l2m2m))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0xf3fe0] Performing useless memcpy() on output pool because buffers do not match 
[h264_v4l2m2m @ 0xf3fe0] This could be avoided by using av_v4l_buffer_pool_get_buffer*() or av_v4l_buffer_pool_make_pipe()
[mp4 @ 0xb9c70] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 0xb9c70] Encoder did not produce proper pts, making some up.
[h264_v4l2m2m @ 0xf3fe0] No event occurred while waiting.01 bitrate=36881.3kbits/s speed=1.84x    
frame= 2027 fps= 44 q=-0.0 Lsize=  383559kB time=00:01:25.20 bitrate=36877.0kbits/s speed=1.84x    
video:379964kB audio:3551kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.011593%

Known issues:

* All known issues have been solved. Yay!




Using gstreamer
2. Compile gstreamer from memeka's branch

Code: Select all

$ apt-get install gstreamer1.0-plugins-bad
$ git clone https://github.com/mihailescu2m/gst-plugins-good
$ cd gst-plugins-good/
$ sudo apt-get install libgstreamer1.0-dev libgudev-1.0-dev libgstreamer-plugins-base1.0-dev dh-autoreconf automake autoconf libtool autopoint cdbs gtk-doc-tools libshout3-dev libaa1-dev libflac-dev libdv4-dev libdv-dev libgtk-3-dev libtag1-dev  libsoup2.4-dev gstreamer1.0-doc gstreamer1.0-plugins-base-doc
$ dpkg-buildpackage -us -uc -b -j4
$ cd ../
$ sudo dpkg -i gstreamer*.deb

Alternatively you can get the pre-built deb packages from here: https://www.mediafire.com/folder/5lz2po ... -odroidxu4
If you intend to keep these packages installed, mark them as manual so that they are not updated by apt:

Code: Select all

sudo apt-mark hold gstreamer1.0-plugins-good-dbg gstreamer1.0-plugins-good-doc gstreamer1.0-plugins-good gstreamer1.0-pulseaudio libgstreamer-plugins-good1.0-0 libgstreamer-plugins-good1.0-dev

To transcode something here's what you can try:
Identify which is the encoding and the decoding interfaces (they change after boot)

Code: Select all

$ decoder=`gst-inspect-1.0 | grep v4l2 | grep videodec | cut -d ":" -f 2`
$ encoder=`gst-inspect-1.0 | grep v4l2 | grep h264enc | cut -d ":" -f 2`
$ gst-launch-1.0 filesrc location=big_buck_bunny_720p_h264.mov ! qtdemux ! h264parse ! $decoder !  $encoder extra-controls="encode,h264_level=10,h264_profile=4,frame_level_rate_control_enable=1,video_bitrate=2097152" ! h264parse ! matroskamux ! filesink location=bbb.mkv

[mad_ady]

The kernel should have everything you need for mfc encoding. No need for a custom kernel anymore. But you still need a custom gstreamer or ffmpeg. If you don't want to compile it try installing the packages I link to in the first post!

[tweekzilla]

It's not on the master branch, but on a subbranch.

I guess what I was asking was is the code now folded into the ffmpeg main repository - i.e. I can compile the latest version of ffmpeg with some configure flags or is it only that specific branch in the forked repository

[mad_ady]

As far as I know it's still a specific branch

[jongerenchaos]

Is hardware encoding with VP8/9 also possible with these packages?

[odroid]

There is no VP8/9 hardware accelerator in Exynos-5422.

[jongerenchaos]

There is no VP8/9 hardware accelerator in Exynos-5422.

On the website http://wiki.webmproject.org/zz-obsolete/devices is the chipset also vp8/vp9 encoding ready. Is it only not supported yet in the software?

[odroid]

I can't find Exynos-5422 in the list of VP9 capable SoCs.
http://wiki.webmproject.org/hardware/socs

[ard]

I can't find Exynos-5422 in the list of VP9 capable SoCs.
http://wiki.webmproject.org/hardware/socs

But it is in the vp8 list.
The mfc @ exynos 5 can do vp8. Exynos 7 and higher also support vp9.
I also saw v4l2vp8enc in the gstreamer master branch.

[wiley]

So I've been trying to get a Logitech BRIO webcam (USB 3.0 + 4K or 1080p60) to work with the hardware encoding in the XU4. I've been working with gstreamer, and so far the best I can do is this:

gst-launch-1.0 v4l2src device=/dev/video0 ! \
'video/x-raw,width=1280,height=720,framerate=30/1,format=NV12' ! \
v4l2video11h264enc extra-controls="encode,video_bitrate=15728640" ! \
h264parse ! \
mux. \
alsasrc device=hw:1 ! \
"audio/x-raw, format=S16LE, rate=48000,channels=2" ! \
voaacenc bitrate=128000 ! \
aacparse ! \
audio/mpeg,mpegversion=4,stream-format=raw ! \
queue ! flvmux streamable=true name=mux ! \
progressreport ! \
rtmpsink location="<youtube stream server> live=true"

This works, the result looks pretty nice, but it's just 720p30.

For reference, the gstreamer output looks like this:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstAudioSrcClock
Redistribute latency...
Redistribute latency...
WARNING: from element /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Can't record audio fast enough
Additional debug info:
gstaudiobasesrc.c(847): gst_audio_base_src_create (): /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
Dropped 33600 samples. This is most likely because downstream can't keep up and is consuming samples too slowly.
progressreport0 (00:00:05): 4 seconds
progressreport0 (00:00:10): 9 seconds
progressreport0 (00:00:15): 14 seconds
progressreport0 (00:00:20): 19 seconds
progressreport0 (00:00:25): 24 seconds
progressreport0 (00:00:30): 29 seconds
progressreport0 (00:00:35): 34 seconds
progressreport0 (00:00:40): 39 seconds
progressreport0 (00:00:45): 44 seconds
progressreport0 (00:00:50): 49 seconds
progressreport0 (00:00:55): 54 seconds
progressreport0 (00:01:00): 59 seconds
progressreport0 (00:01:05): 64 seconds
progressreport0 (00:01:10): 69 seconds
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:01:14.060786061
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

and the kernel output looks like this:

Sep 05 05:22:37 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000
Sep 05 05:22:37 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000
Sep 05 05:22:37 odroid kernel: vidioc_g_parm:1721: Setting FPS is only possible for the output queue
Sep 05 05:22:37 odroid kernel: s5p-mfc 11000000.codec:: Encoding not initialised
Sep 05 05:22:38 odroid kernel: vidioc_g_parm:1721: Setting FPS is only possible for the output queue
Sep 05 05:23:51 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000

If I try the same pipeline with 1080p30, I get this instead:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstAudioSrcClock
Redistribute latency...
Redistribute latency...
WARNING: from element /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Can't record audio fast enough
Additional debug info:
gstaudiobasesrc.c(847): gst_audio_base_src_create (): /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
Dropped 35520 samples. This is most likely because downstream can't keep up and is consuming samples too slowly.
ERROR: from element /GstPipeline:pipeline0/v4l2video11h264enc:v4l2video11h264enc0: Failed to process frame.
Additional debug info:
gstv4l2videoenc.c(577): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline0/v4l2video11h264enc:v4l2video11h264enc0:
Maybe be due to not enough memory or failing driver
Execution ended after 0:00:11.420726313
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

and the kernel output looks like:

Sep 05 05:27:13 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000
Sep 05 05:27:13 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000
Sep 05 05:27:13 odroid kernel: vidioc_g_parm:1721: Setting FPS is only possible for the output queue
Sep 05 05:27:13 odroid kernel: s5p-mfc 11000000.codec:: Encoding not initialised
Sep 05 05:27:14 odroid kernel: vidioc_g_parm:1721: Setting FPS is only possible for the output queue
Sep 05 05:27:24 odroid kernel: s5p_mfc_watchdog:160: Time out during waiting for HW
Sep 05 05:27:24 odroid kernel: s5p_mfc_watchdog_worker:178: Driver timeout error handling
Sep 05 05:27:24 odroid kernel: vidioc_dqbuf:1284: Call on DQBUF after unrecoverable error
Sep 05 05:27:24 odroid kernel: vidioc_dqbuf:1284: Call on DQBUF after unrecoverable error
Sep 05 05:27:24 odroid kernel: usb 4-1.2: current rate 16000 is different from the runtime rate 48000
Sep 05 05:27:24 odroid kernel: s5p_mfc_handle_error:471: Interrupt Error: 0000001e
Sep 05 05:27:24 odroid kernel: s5p_mfc_close_mfc_inst:472: Err returning instance

Any idea what's going wrong here? The MFC codec is supposed to be capable of 1080p60, right?

[memeka]

try first w/o audio
if that works, try another audio encoding or another audio format

[wiley]

try first w/o audio
if that works, try another audio encoding or another audio format

I went ahead and tried just running the video to a testsink:

gst-launch-1.0 v4l2src device=/dev/video0 ! \
'video/x-raw,width=1920,height=1080,framerate=30/1,format=NV12' ! \
v4l2video11h264enc extra-controls="encode,video_bitrate=15728640" ! \
h264parse ! \
testsink

Unfortunately, I see the same MFC timeout and error.

I just tried two other things that seem to conflict, though:
- If I cut out the encoder completely and just run raw camera frames to the testsink, it works and I get reasonable CPU usage, which seems to imply the USB bus bandwidth isn't the problem (which makes sense - at 12 bits per pixel, this is 1920*1080*30*12 = 747 megabits/sec over a 5 gigabit bus).
- If I replace the v4l2src with a videotestsrc, the entire pipeline works. This seems to imply that the camera is the problem after all.

Now I'm really confused...

[wiley]

Well, if I assume the previous problem was, in fact, due to bandwidth constraints, the next logical step is to switch to MJPEG (which is needed for 1080p60 anyway). Looking into that, I can software-decode 60 jpegs per second, but the software decoder can't do NV12/NV21, so I have to do a colorspace conversion. This lowers the FPS down to ~12 (and switching to the hardware converter only gets ~14).

I should be able to get around this by using the hardware jpeg decoder to go straight to NV12, but I get this behavior

wiley@odroid:~$ gst-launch-1.0 videotestsrc ! \
> 'video/x-raw,width=1920,height=1080,framerate=60/1' ! \
> jpegenc ! \
> 'image/jpeg,width=1920,height=1080,framerate=60/1' ! \
> v4l2video21videodec ! \
> 'video/x-raw,width=1920,height=1080,framerate=60/1,format=NV12' ! \
> testsink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element /GstPipeline:pipeline0/v4l2video21videodec:v4l2video21videodec0: Failed to allocate required memory.
Additional debug info:
gstv4l2videodec.c(711): gst_v4l2_video_dec_handle_frame (): /GstPipeline:pipeline0/v4l2video21videodec:v4l2video21videodec0:
Buffer pool activation failed
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

I think the error is incorrect because the same thing occurs if I lower everything to 640x480@10, which should reduce the memory usage quite a bit. I do get this heap from the kernel:

Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3631564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3631564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3631564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3631564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3631564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3136564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3136564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3136564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3136564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3136564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3432564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3432564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3432564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3432564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x3432564e) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x59455247) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x59455247) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x59455247) invalid.
Sep 05 07:35:53 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x59455247) invalid.
Sep 05 07:35:54 odroid kernel: s5p-jpeg 11f50000.jpeg:: Fourcc format (0x59455247) invalid.

... but to me, that looks like it's just gstreamer probing the supported output formats from the jpeg decoder.

[wiley]

Has anyone managed to get the hardware jpeg decoder to work in gstreamer or ffmpeg?

[trungnmh90]

You can use jpeg encoder/decoder using V4L2 call, my problem is just don't know how to set jpeg compression on encoder. Both `VIDIOC_G_JPEGCOMP` and `VIDIOC_S_JPEGCOMP` return "Inappropriate ioctl for device".

Code: Select all

sudo v4l2-ctl -d /dev/video30 -D
Driver Info (not using libv4l2):
        Driver name   : s5p-jpeg
        Card type     : s5p-jpeg encoder
        Bus info      : platform:11f50000.jpeg
        Driver version: 4.14.11
        Capabilities  : 0x84208000
                Video Memory-to-Memory
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04208000
                Video Memory-to-Memory
                Streaming
                Extended Pix Format

Code: Select all

sudo v4l2-ctl -d /dev/video30 --get-jpeg-comp
VIDIOC_G_JPEGCOMP: failed: Inappropriate ioctl for device

Does anybody know how to set compression value for jpeg encoder? And I am sure it is there.

Code: Select all

sudo v4l2-ctl -d /dev/video30 --all
...
JPEG Compression Controls

             chroma_subsampling (menu)   : min=0 max=5 default=1 value=1
               restart_interval (int)    : min=0 max=3 step=65535 default=0 value=0
            compression_quality (int)    : min=0 max=3 step=1 default=3 value=3

[mad_ady]

It looks like there is a compression_quality attribute bound to /dev/video30 that you should be able to set with a similar command (I'll look it up)

[elector]

Hi guys

For the last half year or so (on and mostly off...) I've been trying to get my XU4 to encode a video/audio stream coming from a GoPro->HDMI->AJA UTAP->USB3.0->XU4 but have so far not been able to do so. Every time I seem to run into some (to me) weird issues. But now I'm trying again, and hope that you guys have some good ideas on how to do this.

I have installed ffmpeg and gst following the advices posted earlier in this thread.

root@odroid:~# uname -a
Linux odroid 4.14.12-99 #1 SMP PREEMPT Mon Jan 8 03:14:09 UTC 2018 armv7l armv7l armv7l GNU/Linux

root@odroid:~# ffmpeg -version
ffmpeg version 3.3.3-5 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) 20160609
[rest of output omitted, let me know if it's needed]

root@odroid:~# gst-launch-1.0 --version
gst-launch-1.0 version 1.11.91
GStreamer 1.11.91
https://launchpad.net/distros/ubuntu/+s ... treamer1.0


So... when I try a relative simple capture with gst I see the following, how come?

root@odroid:~# gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw,width=1280,height=720,framerate=25/1,format=NV12' ! v4l2video11h264enc extra-controls="encode,video_bitrate=15728640" ! h264parse ! testsink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(2939): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.059362369
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
root@odroid:~#

[EDIT]
Just as a test that the camera is working I can stream while encoding in MPEG2, at a low framerate though, but it works.

root@odroid:~# ffmpeg -re -f v4l2 -framerate 25 -video_size 1280x720 -i /dev/video0 -b:v 5000k -c:a aac -b:a 128k -loglevel info -f mpegts udp://192.168.0.1:1234
[not so interesting output omitted]
Input #0, video4linux2,v4l2, from '/dev/video0':
Duration: N/A, start: 7033.542778, bitrate: 368640 kb/s
Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 1280x720, 368640 kb/s, 25 fps, 25 tbr, 1000k tbn, 1000k tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg2video (native))
Press [q] to stop, [?] for help
No pixel format specified, yuv422p for MPEG-2 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
Output #0, mpegts, to 'udp://192.168.0.1:1234':
Metadata:
encoder : Lavf57.71.100
Stream #0:0: Video: mpeg2video (4:2:2), yuv422p, 1280x720, q=2-31, 5000 kb/s, 25 fps, 90k tbn, 25 tbc
Metadata:
encoder : Lavc57.89.100 mpeg2video
Side data:
cpb: bitrate max/min/avg: 0/0/5000000 buffer size: 0 vbv_delay: -1
frame= 326 fps=6.8 q=2.0 Lsize= 11450kB time=00:00:43.04 bitrate=2179.3kbits/s speed=0.892x x
[I hit CTRL+C here]
video:10580kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.219366%
Exiting normally, received signal 2.


Any advice greatly appreciated.

[memeka]

1. Not sure NV12 is accepted input. So try going first through v4l2video20convert (check that is the actual gst node) before the encoder.
2. Also use fakesink in your tests instead of testsink
3. Try puting some ! queue ! between elements, eg. before and after the encoder

[elector]

Thanks for your input. Unfortunately it didn't change much.

While tailing /var/log/syslog I got the folowing

root@odroid:~# gst-launch-1.0 v4l2src device=/dev/video0 ! v4l2video20convert ! queue ! 'video/x-raw,width=1280,height=720,framerate=25/1' ! queue ! v4l2video11h264enc extra-controls="encode,video_bitrate=15728640" ! queue ! h264parse ! queue ! fakesink
Setting pipeline to PAUSED ...
Jan 19 19:23:11 odroid kernel: [ 4418.152760] Wrong buffer/video queue type (1)
[cut 13 lines of the same as these two above and below]
Jan 19 19:23:11 odroid kernel: [ 4418.208589] Wrong buffer/video queue type (1)
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Jan 19 19:23:12 odroid kernel: [ 4418.213081] Wrong buffer/video queue type (1)
Jan 19 19:23:12 odroid kernel: [ 4418.462747] xhci-hcd xhci-hcd.3.auto: ERROR unknown event type 37
Jan 19 19:23:12 odroid kernel: [ 4418.555047] xhci-hcd xhci-hcd.3.auto: ERROR unknown event type 37
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(2939): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.327502941
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

I have also tried a few more ! queue ! statements, and v4l2video21convert but no go.

root@odroid:~# gst-inspect-1.0 | grep video4
video4linux2: v4l2src: Video (video4linux2) Source
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2: v4l2video10videodec: V4L2 Video Decoder
video4linux2: v4l2video11h264enc: V4L2 H.264 Encoder
video4linux2: v4l2video31videodec: V4L2 Video Decoder
video4linux2: v4l2video33videodec: V4L2 Video Decoder
video4linux2: v4l2video20convert: V4L2 Video Converter
video4linux2: v4l2video21convert: V4L2 Video Converter

Any good ideas on how to proceed?

[SamVDB²]

Here is the pipeline I use on a 4.14.13 kernel with a XU4: https://github.com/ardje/xu4-gst-quicks ... est.c#L518

It grabs video from /dev/video0 and audio from alsa card hw:1.

[mad_ady]

I'm reopening this thread because I've switched to kernel 4.14 and am now testing with Hardkernel's ffmpeg build. I'm on kernel 4.14.16-104 currently.

There are a few problems left. I've tried these three encoding tests that I tried initially (in the first thread) (not ok means playing with black screen):

Code: Select all

ffmpeg -i big_buck_bunny_720p_h264.mov -acodec aac -vcodec h264 -b:v 2M -pix_fmt nv21 bbb.mp4
frame=  518 fps=2.3 q=-0.0 Lsize=   29551kB time=00:09:56.48 bitrate= 405.9kbits/s dup=0 drop=13792 speed=2.61x    
video:4572kB audio:24861kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.401162%
//ok in mplayer, not ok in vlc, gstreamer. Also gets stuck after ~22s (518 frames). Required a reboot of the XU4 (HC2 actually) to unstuck. Even when it encoded without getting stuck, the video plays until this frame and afterwards is black. Weird.


ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -r 24 -i - -an  -vcodec h264 -b:v 2M -profile:v 10 -pix_fmt nv21 bbb.mp4
//ok in mplayer, not ok in vlc, gstreamer. High fps (85).


ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -r 24 -i - -i big_buck_bunny_720p_h264.mov -map 0:v:0 -vcodec h264 -b:v 2M -profile:v 10 -pix_fmt nv21  -map 1:a:0 -c:a:1 aac bbb.mp4
//locks every time.

I think the black screen problem is related to what memeka said about V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. Any idea how I can check how the header is set or how to set it "the compliant way"?

[memeka]

You don’t need annexb conversion with latest ffmpeg

[mad_ady]

That's good to know, though it has much higher encode fps.
I will also test low bitrate encoding, but I was hoping to get universally playable files with ffmpeg.
Maybe I will look into gstreamer after all. I take it that the stock gstreamer won't do, right?

[mad_ady]

I've restarted playing with ffmpeg before I give up on it for good

If I produce mkv output instead of mp4, it's played better on other players:

Code: Select all

ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -y -r 24 -i - -an  -vcodec h264 -b:v 500k -profile:v 10 -pix_fmt nv21 bbb.mkv

* this plays fine in vlc and gstreamer based players. On Android it plays with MxPlayer, but the video is not seekable for some reason!

This pipeline encodes the first 518 frames only, and then hangs, but without any dmesg output:

Code: Select all

ffmpeg -y -i big_buck_bunny_720p_h264.mov -acodec aac -vcodec h264 -b:v 2M -pix_fmt nv21 bbb.mkv

I am still not able to encode sound and video at the same time using the first pipeline:
* this doesn't work

Code: Select all

ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -y -r 24 -i - -i big_buck_bunny_720p_h264.mov -map 0:v:0 -vcodec h264 -b:v 2M -profile:v 10 -pix_fmt nv21 -map 1:2 -c:a aac bbb.mkv

* this should work (audio muxed in in the second step):

Code: Select all

ffmpeg -i big_buck_bunny_720p_h264.mov -codec:v copy -codec:a none -bsf:v h264_mp4toannexb -f rawvideo - | ffmpeg -y -r 24 -i - -an  -vcodec h264 -b:v 500k -profile:v 10 -pix_fmt nv21 bbb-video.mkv
ffmpeg -i bbb-video.mkv -i big_buck_bunny_720p_h264.mov -map 0:v:0 -vcodec copy -map 1:a:0 -c:a aac bbb.mkv

... but doesn't. Produced a file with unplayable video, but audio works . When encoding the second step I do get this error:

Code: Select all

[h264 @ 0x4c6a30] missing picture in access unit with size 23
[h264 @ 0x4c6a30] no frame!

Does anyone have other suggestions about what I'm doing wrong? Or should I throw in the towel and start reading on gst instead?
It does seem that ffmpeg produces non-standard output, that even ffmpeg can't read:

Code: Select all

root@odroid:/media/home/home/temp# ffmpeg -i bbb-video.mkv -vcodec copy bbb2.mkv
ffmpeg version 3.3.3-5ubuntu1 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 5.4.1 (Ubuntu/Linaro 5.4.1-2ubuntu1~16.04) 20160904
  configuration: --prefix=/usr --extra-version=5ubuntu1 --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --disable-d3d11va --disable-dxva2 --disable-nvenc --disable-vaapi --disable-vda --disable-vdpau --disable-videotoolbox --enable-v4l2_m2m --enable-libdc1394 --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
[h264 @ 0x4c79c0] missing picture in access unit with size 23
[h264 @ 0x4c79c0] no frame!
Input #0, matroska,webm, from 'bbb-video.mkv':
  Metadata:
    ENCODER         : Lavf57.71.100
  Duration: 00:09:56.25, start: 0.000000, bitrate: 477 kb/s
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 24 fps, 24 tbr, 1k tbn, 2k tbc (default)
    Metadata:
      ENCODER         : Lavc57.89.100 h264_v4l2m2m
      DURATION        : 00:09:56.250000000
Output #0, matroska, to 'bbb2.mkv':
  Metadata:
    encoder         : Lavf57.71.100
    Stream #0:0: Video: h264 (High) (H264 / 0x34363248), yuv420p(progressive), 1280x720, q=2-31, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      ENCODER         : Lavc57.89.100 h264_v4l2m2m
      DURATION        : 00:09:56.250000000
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame=14310 fps=0.0 q=-1.0 Lsize=   34733kB time=00:09:56.20 bitrate= 477.2kbits/s speed= 644x    
video:34585kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.428139%

[meveric]

I just tried on my image as well:
first I tried using hardware decoder and hardware encoder:

Code: Select all

ffmpeg -vcodec h264_v4l2m2m -i big_buck_bunny_720p_h264.mov -vcodec h264_v4l2m2m -b:v 2M -profile:v 10 -pix_fmt nv21 bbb-video.mkv
[h264_v4l2m2m @ 0x4c9000] driver 'exynos-gsc' on card 'exynos-gsc gscaler'
    Last message repeated 1 times
[h264_v4l2m2m @ 0x4c9000] driver 's5p-mfc' on card 's5p-mfc-enc'
[h264_v4l2m2m @ 0x4c9000] driver 's5p-mfc' on card 's5p-mfc-dec'
[h264_v4l2m2m @ 0x4c9000] Using device /dev/video10
[h264_v4l2m2m @ 0x4c9000] driver 's5p-mfc' on card 's5p-mfc-dec'
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> h264 (h264_v4l2m2m))
  Stream #0:2 -> #0:1 (aac (native) -> vorbis (libvorbis))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0x4c9000] output  POLLERR
[h264_v4l2m2m @ 0x4bcb90] driver 'exynos-gsc' on card 'exynos-gsc gscaler'
    Last message repeated 1 times
[h264_v4l2m2m @ 0x4bcb90] driver 's5p-mfc' on card 's5p-mfc-enc'
[h264_v4l2m2m @ 0x4bcb90] Using device /dev/video11
[h264_v4l2m2m @ 0x4bcb90] driver 's5p-mfc' on card 's5p-mfc-enc'
[h264_v4l2m2m @ 0x4bcb90] h264_v4l2m2m encoder: enabling bit rate control: 500000
[h264_v4l2m2m @ 0x4bcb90] h264 profile not found
[h264_v4l2m2m @ 0x4bcb90] Encoder adjusted: qmin (0), qmax (51)

(as you can see it uses both /dev/video10 for decoding and /dev/video11 for encoding)
The result was not good, after about 25~30 seconds, the decoder produced an error and the encoder started dropping all frames. Also the speed was about 30 FPS for encoding.
Then I switched to software decoder:

Code: Select all

ffmpeg -i big_buck_bunny_720p_h264.mov -vcodec h264_v4l2m2m -b:v 2M -profile:v 10 -pix_fmt nv21 bbb-video.mkv
[h264_v4l2m2m @ 0x49cfb0] driver 'exynos-gsc' on card 'exynos-gsc gscaler'
    Last message repeated 1 times
[h264_v4l2m2m @ 0x49cfb0] driver 's5p-mfc' on card 's5p-mfc-enc'
[h264_v4l2m2m @ 0x49cfb0] Using device /dev/video11
[h264_v4l2m2m @ 0x49cfb0] driver 's5p-mfc' on card 's5p-mfc-enc'
[h264_v4l2m2m @ 0x49cfb0] h264_v4l2m2m encoder: enabling bit rate control: 500000
[h264_v4l2m2m @ 0x49cfb0] h264 profile not found
[h264_v4l2m2m @ 0x49cfb0] Encoder adjusted: qmin (0), qmax (51)

The result was very good, in fact it was way better then using hardware decoder. First, the video was completely converted without dropping of frames. Second, it was encoded in 60 FPS, so way faster than using hardware decoder.
The result seemed to be fine on mpv (software decoding) ffplay (software decoding) ffplay (hardware decoding) and even on vlc and cvlc.

I only tested via VNC so although it looked fine to me, it could be that the colors are off, as I use remote control to look at it (but I think it should be fine).
Edit:
yeah colors seem fine, just tried x11grab again and it's still horrible and colors are broken, so the above should work.

[mad_ady]

Thanks for that input. So, the difference in order to do hardware decoding is that you specify '-vcodec h264_v4l2m2m' before '-i'. The thing is - for me with the "stock" ffmpeg from HK I get the same behavior (hardware decoding) even when I don't specify '-vcodec h264_v4l2m2m' before '-i'.
But at least we know the lockup is an issue in the decoder, not encoder.

My ffmpeg version is '3.3.3-5ubuntu1', and lists these h264 decoders:

Code: Select all

 V..... h264_v4l2m2m         V4L2 mem2mem H.264 decoder wrapper (codec h264)
 VFS..D h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10

I played with it and if I can force it to use the software decoder by using this instead:

Code: Select all

ffmpeg -vcodec h264 -i big_buck_bunny_720p_h264.mov -vcodec h264_v4l2m2m -b:v 2M -profile:v 10 -pix_fmt nv21 bbb-video.mkv

Thanks for the input again - I shall return to playing now.

[mad_ady]

I've played with encoding a bit more and I'm happier with the results (not happiest though).

So - doing software decoding to avoid lockups (thanks @meveric). Also, encoding sound together with video works this time.
* Encoding to mp4 results in a file which is not playable on Android and in gstreamer based players. Booo

Code: Select all

fmpeg -vcodec h264 -i big_buck_bunny_720p_h264.mov -vcodec h264_v4l2m2m -b:v 500k -profile:v 10 -pix_fmt nv21 -acodec aac  bbb.mp4

* Encoding to mkv + sound results in a file which is playable everywhere. Yay!

Code: Select all

ffmpeg -vcodec h264 -i big_buck_bunny_720p_h264.mov -vcodec h264_v4l2m2m -b:v 500k -profile:v 10 -pix_fmt nv21 -acodec aac  bbb.mkv

I will extend my tests to other input formats and add truncating the input if necessary to respect the 64bit boundary.

[elatllat]

Try "-movflags +faststart" when making the mp4.
also ffprobe a working and non-working mp4 to see what the diff.

[mad_ady]

Thanks for the idea. I tried it but ffprobe fails to load both files (with and without those options):

Code: Select all

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x4ff010] st: 0 edit list: 1 Missing key frame while searching for timestamp: 0
[h264 @ 0x5002a0] no frame!

It's as if the stream starts without a keyframe? I'll need to see if I can force a keyframe. Strange that mkv does not complain, and that's just a different container - the video stream should be the same, right?

[meveric]

have you tried using mpeg4_v4l2m2m instead when making mp4 file?
When I try with h264_v4l2m2m I can't open the .mp4 in vlc but if I use mpeg4_v4l2m2m instead it seems to be working.

Code: Select all

Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mpeg4 (mpeg4_v4l2m2m))
  Stream #0:2 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[mpeg4_v4l2m2m @ 0x4ddae0] driver 'exynos-gsc' on card 'exynos-gsc gscaler'
    Last message repeated 1 times
[mpeg4_v4l2m2m @ 0x4ddae0] driver 's5p-mfc' on card 's5p-mfc-enc'
[mpeg4_v4l2m2m @ 0x4ddae0] Using device /dev/video11
[mpeg4_v4l2m2m @ 0x4ddae0] driver 's5p-mfc' on card 's5p-mfc-enc'
[mpeg4_v4l2m2m @ 0x4ddae0] h264_v4l2m2m encoder: enabling bit rate control: 2000000
[mpeg4_v4l2m2m @ 0x4ddae0] mpeg4 profile not found
[mpeg4_v4l2m2m @ 0x4ddae0] Encoder adjusted: qmin (1), qmax (31)

[mad_ady]

I had no idea that was a thing...
I thought it referred to the encoding codec (h264), not container(mpeg4).
I will try it, thanks!

[elatllat]

maybe a 'muxing' difference vs 'encoding' or 'container' ~

[mad_ady]

So, I have tried

Code: Select all

ffmpeg -vcodec h264 -i big_buck_bunny_720p_h264.mov -vcodec mpeg4_v4l2m2m -b:v 500k -profile:v 10 -movflags +faststart -pix_fmt nv21 -acodec aac  bbb_mpeg4.mp4

But the output doesn't play in mplayer nor in Android players. VLC plays it though.

[dumbledore]

I'm having issues with the hardware encoder crashing as well. (see here: viewtopic.php?t=30548). I'm able to encode using some of the pipelines recommended here, but the encoder seems to have many more issues than the software encoder (which works with seemingly anything you throw at it).

[_om_]

Hi guys,
I've faced hw encoder locks as well with last 4.14, while decoder is mostly works ok.
So, what is the recommended Ubuntu image so far to drive both hw decoder and encoder (h264 is enough) with gstreamer at the same time?
Or if there is no stable one, may be a linux-kernel deb package exists with proper config.
Thank you.

[memeka]

I have recompiled ffmpeg 3.4.2 with some extra patches, and encoder seems to be more stable.
I get no issues using hw transcoding now fully.

test for bbb 1080 trailer:
sw dec + hw enc: speedup = 1.6x, 2 big cores used @ ~55% each
hw dec + hw enc: speedup = 1.2x, 1 big core used @ ~40%

it's a bit slower, but using less than half the CPU of sw dec.

[odroid]

@memeka,
Great! Can you please share the patches?
We want to update the ffmpeg package in our PPA server.

[memeka]

@memeka,
Great! Can you please share the patches?
We want to update the ffmpeg package in our PPA server.

https://github.com/mihailescu2m/FFmpeg branch v4l2m2m-3.4.2
just clone and dpkg-buildpackage