Which OS/Kernel for HW acceleration?

Post Reply
shrub77
Posts: 2
Joined: Mon Feb 07, 2022 10:04 pm
languages_spoken: english
ODROIDs: Odroid N2+ 4gb
Has thanked: 0
Been thanked: 0
Contact:

Which OS/Kernel for HW acceleration?

Post by shrub77 »

Hi,

I am a new owner of the N2+. I purchased it mainly for the impressive CPU/GPU performance on the product page, but it seems the software support/drivers for the GPU is unclear. My main use-case is to run a headless server with jellyfin + hardware acceleration. Is this currently possible with the N2+? If so, which OS provides this? Currently I am using Armbian 21.08 but it seems this does not include the kernel.

I see other OSes produced by forum users tobetter and meveric but it is unclear to me if either offers what I am looking for. Overall, I am confused about what software is available to us N2 users, the wiki looks out of date.

Thanks in advance for any info!

User avatar
mad_ady
Posts: 10585
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4 (HC1, HC2), C1+, C2, C4 (HC4), N1, N2, H2, Go, Go Advance, M1
Location: Bucharest, Romania
Has thanked: 643 times
Been thanked: 901 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by mad_ady »

The stock kernel supports decoding and encoding via amlogic's drivers, but there are few open source tools that make use of them (kodi, crashoverride's c4play). In the past crashoverride released c2enc which might have a branch that works on N2, but tools like jellyfin don't make use of it.
The mainline kernel brings decent gpu support via reverse engineering, but the vpu is underdeveloped. Currently it can decode some h264 streams and I think work was being dine to support h265. I don't think the encoder is supported though.

If jellyfin works on Android, that might be the best supported OS in terms of acceleration.

crashoverride
Posts: 5658
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 534 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by crashoverride »

I took a look at the jellyfin code repo. It appears to be hard-coded to use only FFMPEG. There does not appear to be any interface or base class to offer alternative means of hardware acceleration.

This means that any hardware acceleration would have to be added to FFMPEG itself or through the use of an API that FFMPEG can consume such as V4L2 or VAAPI. Neither option is a good choice as both are very poorly designed. However, if forced to choose one, it would be VAAPI.

VAAPI is created by Intel:
https://01.org/linuxmedia/vaapi

It is described as:
VAAPI (Video Acceleration API) is an open-source library and API specification
In reality, I have been unable to find the aforementioned "specification". The documentation links are all "404" (not found).
https://01.org/linuxmedia/documentation ... umentation

A year ago I considered creating a VAAPI wrapper for ODROID-C4/N2. Having previous experience with other video APIs, I did not consider the task to be impossible. I was wrong! Its one thing to consume a poorly designed and documented API, but it's an entirely different thing to create an implementation of it.

FWIW, some documentation can be found at the link below. I have yet to find the "specification" documentation.
http://intel.github.io/libva/

If the above sounds like a rant, that is because it is. Developers are frustrated with the options for media/graphics on Linux. This is the reason for the persistent state of "my PC supports it but my ARM SBC doesn't". Intel makes chips for PCs, not SBCs.

[edit]
In fairness to Intel, I am sure that I am as bad at chip design as they are at software design!
;)
These users thanked the author crashoverride for the post:
odroid (Tue Feb 08, 2022 4:54 pm)

chewitt
Posts: 177
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 132 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by chewitt »

@crashoverride I'd pick V4L2 as it's more likely to attract the attention of folks in the Linux media community who'd help with development (guiding, but not coding). The gene pool of people with some driver skills and knowledge of Amlogic silicon is shockingly small, so if you have any interest in that direction I'd love to chat more.

User avatar
mad_ady
Posts: 10585
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4 (HC1, HC2), C1+, C2, C4 (HC4), N1, N2, H2, Go, Go Advance, M1
Location: Bucharest, Romania
Has thanked: 643 times
Been thanked: 901 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by mad_ady »

Isn't there a fundamental problem that one API is stateful and one is stateless?

crashoverride
Posts: 5658
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 534 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by crashoverride »

chewitt wrote:
Wed Feb 09, 2022 2:51 am
I'd pick V4L2 as it's more likely to attract the attention of folks in the Linux media community who'd help with development (guiding, but not coding).
In my opinion, it would be a complete waste of time to invest in V4L2 development. People far more talented than me began the decoder effort in 2018 (its 2022 at the time time of this post), and as far as I can tell, it has not even made it to mainline yet (Linux 5.16.8 at the time of this post). In the context of this discussion (transcoding), both a decoder and encoder are required. I do not believe that it would be a realistic expectation to complete this work targeting V4L2 before one of two things happens: the platform becomes obsolete or Amlogic releases a V4L2 driver as part of their next kernel (5.x). Further complicating the matter is the lack of a stable kernel driver API. A V4L2 driver requires constant maintenance whith each kernel release. This implies the need for a maintainer which I have no interest in doing.

I would propose an alternative, but it would garner zero interest from anyone in the linux media community. It would be far simpler to create a V4L2 driver that just wraps the current Amlogic kernel (4.9) codec implementation. This would have no chance of ever being "upstreamed".

Another proposal that is a compromise is to introduce the ability to create V4L2 drivers in user-space. This would be similar to how FunctionFS allows USB drivers to be composed in user-space. I would really like to see the V4L2 maintainers embrace this concept. However, it is also useless in the context of this discussion. Due to the previously mentioned lack of a stable kernel driver API, it could not be compiled on the current Amlogic 4.9 kernel (assuming new drivers would be authored against mainline).

The above helps to inform regarding my statement of VAAPI preference over V4L2. VAAPI is a far more stable API than V4L2 or AVCodec (FFMPEG). It is also a user-space API which greatly eases development.

[edit]
For those interested in my opinion regarding V4L2 being poorly designed, it may help to know that there is not actually a codec API in V4L2. The "M2M" (memory to memory) is conceptualized as a video output layer glued to a video capture device. The "output" is actually the input (codec data) and the "capture" is actually the output (picture) for a decoder.

[edit2]
I actually had to make multiple edits to my previous edit! That is how obtuse the V4L2 API is for developers. :lol:
These users thanked the author crashoverride for the post:
brad (Thu Feb 10, 2022 5:12 pm)

brad
Posts: 1595
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 C4 HC4 N1 N2 N2+ H2 H2+ M1 (64 bit ftw)
Location: Australia
Has thanked: 178 times
Been thanked: 251 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by brad »

@chewitt @crashoverride I spend some time last year trying to get a grasp on vappi / v4l2 but it is a large learning curve and I really don't have the understanding or experience.

I found Chromium source an interesting place to look for examples for how userspace been done over the years for other implementations (including android, chromeos, arm, amlogic and other SOC's). For reference the V4L2 codebase https://chromium.googlesource.com/chrom ... a/gpu/v4l2

The landscape is constantly changing (kernel API's changing) but for mainline kernel at does at least look like a standardized API is on the cards and has more traction since later 5.x kernels.

I wish I had more time and expertise to help out. There is also the concept of having a VAPPI wrapper for the kernel v4l2 implementation but there is not much in the way of development these days and the amlogic implementation is different so that would need work as well (stateless vs non stateful) - https://github.com/bootlin/libva-v4l2-request

chewitt
Posts: 177
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 132 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by chewitt »

The terminology around GBM/V4L2 is a bit obtuse in places, but it is _the_ framework the kernel has chosen to adopt, and as rate of adoption increases (all new drivers are being written around it) the body of knowledge and upstream "prior art" is growing. Sometime soon the Pi Foundation will hopefully start the push of userspace changes for FFmpeg which will help both stateless and stateful move forwards (gstreamer is already well supported by its mostly-Collabora maintainers). All efforts I've seen that start around VAAPI in userspace "as it's easier" capitulate to a direct V4L2 design once the developer gets their heads around the kernel V4L2 code (which they inevitably have to do). It's how Kodi and ffmpeg stateless V4L2 started. The bootlin code linked above is dormant for the last two years now for the same reason.

The V4L2 stateless UAPIs are finalised for most codecs now (H264, HEVC, VP9, VP8?, MPEG1/2) but stateful is still in staging .. hence the Amlogic driver is still in staging. It works well with H264 and VP9, with 8-bit output. HEVC and 10-bit ouptut are more complex over different hardware generations and sadly that work stalled when Maxime had to stop for health reasons. Some work has been done since to evolve ffmpeg (on the userspace side) and HEVC (kernel side) on GXBB/GXL/GXM hardware and I have 4K HDR etc. working quite well on GXL/GXM devices. Finishing HEVC and 10-bit (and newer hardware) and fixing MPEG1/2 support is proving hard to resource. The gap is not so huge as people like to make out .. but it needs a certain skillset and mindset.

Amlogic will continue to upstream core board support for new hardware as it arises, but their media drivers are achitecturally old and violate modern norms/rules for memory use and management. They are also complex due to being written as a toolkit to expose every possible silicon feature; whereas we only need a subset of those capabilities (the ones that are actually used) to go upstream. That again means the task is not as complicated as the vendor sources make out, but it still needs to be done. There are some commercial interests slowly taking an interest as the industrial userbase for Amlogic grows, but commercial interests have a repeating habit of starting work and then "sorry, mangement priorities changed and we can't spend more hours on this untiil next month" so IMHO it needs a mix of commercial and community brains to create and carry the momentum.

Low hanging fruit for learning are the upstream (staging) MPEG1/2 codec which was working when it was first submitted but is currently broken. It's massively more simple than HEVC and probrbaly doesn't need much to resolve whatever oversight or bitrot has occurred. I'm considering offering a bounty to get it fixed.. LE has been given some funds to sponsor that kind of thing; I'm just not sure how much to put on it, perhaps USD $500 for a working patch submitted to the kernel. I also believe a VENC (encoder) shouldn't be so much effort, and would be quite strategic at attracting more commercial interest; which in turn attracts more talent towards gaps in VDEC support. I'd also like to see ge2d (which exists upstream for AXG devices) to be extended to a wider range of hardware, and then improved to add HDR > SDR conversions. Those things are larger (but still smaller) efforts and would need larger bounties.. but if that approach works to get things done, I know how and where to raise funds. Unfortunately the $$ sums needed for proper commercial work (Baylibre, etc.) are too large. We need to break down the bigger problems into smaller (more achievebale and lower-cost) pieces.
These users thanked the author chewitt for the post (total 2):
brad (Wed Feb 09, 2022 3:07 pm) • rooted (Wed Feb 09, 2022 7:58 pm)

User avatar
rooted
Posts: 9436
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 758 times
Been thanked: 479 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by rooted »

Will there ever be a modern SoC running mainline kernel or even LTS with a fully functional GPU and VPU? I know a lot of developers are working hard to achieve such a thing but is this the goal of any company such as Rockchip or Amlogic?

It seems like the task is improbable if not impossible.

chewitt
Posts: 177
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 132 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by chewitt »

Sure. There _are_ now Linux GBM/V4L2 standards and the next generations of chips will increasingly conform to them because upstream says "this is the way" and if you do something else your code doesn't get merged. Google is also banging the drum on reducing upstream/downstream divergence for Android very hard - although the major area of difference is all about video rendering (it's where the vendor has unique IP, everything else is an assembly of common ARM ecosystem stuff). Existing SoC generations will continue to be a mixed bag as todays' commercial money invests in generation next not generation past, but over time and kernel iterations the problem will continue to reduce and older lesser-supported hardware continually grandfathers itself out of scope. Users get upset and pissy because there is no quick now/today fix (without lots of money) and some vendors will continue to have no interest in upstream, but the majority do care (and Google is telling them to care, or else) so the future looks a lot more standards based than it has ever looked before. Hell froze over recently with nVidia capitulating on GBM vs EGL Streams to avoid missing the Mesa/Wayland train; it's not V4L2 but it's another piece of the Linux graphics puzzle falling into line with maket acceptance of the now-established standards.

I'd argue Rockchip has the most-complete support for current chips today/now, with Allwinner a close second as both of them are mostly derived from similar Hantro IP blocks and share a lot of code (and development effort). Then SoCs like iMX8 which won't figure on most HK forum users' radar because they're not consumer items. Mediatek have also firmly embraced upstream and have made huge strides in support for their hardware - but again not typical HK consumer stuff. Amlogic are not anywhere near the top of the list of offenders, but they are also nowhere near the bottom (Amlogic's vendor kernel is a thing of great beauty compared to Realtek sources I've seen).

Final comment on standards https://xkcd.com/927/
These users thanked the author chewitt for the post (total 2):
istanbulls (Wed Feb 09, 2022 5:51 pm) • rooted (Wed Feb 09, 2022 7:57 pm)

brad
Posts: 1595
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 C4 HC4 N1 N2 N2+ H2 H2+ M1 (64 bit ftw)
Location: Australia
Has thanked: 178 times
Been thanked: 251 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by brad »

chewitt wrote:
Wed Feb 09, 2022 3:13 pm
Final comment on standards https://xkcd.com/927/
I was going to have a good look at the standards but :D

User avatar
rooted
Posts: 9436
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 758 times
Been thanked: 479 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by rooted »

@chewitt thanks for that answer, it's really nice to hear companies are making strides towards this goal. It will add much value and longevity to the hardware we purchase when it's not already half depreciated at time of sale.

chewitt
Posts: 177
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 132 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by chewitt »

brad wrote:
Wed Feb 09, 2022 6:13 pm
I was going to have a good look at the standards but :D
Ignore docs, it's all rather confusing and intuition will get you further. Just get some MPEG2 media and start adding printk stuff in the vdec to understand how it works and where it's broken. Use https://github.com/jc-kynesim/rpi-ffmpe ... i_import_1 for ffmpeg and my Linux 5.16.y branch to pick-up some needed vdec patches. NB: I'm serious about putting a bounty on fixes and improvements.

brad
Posts: 1595
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 C4 HC4 N1 N2 N2+ H2 H2+ M1 (64 bit ftw)
Location: Australia
Has thanked: 178 times
Been thanked: 251 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by brad »

chewitt wrote:
Wed Feb 09, 2022 11:08 pm
Ignore docs, it's all rather confusing and intuition will get you further. Just get some MPEG2 media and start adding printk stuff in the vdec to understand how it works and where it's broken. Use https://github.com/jc-kynesim/rpi-ffmpe ... i_import_1 for ffmpeg and my Linux 5.16.y branch to pick-up some needed vdec patches. NB: I'm serious about putting a bounty on fixes and improvements.
I'm having a look out of interest and learning, let's see how far that gets me.

With a standard 5.16.x getting similar issues to what appears to be fixed with the patches in your tree (trying your branch shortly), additionally height and widths are not as expected.

Would I be safe to assume that there is no public documentation on the amlogic video codec firmware besides the code within the amlogic's buildroot?

chewitt
Posts: 177
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 132 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by chewitt »

brad wrote:
Mon Feb 14, 2022 2:30 pm
Would I be safe to assume that there is no public documentation on the amlogic video codec firmware besides the code within the amlogic's buildroot?
Correct. Prior art from the vendor kernels is all we have. It may also be useful to compare https://github.com/Elyotna/linux/commit ... 4l2-m2m-pr with what is now upstream. IIRC the rough time-sequence of events starts with Maxime sending the initial (MPEG1/2) driver submission - I'd have to assume it worked at this point. Then H264 was added, and work on VP9 started. Then the first formal draft of V4L2 stateful API was made and this triggered a large round of "conformance" rework on H264; as by this point Google was funding things so all the effort was on H264 and VP9 so rework on MPEG2 was low priority, and ultimately Maxime never quite got around to it before he stopped work. Neil subsequently fixed up the H264 and VP9 bits to complete Google's work-packages, but as funding had effectively stopped at that point MPEG2 was again not looked at. I have a hunch that we don't do the right thing on the ffmpeg side (don't send the right data) but it's not done wrongly enough to trigger driver errors (esparser debug appears to show a stream of data) but as we're not sending the right data in, we don't get any data out of the firnware. If you look at the quirk JC added in the ffmpeg branch to handle the different init needed by Amlogic (vs RPi) for H264/HEVC .. it may be that the quirk is also codec dependent and something different again is needed for MPEG2. As I don't actually write code I wouldn't assign too much value to my hunches though :)
These users thanked the author chewitt for the post:
brad (Wed Feb 16, 2022 4:06 pm)

brad
Posts: 1595
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 C4 HC4 N1 N2 N2+ H2 H2+ M1 (64 bit ftw)
Location: Australia
Has thanked: 178 times
Been thanked: 251 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by brad »

I now have *some* understanding of the design but there is so much information to unpack, my current understanding is something like this

ffmpeg / mpv grabs some capabilities of video0, test's the capabilities and decides it is ready to go. The vdec set's up the io mem buffers and asks to start the streaming thread.

As part of the stream startup the vdec sets up formats but it fails to setup the mpeg capture side with an invalid ioctl on device video0 (mostly silently on the VIDIOC_S_SELECTION):

Code: Select all

Feb 18 22:14:30 kvm kernel: [ 2860.880298][ T2981] DEBUG: codec_mpeg12_start - allocated ram workspace_vaddr 00000000631f9c77 workspace_paddr 0000000019942349 SIZE_WORKSPACE 20000
Feb 18 22:14:30 kvm kernel: [ 2860.880312][ T2981] DEBUG: codec_mpeg12_start - SESSION width 780 height 438 colorspace 0, ycbcr_enc 0 quantization 0 xfer_func 0
Feb 18 22:14:30 kvm kernel: [ 2860.880317][ T2981] DEBUG: codec_mpeg12_start - set canvases return = 0
Feb 18 22:14:30 kvm kernel: [ 2860.880318][ T2981] DEBUG: codec_mpeg12_start - Writing out DOS registers
Feb 18 22:14:30 kvm kernel: [ 2860.880458][ T2981] video0: VIDIOC_STREAMON: type=vid-out-mplane
Feb 18 22:14:30 kvm kernel: [ 2860.880489][ T2981] video0: VIDIOC_G_FMT: type=vid-cap-mplane, width=1920, height=1080, format=NM12 little-endian (0x32314d4e), field=none, colorspace=0, num_planes=2, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
Feb 18 22:14:30 kvm kernel: [ 2860.880500][ T2981] plane 0: bytesperline=1920 sizeimage=2097152
Feb 18 22:14:30 kvm kernel: [ 2860.880502][ T2981] plane 1: bytesperline=1920 sizeimage=1048576
Feb 18 22:14:30 kvm kernel: [ 2860.880508][ T2981] video0: VIDIOC_S_SELECTION: error -25: type=vid-cap, target=0, flags=0x0, wxh=1920x1080, x,y=0,0
Feb 18 22:14:30 kvm kernel: [ 2860.880518][ T2981] video0: VIDIOC_G_FMT: type=vid-cap-mplane, width=1920, height=1080, format=NM12 little-endian (0x32314d4e), field=none, colorspace=0, num_planes=2, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
Feb 18 22:14:30 kvm kernel: [ 2860.880526][ T2981] plane 0: bytesperline=1920 sizeimage=2097152
Feb 18 22:14:30 kvm kernel: [ 2860.880528][ T2981] plane 1: bytesperline=1920 sizeimage=1048576
The capture buffers are all queued (8 of them) but when it tries to wakeup and dequeue the buffers it goes wrong (Buffer X done but it doesn't exist in m2m_ctx"). capture thread is now hung as we only have 8 buffers and they are all still consumed.

Code: Select all

Feb 18 22:14:30 kvm kernel: [ 2860.921533][ T2981] video0:
Feb 18 22:14:30 kvm kernel: [ 2860.921534][    C2] DEBUG: codec_mpeg12_isr - irq wake
Feb 18 22:14:30 kvm kernel: [ 2860.921535][ T2981] VIDIOC_DQBUF: 00:00:00.000000 index=0, type=vid-out-mplane, request_fd=0, flags=0x00004001, field=none, sequence=80, memory=mmap
Feb 18 22:14:30 kvm kernel: [ 2860.921541][  T949] DEBUG: codec_mpeg12_threaded_isr
Feb 18 22:14:30 kvm kernel: [ 2860.921542][ T2981] plane 0: bytesused=14718, data_offset=0x00000000, offset/userptr=0x0, length=1555328
Feb 18 22:14:30 kvm kernel: [ 2860.921544][  T949] DEBUG: codec_mpeg12_update_dar  - seq 80010013 ar 3
Feb 18 22:14:30 kvm kernel: [ 2860.921545][ T2981] timecode=00:00:00 type=0, flags=0x00000000, frames=0, userbits=0x00000000
Feb 18 22:14:30 kvm kernel: [ 2860.921547][  T949] DEBUG: codec_mpeg12_threaded_isr - reg 3 buffer_index 2  offset cf8d4  pic_info 19000  is_progressive 8000  field 1
Feb 18 22:14:30 kvm kernel: [ 2860.921550][  T949] DEBUG: codec_mpeg12_threaded_isr - reg 3 - SESSION
Feb 18 22:14:30 kvm kernel: [ 2860.921550][  T949]  width 780 height 438 colorspace 0 ycbcr_enc 0 quantization 0 xfer_func 0
Feb 18 22:14:30 kvm kernel: [ 2860.921553][  T949] DEBUG: codec_mpeg12_threaded_isr - reg 3 - SESSION canvas_num 0 vififo_size 1000000 last_irq_jiffies 10009c502 last_offset 282d1d wrap_count 0
Feb 18 22:14:30 kvm kernel: [ 2860.921565][  T949] meson-vdec ff620000.video-decoder: Buffer 2 done but it doesn't exist in m2m_ctx
Feb 18 22:14:30 kvm kernel: [ 2860.921567][ T2981] video0: VIDIOC_QBUF: 00:00:00.000000 index=0, type=vid-out-mplane, request_fd=0, flags=0x0000400b, field=none, sequence=0, memory=mmap
Feb 18 22:14:30 kvm kernel: [ 2860.928827][    C2] DEBUG: codec_mpeg12_isr - irq wake
Feb 18 22:14:30 kvm kernel: [ 2860.928827][ T2981] plane 0: bytesused=10469, data_offset=0x00000000, offset/userptr=0x0, length=1555328
Feb 18 22:14:30 kvm kernel: [ 2860.928829][  T949] DEBUG: codec_mpeg12_threaded_isr
Feb 18 22:14:30 kvm kernel: [ 2860.928830][ T2981] timecode=00:00:00 type=0, flags=0x00000000, frames=0, userbits=0x00000000
Feb 18 22:14:30 kvm kernel: [ 2860.928831][  T949] DEBUG: codec_mpeg12_update_dar  - seq 80010013 ar 3
Feb 18 22:14:30 kvm kernel: [ 2860.928834][  T949] DEBUG: codec_mpeg12_threaded_isr - reg 4 buffer_index 3  offset 107024  pic_info 19000  is_progressive 8000  field 1
Feb 18 22:14:30 kvm kernel: [ 2860.928837][  T949] DEBUG: codec_mpeg12_threaded_isr - reg 4 - SESSION
Feb 18 22:14:30 kvm kernel: [ 2860.928837][  T949]  width 780 height 438 colorspace 0 ycbcr_enc 0 quantization 0 xfer_func 0
Feb 18 22:14:30 kvm kernel: [ 2860.928840][  T949] DEBUG: codec_mpeg12_threaded_isr - reg 4 - SESSION canvas_num 0 vififo_size 1000000 last_irq_jiffies 10009c504 last_offset 28669b wrap_count 0
Feb 18 22:14:30 kvm kernel: [ 2860.928844][  T949] meson-vdec ff620000.video-decoder: Buffer 3 done but it doesn't exist in m2m_ctx
The output buffering (queuing and dequeuing) continues on corrupting my display (or the output stream)

So I think if can find the format setup issue for mpeg capture it might be a step closer to working. Will add some function tracing to the kernel and another recompile and see if I can find where it goes wrong. Been stracing on the ffmpeg and mpv sides and what it is passing to video0 sort of looks ok to me.

crashoverride
Posts: 5658
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 534 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by crashoverride »

brad wrote:
Sat Feb 19, 2022 12:26 pm
As part of the stream startup the vdec sets up formats but it fails to setup the mpeg capture side with an invalid ioctl on device video0 (mostly silently on the VIDIOC_S_SELECTION):
The error is -25 (-ENOTTY) which means the IOCTL is not implemented:
https://www.kernel.org/doc/html/v5.16/u ... s.html#id1
ENOTTY The ioctl is not supported by the driver, actually meaning that the required functionality is not available, or the file descriptor is not for a media device.
Checking the driver source code, the IOCTL is indeed not implemented:
https://elixir.bootlin.com/linux/v5.16. ... dec.c#L787

However, this appears to be the case for ALL codecs, not just MPEG1/2 since vdec.c is common to all.
These users thanked the author crashoverride for the post:
brad (Sat Feb 19, 2022 1:24 pm)

chewitt
Posts: 177
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 132 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by chewitt »

@brad @crashoveride .. comment from JC the Pi Foundation ffmpeg dev I've been working with:

"I'm pretty sure that the VIDIOC_G/S_SELECTION is optional so erroring out shouldn't cause userspace to give up. It is however where cropping would be reported (e.g. 1920x1088 -> 1920x1080) so is at least useful. I don't believe that my ffmpeg stateful driver ever tries to set it, though it does try to get it but doesn't care too much if it fails."

brad
Posts: 1595
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 C4 HC4 N1 N2 N2+ H2 H2+ M1 (64 bit ftw)
Location: Australia
Has thanked: 178 times
Been thanked: 251 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by brad »

chewitt wrote:
Sun Feb 20, 2022 9:17 pm
@brad @crashoveride .. comment from JC the Pi Foundation ffmpeg dev I've been working with:

"I'm pretty sure that the VIDIOC_G/S_SELECTION is optional so erroring out shouldn't cause userspace to give up. It is however where cropping would be reported (e.g. 1920x1088 -> 1920x1080) so is at least useful. I don't believe that my ffmpeg stateful driver ever tries to set it, though it does try to get it but doesn't care too much if it fails."
Thanks yes and I believe that is correct. As crashoveride mentioned the function is not implemented in any of the codecs (also same error with h264 but it works fine).

Going through the memory structures now, checking that pix formats are correct (8bit vs 10 bit) and checking the canvas / planes are defined correctly.

chewitt
Posts: 177
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 132 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by chewitt »

From ndufresne (gstreamer maintainer): Here's the spec, https://www.kernel.org/doc/html/v5.4/me ... coder.html - G_SELECTION is optional and only valid after the initial header was parsed, so after the SRC_CH event, S_SELECTION is entirely optional.That being said, the only way a decoder can avoid G_SELECTION is by allocating the reference frames independently from the CAPTURE buffer (CODA driver does this).
These users thanked the author chewitt for the post:
brad (Mon Feb 21, 2022 12:17 pm)

brad
Posts: 1595
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 C4 HC4 N1 N2 N2+ H2 H2+ M1 (64 bit ftw)
Location: Australia
Has thanked: 178 times
Been thanked: 251 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by brad »

Still working through this and making some progress, but interested in what type of devices might of worked in the past with the driver? (maybe gxbb)

First problem is the canvas setup, We attempt to loop through all 8 destination buffers and assign canvases to them but it is done too early and they don't yet exist so we have nothing to loop through https://elixir.bootlin.com/linux/v5.13. ... ers.c#L192. Can put a hack in to assign them later in the threaded function (check if canvases exist and assign them if they don't) - https://elixir.bootlin.com/linux/v5.13. ... g12.c#L153

Canvases are actually being created (and used now):

Code: Select all

eb 25 01:31:30 kvm kernel: [  226.285658][   T48] DBG esparser_queue
Feb 25 01:31:30 kvm kernel: [  226.287132][    C2] DEBUG: codec_mpeg12_isr - irq wake
Feb 25 01:31:30 kvm kernel: [  226.287159][  T877] DEBUG: codec_mpeg12_threaded_isr
Feb 25 01:31:30 kvm kernel: [  226.287162][  T877] DEBUG - amvdec_set_canvases ------------------------------------------
Feb 25 01:31:30 kvm kernel: [  226.287165][  T877] DEBUG - sess->m2m_ctx - new_frame 1 is_draining 0 next_buf_last 0 has_stopped 0 job_flags 3
Feb 25 01:31:30 kvm kernel: [  226.287169][  T877] DEBUG amvdec_set_canvases - ctx queues cap_q_ctx->num_rdy 8 cap_q_ctx->buffered 0
Feb 25 01:31:30 kvm kernel: [  226.287171][  T877] DEBUG amvdec_set_canvases - ctx queues out_q_ctx->num_rdy 0 out_q_ctx->buffered 0
Feb 25 01:31:30 kvm kernel: [  226.287175][  T877] DEBUG amvdec_set_canvases - reg_cur 2700 reg_num_cur 0 reg_base_cur 0 pixfmt 32314D4E buf->vb.vb2_buf.index 0
Feb 25 01:31:30 kvm kernel: [  226.287184][  T877] DEBUG amvdec_set_canvases - reg_cur 2704 reg_num_cur 1 reg_base_cur 0 pixfmt 32314D4E buf->vb.vb2_buf.index 1
Feb 25 01:31:30 kvm kernel: [  226.287189][  T877] DEBUG amvdec_set_canvases - reg_cur 2708 reg_num_cur 2 reg_base_cur 0 pixfmt 32314D4E buf->vb.vb2_buf.index 2
Feb 25 01:31:30 kvm kernel: [  226.287193][  T877] DEBUG amvdec_set_canvases - reg_cur 270c reg_num_cur 3 reg_base_cur 0 pixfmt 32314D4E buf->vb.vb2_buf.index 3
Feb 25 01:31:30 kvm kernel: [  226.287198][  T877] DEBUG amvdec_set_canvases - reg_cur 2710 reg_num_cur 4 reg_base_cur 0 pixfmt 32314D4E buf->vb.vb2_buf.index 4
Feb 25 01:31:30 kvm kernel: [  226.287202][  T877] DEBUG amvdec_set_canvases - reg_cur 2714 reg_num_cur 5 reg_base_cur 0 pixfmt 32314D4E buf->vb.vb2_buf.index 5
Feb 25 01:31:30 kvm kernel: [  226.287208][  T877] DEBUG amvdec_set_canvases - reg_cur 2718 reg_num_cur 6 reg_base_cur 0 pixfmt 32314D4E buf->vb.vb2_buf.index 6
Feb 25 01:31:30 kvm kernel: [  226.287213][  T877] DEBUG amvdec_set_canvases - reg_cur 271c reg_num_cur 7 reg_base_cur 0 pixfmt 32314D4E buf->vb.vb2_buf.index 7
Feb 25 01:31:30 kvm kernel: [  226.287218][  T877] DEBUG: codec_mpeg12_update_dar  - seq d0d0c ar c
Feb 25 01:31:30 kvm kernel: [  226.287220][  T877] DEBUG: codec_mpeg12_threaded_isr - reg 1 buffer_index 0  offset 28  pic_info f0f0e  is_progressive 0  field 9
Feb 25 01:31:30 kvm kernel: [  226.287223][  T877] DEBUG: codec_mpeg12_threaded_isr - reg 1 - SESSION
Feb 25 01:31:30 kvm kernel: [  226.287223][  T877]  width 780 height 438 colorspace 0 ycbcr_enc 0 quantization 0 xfer_func 0
Feb 25 01:31:30 kvm kernel: [  226.287227][  T877] DEBUG: codec_mpeg12_threaded_isr - reg 1 - SESSION canvas_num 10 vififo_size 1000000 last_irq_jiffies ffffb805 last_offset cf8ca wrap_count 0
Feb 25 01:31:30 kvm kernel: [  226.287231][  T877] DEBUG codec_mpeg12_threaded_isr (BEFORE) - scratch 0=50504 1=70706 2=90908 3=b0b0a 4=d0d0c 5=f0f0e
Feb 25 01:31:30 kvm kernel: [  226.287234][  T877] DEBUG 6=111110 7=131312 8=0 9=1 a=7800438 b=b7621400 c=0 d=28 e=0 f=0
Feb 25 01:31:30 kvm kernel: [  226.287254][  T877] DEBUG codec_mpeg12_threaded_isr (DONE) - scratch 0=50504 1=70706 2=90908 3=b0b0a 4=d0d0c 5=f0f0e
Feb 25 01:31:30 kvm kernel: [  226.287257][  T877] DEBUG 6=111110 7=131312 8=0 9=1 a=7800438 b=b7621400 c=0 d=28 e=0 f=0
Feb 25 01:31:30 kvm kernel: [  226.288522][   T48] esparser_queue_all_src - cont
The next problem is that something is corrupting my primary canvas (not the decoder canvas here but my main window), this will even occur if I try to decode something to a file (ie not trying to display the output) so having some fun trying to work out where that is coming from at this stage. Suspect we are using wrong area of memory or wrong offset where we are decoding so working through that now. (and trying to make sure formats are set ok).

chewitt
Posts: 177
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 132 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by chewitt »

@brad Maxime was mostly working from a LePotato board (GXL) but MPEG1/2 is an older codec so the IP block in the SoC probably hasn't evolved much (or at all) over different hardware generations. BTW, what board are you using for test/dev?

brad
Posts: 1595
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 C4 HC4 N1 N2 N2+ H2 H2+ M1 (64 bit ftw)
Location: Australia
Has thanked: 178 times
Been thanked: 251 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by brad »

chewitt wrote:
Fri Feb 25, 2022 6:56 pm
@brad Maxime was mostly working from a LePotato board (GXL) but MPEG1/2 is an older codec so the IP block in the SoC probably hasn't evolved much (or at all) over different hardware generations. BTW, what board are you using for test/dev?
Have been testing on Odroid N2 (gb12) but have C4 (sm1) & C2 (gxbb) to do testing on as well.

Was looking at newer amlogic vendor code (particularly the slightly different "multi" style mpeg driver) to see if amlogic made any changes for newer soc revisions - https://github.com/hardkernel/linux/tre ... der/mpeg12

It does look like it works the same for MESON8 on wards but I might just spend some time getting a C2 running and see how current mainline code works there.

chewitt
Posts: 177
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 132 times
Contact:

Re: Which OS/Kernel for HW acceleration?

Post by chewitt »

I spoke to Neil and he confirms the MPEG1/2 codec is the same through all generations. No fancy stuff.

Post Reply

Return to “General Topics”

Who is online

Users browsing this forum: No registered users and 1 guest