DRM what else is left to do?

Post Reply
wallyz21
Posts: 293
Joined: Thu Apr 04, 2019 11:00 am
languages_spoken: english
ODROIDs: N2, N2+
Has thanked: 15 times
Been thanked: 32 times
Contact:

DRM what else is left to do?

Post by wallyz21 »

I have noticed there is DRM support in the 5.13 kernel.

I am running Ubuntu 21.04 with kernel 5.13.

So how do I configure Firefox or Chrome to use the DRM?
Walter Zambotti
N2 - HK 18.04 Ubuntu Mate Desktop
N2+ - 21.04 Ubuntu Mate Desktop (Panfrost)

crashoverride
Posts: 5412
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 466 times
Contact:

Re: DRM what else is left to do?

Post by crashoverride »

In the Linux kernel, DRM is short for Direct Rendering Manager. It defines an interface between software in user space and video related drivers in kernel space.

Firefox and Chrome require more than DRM to operate. They require a desktop environment (DE) that provides additional services such as input management, font services, sound management, clipboard services, etc.

The implication is that a DE is configured to use DRM. In X11, this is through use of the ModeSetting or Armsoc driver. In Wayland, this is through an implementation such as Weston. Once a DE is established that used DRM, Firefox and Chrome transparently use it through the DE's interfaces.
These users thanked the author crashoverride for the post (total 2):
mctom (Thu Aug 19, 2021 1:44 am) • odroid (Thu Aug 19, 2021 9:25 am)

wallyz21
Posts: 293
Joined: Thu Apr 04, 2019 11:00 am
languages_spoken: english
ODROIDs: N2, N2+
Has thanked: 15 times
Been thanked: 32 times
Contact:

Re: DRM what else is left to do?

Post by wallyz21 »

crashoverride wrote:
Wed Aug 18, 2021 7:12 pm
In the Linux kernel, DRM is short for Direct Rendering Manager. It defines an interface between software in user space and video related drivers in kernel space.

Firefox and Chrome require more than DRM to operate. They require a desktop environment (DE) that provides additional services such as input management, font services, sound management, clipboard services, etc.

The implication is that a DE is configured to use DRM. In X11, this is through use of the ModeSetting or Armsoc driver. In Wayland, this is through an implementation such as Weston. Once a DE is established that used DRM, Firefox and Chrome transparently use it through the DE's interfaces.
Given the kernel now supports DRM and that versions of Ubuntu DE's exist that are already configured to use DRM when is it anticipated that the DE'S (we are using) will be configured? And when you say "configured" you mean recompiled!?!?
Walter Zambotti
N2 - HK 18.04 Ubuntu Mate Desktop
N2+ - 21.04 Ubuntu Mate Desktop (Panfrost)

crashoverride
Posts: 5412
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 466 times
Contact:

Re: DRM what else is left to do?

Post by crashoverride »

wallyz21 wrote:
Thu Aug 19, 2021 11:14 am
when is it anticipated that the DE'S (we are using) will be configured?
If you are using one of the images with a 5.x kernel, DRM is already being used.
wallyz21 wrote:
Thu Aug 19, 2021 11:14 am
And when you say "configured" you mean recompiled!?!?
In X11, its is automatically probed for using the "ModeSetting" driver or manually specified in the xorg.conf file for "Armsoc" driver (as seen with XU4). Weston only uses DRM to my knowledge (as opposed to FBDEV).

---

DRM is primarily concerned with displays. It provides the framework necessary to connect a framebuffer to a display controller and the display controller to an encoder (HDMI, DisplayPort, CVBS, etc). An additional subsystem (TTM/GEM/DMABUF) is used to define and manage an area of memory for the framebuffer to use. GPUs and other devices consume these areas of managed memory to directly render their output. This eliminates the need to copy the framebuffer contents across devices.

Firefox and Chrome do not manage the display. They use the DE's provided interfaces, not DRM, to render content in a controlled and well defined manner. This allows the display to be shared with other applications. With DRM, only a single application (such as the DE) can use the display.

Typically when this type of question is asked, what is actually being requested is "how to I enable hardware acceleration of _____".

For hardware acceleration of web page rendering, the answer is to use OpenGLES. Both Firefox and Chrome use Skia to render. Skia supports OpenGLES.

For hardware acceleration of video decoding, the answer is far more complicated as many methods exist: GStreamer, VA-API, VDPAU, V4L2, and OpenMAX. Firefox can use VA-API, however, there are no VA-API implementations for hardware codecs typically found in ARM based devices.
These users thanked the author crashoverride for the post:
wallyz21 (Mon Aug 23, 2021 11:02 am)

chewitt
Posts: 126
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 112 times
Contact:

Re: DRM what else is left to do?

Post by chewitt »

There were some early VAAPI experiments with RPi4 HEVC and some of the Hantro based vdecs (all ARM devices) but those efforts fizzled out and ended up being displaced by the stateless V4L2 UAPI, but that's a long story and of no relevance to anything with an Amlogic (stateful) chip.

The current Amlogic vdec and key userspace apps (ffmpeg/gstreamer) needs rework to be usable. Development was started in 2018 and reached a fairly usable state, but then the main developer had to step down for personal reasons, and then the stateful V4L2 UAPI evolved which caused regressions that nobody has worked on. The vdec hasn't really been touched in anger since late 2019 (minor fixups were upstreamed in early 2020, but nothing since). The stateful UAPI is now advancing via work Raspberry Pi Foundation is doing (RPi H264 is stateful too) but there's no effort on the Amlogic vdec to match the progress seen there.

The current Amlogic vdec attempts to bridge from GXBB throught to SM1 in the same driver but "in hindsight" this is not the correct approach so the driver needs to be split up and reworked to have different memory management approaches for GXBB > GXM and G12A > SM1 devices. Sadly there is a lack of developer talent in this area. People with the skills to archtect a modern driver from scratch are professional developers who expect to be paid for their efforts and there are no commercial interests funding development in this area at the moment. There are larger numbers of community developers, but they lack the level of design/coding skill needed and most prefer to tinker around the edges of the BSP codebase which is ugly as hell code "but works" so requires less effort to get a result. It's sad, because the overall useability of the mainline codebase is now very good, but the media features (of a media oriented SoC) are now the weakest part. Amlogic itself it busy pursuing other directions like RDK, so aren't going to contribute much.

My general advice is for people to disable hardware decoding. The newer generation SoCs (G12A, G12B, SM1) are capable of software decoding everything up to 1080p resolution which is good-enough for most desktop use.
These users thanked the author chewitt for the post:
hominoid (Sat Aug 21, 2021 9:00 am)

wallyz21
Posts: 293
Joined: Thu Apr 04, 2019 11:00 am
languages_spoken: english
ODROIDs: N2, N2+
Has thanked: 15 times
Been thanked: 32 times
Contact:

Re: DRM what else is left to do?

Post by wallyz21 »

The Ubuntu package search shows there the necessary packages VA-API for for arm64!!!

Package mesa-va-drivers

groovy (20.10) (libs): Mesa VA-API video acceleration drivers [universe]
20.2.1-1: amd64 arm64 armhf i386 ppc64el s390x

Package libva-drm2

groovy (20.10) (libs): Video Acceleration (VA) API for Linux -- DRM runtime [universe]
2.8.0-1: amd64 arm64 armhf i386 ppc64el s390x

Package libva-glx2

groovy (20.10) (libs): Video Acceleration (VA) API for Linux -- GLX runtime [universe]
2.8.0-1: amd64 arm64 armhf i386 ppc64el s390x

Package libva-wayland2

groovy (20.10) (libs): Video Acceleration (VA) API for Linux -- Wayland runtime [universe]
2.8.0-1: amd64 arm64 armhf i386 ppc64el s390x

Package libva-x11-2

groovy (20.10) (libs): Video Acceleration (VA) API for Linux -- X11 runtime [universe]
2.8.0-1: amd64 arm64 armhf i386 ppc64el s390x

Packages also available for Hirsute.

So what's missing or incompatible?
Walter Zambotti
N2 - HK 18.04 Ubuntu Mate Desktop
N2+ - 21.04 Ubuntu Mate Desktop (Panfrost)

hominoid
Posts: 568
Joined: Tue Feb 28, 2017 3:55 am
languages_spoken: english
ODROIDs: C2, C4, XU4, MC1, N1, N2, N2+, HC4
Location: Lake Superior Basin, USA
Has thanked: 64 times
Been thanked: 213 times
Contact:

Re: DRM what else is left to do?

Post by hominoid »

Just because there is a package available doesn't mean it is complete, works or doesn't have issues. I believe @chewitt's post summarized the issues.

crashoverride
Posts: 5412
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 466 times
Contact:

Re: DRM what else is left to do?

Post by crashoverride »

wallyz21 wrote:
Fri Aug 20, 2021 9:38 am
So what's missing or incompatible?
The "mesa-va-drivers" package only contains drivers for NVIDIA and AMD. Drivers for Amlogic are not provided.
https://packages.ubuntu.com/groovy/arm6 ... s/filelist

chewitt
Posts: 126
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 112 times
Contact:

Re: DRM what else is left to do?

Post by chewitt »

https://github.com/bootlin/libva-v4l2-request/pull/38 <= looks like libva-v4l2-request is being revived, it was unmaintained for some time, but (again) this is of no use to Amlogic SoC devices which use the stateful v4l2m2m UAPI. In the ideal world Amlogic would release the sources for their firmware blobs so that a stateless approach could be taken for hardware decoding (would be more attractive to community developers than closed-source blobs) but I've asked Amlogic managemeent and they have no plans or desire to do that, and the alternative is reverse engineering, but we've not been able to identify the ISA used in the firmware to further that idea.

AreaScout
Posts: 1719
Joined: Sun Jul 07, 2013 3:05 am
languages_spoken: german, english
ODROIDs: X2, U3, XU3, C2, HiFi Shield, XU4, XU4Q,
N1, Go, VU5A, Show2, CloudShell2,
H2, N2, VU7A, VuShell, Go2, C4
Has thanked: 103 times
Been thanked: 318 times
Contact:

Re: DRM what else is left to do?

Post by AreaScout »

chewitt wrote:
Sat Aug 21, 2021 8:43 pm
https://github.com/bootlin/libva-v4l2-request/pull/38 <= looks like libva-v4l2-request is being revived
That's good for the N1, so sad this board fail because of RAM parts :(

wallyz21
Posts: 293
Joined: Thu Apr 04, 2019 11:00 am
languages_spoken: english
ODROIDs: N2, N2+
Has thanked: 15 times
Been thanked: 32 times
Contact:

Re: DRM what else is left to do?

Post by wallyz21 »

Ok. To sum up. The packages are there but they are not complete and don't have the required support for the AMLOGIC chipset.

AMLOGIC haven't provided the source for the drivers!

How do they get around this problem in the Android world?
Walter Zambotti
N2 - HK 18.04 Ubuntu Mate Desktop
N2+ - 21.04 Ubuntu Mate Desktop (Panfrost)

chewitt
Posts: 126
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 112 times
Contact:

Re: DRM what else is left to do?

Post by chewitt »

Android uses the mediacodec framework to abstract itself from hardware decoders and the Amlogic BSP kernel supports that via proprietary Amcodec drivers (and userspace bits hacked to match). The main issues with Amcodec and BSP kernel DRM things are: .. it's existence predates many modern kernel frameworks (so it doesn't use them), it is over-complex (tries to support all features of hardware rather than the subset which is actually used), it's poorly architected (when your code is never subject to propre peer-review all your bad ideas are great ideas) and it only supports the latest generation devices (Amlogic reinvents the wheel for each new chip at the expense of support for older no-longer-supported chips). Amcodec "works" in the Android world where an Amlogic customer only needs to support a single device (so you hack the codebase for that specific device)/ Google forces an abstraction via mediacodec to reduce the impact on their userspace apps. However this means Amlogic are never really forced to evolve, so Amcodec remains as something that could never be cleaned up and sent upstream to the Linux kernel (too much work). Amlogic do provide sources for Amcodec, but not for the firmware blobs.
These users thanked the author chewitt for the post (total 2):
wallyz21 (Mon Aug 23, 2021 11:01 am) • mctom (Wed Aug 25, 2021 4:24 pm)

wallyz21
Posts: 293
Joined: Thu Apr 04, 2019 11:00 am
languages_spoken: english
ODROIDs: N2, N2+
Has thanked: 15 times
Been thanked: 32 times
Contact:

Re: DRM what else is left to do?

Post by wallyz21 »

chewitt wrote:
Sun Aug 22, 2021 10:31 pm
Android uses the mediacodec framework to abstract itself from hardware decoders and the Amlogic BSP kernel supports that via proprietary Amcodec drivers (and userspace bits hacked to match). The main issues with Amcodec and BSP kernel DRM things are: .. it's existence predates many modern kernel frameworks (so it doesn't use them), it is over-complex (tries to support all features of hardware rather than the subset which is actually used), it's poorly architected (when your code is never subject to propre peer-review all your bad ideas are great ideas) and it only supports the latest generation devices (Amlogic reinvents the wheel for each new chip at the expense of support for older no-longer-supported chips). Amcodec "works" in the Android world where an Amlogic customer only needs to support a single device (so you hack the codebase for that specific device)/ Google forces an abstraction via mediacodec to reduce the impact on their userspace apps. However this means Amlogic are never really forced to evolve, so Amcodec remains as something that could never be cleaned up and sent upstream to the Linux kernel (too much work). Amlogic do provide sources for Amcodec, but not for the firmware blobs.
OK! So what is the connection between the Amcodec drivers and the firmware blobs? Are they dependent on each other? Can they operate independently?
Walter Zambotti
N2 - HK 18.04 Ubuntu Mate Desktop
N2+ - 21.04 Ubuntu Mate Desktop (Panfrost)

chewitt
Posts: 126
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 112 times
Contact:

Re: DRM what else is left to do?

Post by chewitt »

Dependent on each other.

brad
Posts: 1449
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 C4 HC4 N1 N2 N2+ H2 H2+ (64 bit ftw)
Location: Australia
Has thanked: 135 times
Been thanked: 222 times
Contact:

Re: DRM what else is left to do?

Post by brad »

chewitt wrote:
Sat Aug 21, 2021 8:43 pm
https://github.com/bootlin/libva-v4l2-request/pull/38 <= looks like libva-v4l2-request is being revived, it was unmaintained for some time, but (again) this is of no use to Amlogic SoC devices which use the stateful v4l2m2m UAPI. In the ideal world Amlogic would release the sources for their firmware blobs so that a stateless approach could be taken for hardware decoding (would be more attractive to community developers than closed-source blobs) but I've asked Amlogic managemeent and they have no plans or desire to do that, and the alternative is reverse engineering, but we've not been able to identify the ISA used in the firmware to further that idea.
Hi @chewitt
I was asking about amlogic drivers with libva-v4l2 request but was told it would be a complicated task - https://github.com/bootlin/libva-v4l2-r ... -843147661

Do you see libva-v4l2-request working with the amlogic vdec blob and mainline? It would be very nice if this was possible.

Thanks,
Brad.

chewitt
Posts: 126
Joined: Mon Aug 12, 2019 12:27 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 112 times
Contact:

Re: DRM what else is left to do?

Post by chewitt »

I'd expect a separate "libva-v4l2m2m" shim to be required; the stateless and stateful decoding approaches in the kernel have completely different V4L2 plumbing.

https://github.com/bootlin/libva-v4l2-r ... -843147661 <= If you're semi-serious about poking vdec code, ping me for a chat in IRC.
These users thanked the author chewitt for the post:
brad (Thu Aug 26, 2021 12:19 pm)

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 0 guests