Question on C4 SPI frequency / performance

[mattlovell]

I'm hoping someone has some debug ideas regarding how to proceed from here...

I have two ILI9341-based, SPI-connected LCD displays. I also have both an Odroid C4 and a Raspberry Pi 3 (Model B v1.2). The setup for both is as follows:

Linux Kernel:

RPi: Linux raspberrypi 5.4.51-v7+ #1333 SMP Mon Aug 10 16:45:19 BST 2020 armv7l GNU/Linux
C4: Linux C4 4.9.113 #1 SMP PREEMPT Fri Aug 14 20:30:41 UTC 2020 aarch64 GNU/Linux (CoreELEC)

Python software:

luma.core: 1.17.1 on both
luma.lcd: 2.5.0 on both
luma.examples: commit ceb01960 on both (from Wed Oct 21 16:53:13)
spidev: 3.4 on both
Pillow: 7.2.0 on both
RPi.GPIO: 0.7.0 on RPi, 0.6.3post1 (the RPi.GPIO-Odroid version) on C4

Issue:

Using demonstration programs from luma.example, one can invoke something like:

Code: Select all

python bounce.py \
  --interface spi --display ili9341 \
  --width 320 --height 240 \
  --backlight-active high \
  --gpio-reset-hold-time 0.2 --gpio-reset-release-time 0.2 \
  --spi-bus-speed 48000000

on both SBCs. (Using a frequency like 48 MHz requires a local modification to luma.core's serial.py file. At present, it constrains frequencies to be a power-of-2, rather than just an even number. Based on info I found on home site for RPi, it seems like SPI frequency just has to be even.)

Attached are two photos showing the results. The photo with the RPi is getting ~19 fps with a spi-bus-speed of 50 MHz.

The C4, however, doesn't seem to show any speed improvements past 8 MHz. The photo is taken when attempt 48 MHz on the command line.

The command-line option is definitely getting parsed. Specifying frequencies less-than 8 MHz definitely results in it running slower!

The Python spidev module does appear to perform an error check after invoking SPI_IOC_WR_MAX_SPEED_HZ, per these lines of code:

https://github.com/doceme/py-spidev/blo ... 1273-L1277

No error is thus occurring when trying >8 MHz, but no speed improvement seems to happen either.

So, does anyone have suggestions regarding how to debug this further? The C4 is supposed to support very high SPI bus frequencies, but I can't convince myself that I'm setting high speeds successfully.

Cheers,
Matt

[mattlovell]

BTW, trying the loopback spidev test (with a jumper cable installed) from the C4 wiki:

https://wiki.odroid.com/odroid-c4/appli ... e/gpio/spi

I cannot get it to fail, even if I specify presumably ludicrous speeds (like 500 MHz, or 50 GHz). It's always returning the 01 02 03 04 data.

My suspicion is that the SPI_IOC_WR_MAX_SPEED_HZ call is capable of silently failing.

[mattlovell]

Hmm... per an earlier post, I was able to get an OLED not to work when I exceeded its maximum SPI clock frequency (of 10 MHz). That suggests that the C4's SPI bus is doing transfers faster than that.

I wonder if performance is ultimately being limited by spi-meson-spicc. There's this observation that's been present for a long time in the file:

https://github.com/torvalds/linux/blob/ ... .c#L26-L36

Code: Select all

/*
 * The Meson SPICC controller could support DMA based transfers, but is not
 * implemented by the vendor code, and while having the registers documentation
 * it has never worked on the GXL Hardware.
 * The PIO mode is the only mode implemented, and due to badly designed HW :
 * - all transfers are cutted in 16 words burst because the FIFO hangs on
 *   TX underflow, and there is no TX "Half-Empty" interrupt, so we go by
 *   FIFO max size chunk only
 * - CS management is dumb, and goes UP between every burst, so is really a
 *   "Data Valid" signal than a Chip Select, GPIO link should be used instead
 *   to have a CS go down over the full transfer
 */

Implementing transfers with PIO (particularly if any reads are involved) is always going to be somewhat painful, even with a fast core, compared to letting the device or a separate controller handle the work via DMA.

[odroid]

We could get near 100Mhz with the loop back test.
We will capture an oscilloscope screen to make sure.

[odroid]

We tested the SPI clock frequency with the following conditions.

Linux Image : ubuntu-20.04-4.9-mate-odroid-c4-hc4-20201020.img
Kernel Version : Linux odroid 4.9.236-43 #1 SMP PREEMPT Thu Oct 15 04:08:17 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
WIKI : https://wiki.odroid.com/common/applicat ... pi?s[]=spi

Test Code Download & Build :

Code: Select all

sudo wget http://dn.odroid.com/Accessory/examples/spidev_test.c
sudo gcc -o spidev_test spidev_test.c

Test commands for 1Mhz, 10Mhz and 100Mhz. One packet size was 4 bytes.

Code: Select all

./spidev_test -D /dev/spidev0.0 -s 1000000 
./spidev_test -D /dev/spidev0.0 -s 10000000
./spidev_test -D /dev/spidev0.0 -s 100000000

Waveform on the oscilloscope looks fine.

scope_1mhz.png (46.55 KiB) Viewed 787 times

scope_10mhz.png (42.21 KiB) Viewed 787 times

scope_100mhz.png (36.77 KiB) Viewed 787 times

[joshua.yang]

Hi.

Thanks for the detailed report. As @odroid posted, we can confirm that the SPI feature on our latest Ubuntu image is working well.

If you don't mind, it might be worth doing the same test using our Ubuntu image.
Here's the latest minimal image for C4 (2020/10/15 version): https://odroid.in/ubuntu_20.04lts/c4/ub ... 015.img.xz

[mattlovell]

@odroid, thanks for the detailed response. The SPI clk is clearly capable of >8 MHz operation. The lower performance of luma.example's bounce.py must have some other explanation (e.g., small upper bound on SPI transfer size, PIO communication model, etc).

Out of curiosity, what should the spidev.c executable do if the command-line option exceeds the SPI bus' max_frequency? The error check makes it seem like it should balk and just exit. Could it instead cap the frequency (to whatever HW is specifying as the maximum) and run anyway?

@joshua.yang, yes I can try the Ubuntu image later this week.

Thanks again,
Matt

[mattlovell]

Out of curiosity, what should the spidev.c executable do if the command-line option exceeds the SPI bus' max_frequency? The error check makes it seem like it should balk and just exit. Could it instead cap the frequency (to whatever HW is specifying as the maximum) and run anyway?

According to the SPI device driver source code, the maximum transfer rate seems to be 166Mhz.
https://github.com/hardkernel/linux/blo ... cc.c#L1211

When you load the spi device driver, enable the force64b option. And send the 64bit(8 bytes) data as a packet. It will increase the LCD rendering performance significantly.
I am not sure whether your OS distribution has this kernel patch or not.
https://github.com/hardkernel/linux/blo ... #L491-L494

[mattlovell]

I am not sure whether your OS distribution has this kernel patch or not.

I don't see that change in CoreELEC's current meson_spicc_setup_pio_burst().

I don't know how far "behind" Hardkernel's code CE is or even how close they try to track it. In spite of that, I opened a new topic in the Development section of CE's forum for the above.

Thanks!

[mattlovell]

If you don't mind, it might be worth doing the same test using our Ubuntu image.
Here's the latest minimal image for C4 (2020/10/15 version): https://odroid.in/ubuntu_20.04lts/c4/ub ... 015.img.xz

@joshua.yang, I downloaded the MATE version here: https://odroid.in/ubuntu_20.04lts/c4-hc4/

After swapping in RPi.GPIO-Odroid for the native RPi version, I have luma.examples up and running. I also edited luma.core's serial.py file to permit for a wider range of SPI frequencies (a change rm-hull has committed for luma.core).

Trying the bounce.py demo at 50 MHz, it's still only getting ~8 FPS.

In a subsequent post, @odroid writes:

When you load the spi device driver, enable the force64b option.

Hmm... how do I go about that? Looks like Ubuntu boots with spidev and spi_meson_spicc loaded. I'll go read up on how to provide options to kernel modules (or, even better, if one can unload and re-modprobe a driver and add options).

[mattlovell]

Hi @odroid,

When you load the spi device driver, enable the force64b option. And send the 64bit(8 bytes) data as a packet. It will increase the LCD rendering performance significantly.

Trying Hardkernel's Ubuntu_20.04 (4.9.236-41, compiled Thu Oct 15 04:08:17 UTC 2020), I edited /etc/modules to include the following line:

Code: Select all

spi_meson_spicc force64b=1

Is that sufficient or does the option need to be passed in via some other mechanism?

I'm not sure if I'm supposed to make any other changes for the second part of your suggestion -- "send the 64bit (8 bytes) data as a packet" -- or if that was just a description of what the force64b option accomplishes.

Trying luma.examples bounce.py demo after the above change to /etc/modules and a reboot, I'm seeing 8.13 FPS -- no real change from before. (I'll keep poking around to see if the option really is getting to spi_meson_spicc.)

If you are interested and have a SPI display available with which to test, it is pretty trivial to get luma.lcd and luma.examples going under Ubuntu. After installing python3-pip and git, and getting a clone of RPi.GPIO-Odroid available, the steps amount to

Code: Select all

sudo pip3 install luma.lcd
sudo pip3 uninstall RPi.GPIO
<- follow RPi.GPIO-Odroid install steps ->

The invocation for bounce.py that I'm then using is

Code: Select all

sudo python3 bounce.py \
  --interface spi --display ili9341 \
  --width 320 --height 240 \
  --backlight-active high \
  --gpio-reset-hold-time 0.2 \
  --gpio-reset-release-time 0.2 \
  --spi-bus-speed 50000000

Thanks for your help thus far,
Matt

[joshua.yang]

If you are interested and have a SPI display available with which to test, it is pretty trivial to get luma.lcd and luma.examples going under Ubuntu. After installing python3-pip and git, and getting a clone of RPi.GPIO-Odroid available, the steps amount to

Code: Select all

sudo pip3 install luma.lcd
sudo pip3 uninstall RPi.GPIO
<- follow RPi.GPIO-Odroid install steps ->

The invocation for bounce.py that I'm then using is

Code: Select all

sudo python3 bounce.py \
  --interface spi --display ili9341 \
  --width 320 --height 240 \
  --backlight-active high \
  --gpio-reset-hold-time 0.2 \
  --gpio-reset-release-time 0.2 \
  --spi-bus-speed 50000000

Thanks for your help thus far,
Matt

Hi,

We will reproduce the issue on our side by referring to your instruction. Please wait for a few days.

[joshua.yang]

HI @mattlovell,

Any other additional modifications to Python Luma packages except the SPI bus speed? I have been trying this but I cannot see any outputs from the display, which uses the ili9341 controller.
It keeps showing only the white screen after I run that "bounce.py" file with your parameters.

Which LCD model you are using?

Code: Select all

root@odroid:~/luma.examples/examples# sudo python3 bounce.py \
>   --interface spi --display ili9341 \
>   --width 320 --height 240 \
>   --backlight-active high \
>   --gpio-reset-hold-time 0.2 \
>   --gpio-reset-release-time 0.2 \
>   --spi-bus-speed 32000000
Version: luma.lcd 2.6.0 (luma.core 1.17.3)
Display: ili9341
Interface: spi
Dimensions: 320 x 240
------------------------------------------------------------

[joshua.yang]

Don't mind, I have found a way to make it works. In our case, the reset pin and the dc pin have to be specified.
Actually, ours have the swapped pin number each other rather than the regular ili9341 one.

Code: Select all

sudo python3 bounce.py \
  --interface spi --display ili9341 \
  --width 320 --height 240 \
  --backlight-active high \
  --gpio-reset-hold-time 0.2 \
  --gpio-reset-release-time 0.2 \
  --gpio-reset 24 \
  --gpio-data-command 25 \
  --spi-bus-speed 32000000

And, we also can see that the poor framerate. Will fix that and let you know the results.

[mattlovell]

Don't mind, I have found a way to make it works. In our case, the reset pin and the dc pin have to be specified.
Actually, ours have the swapped pin number each other rather than the regular ili9341 one.

...

And, we also can see that the poor framerate. Will fix that and let you know the results.

Yes, the connectivity for the various "extra" pins seems somewhat arbitrary.

Glad you were able to get something running and reproduce the issue!

Cheers,
Matt

[joshua.yang]

Hi.

We have found the root cause and have made a patch for fixing that.

Finally, we can see 31~32 fps for the bounce.py example with 72MHz SPI bus speed. Please see the below video.
https://dn.odroid.com/temp/201030_c4_sp ... xample.mp4

201030_c4_spi_72mhz_bouce_example.jpeg (143.43 KiB) Viewed 607 times

The kernel including this patch will be released in the middle of next week. This might depend on our kernel update conditions.

[mad_ady]

The screen sheild looks nice. Is it something you've built in-house?

[mattlovell]

We have found the root cause and have made a patch for fixing that.

Finally, we can see 31~32 fps for the bounce.py example with 72MHz SPI bus speed. Please see the below video.

The kernel including this patch will be released in the middle of next week. This might depend on our kernel update conditions.

That's great news!! Thank you for investigating it and coming to root cause.

It would be interesting to know the details, or at least a link to the changeset, for communicating with CoreELEC developers.

Cheers,
Matt

[mattlovell]

Hello @mattlovell

Here's the patch that fixes it. https://github.com/hardkernel/linux/com ... af623559b7

Also present on today kernel update for our Ubuntu images

[mattlovell]

@joshua.yang, the CoreELEC developers examined the change and had a question:

https://discourse.coreelec.org/t/hardwa ... ce/13338/6

... What if the SPI device doesn’t support 64 bit words? There’s no way to turn this off.

It does seem a fair question. The performance improvement still seems worth pursuing, if compatibility can be addressed.

Thanks,
Matt

[joshua.yang]

@joshua.yang, the CoreELEC developers examined the change and had a question:

https://discourse.coreelec.org/t/hardwa ... ce/13338/6

... What if the SPI device doesn’t support 64 bit words? There’s no way to turn this off.

It does seem a fair question. The performance improvement still seems worth pursuing, if compatibility can be addressed.

Thanks,
Matt

Hi,

I have tested simply with an SPI LED strip (WS2801) on Ubuntu, and it seems working normally.
We believe that the 64 bit work doesn't affect the other conditions. That will be enabled only when the SPI buffer size is large enough. If it isn't, that 64 bit options won't do anything and the data streams as it is.
You can see the conditions for that.
https://github.com/hardkernel/linux/com ... 13f003R591

So for now, I think it will be fine to be released. But if any issues occurred later, then I will revert that and we should find out the other way to increase the SPI performance for FBTFT LCDs.