Running Cassandra on H2/H2+

Post Reply
domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 113 times
Been thanked: 121 times
Contact:

Running Cassandra on H2/H2+

Post by domih »

Cassandra is a NoSQL distributed database running on x86/64-bit commodity hardware.

Little known companies like ebay, Bank of America, McDonald, Microsoft, Saab, etc use Cassandra. The Internet big gorillas all use it to store petabytes and petabytes of data on thousands and thousands of nodes spread over multiple data centers. When you get a stupid marketing recommendation on a web page, it is highly probable that the "AI" that came up with it used data stored on Cassandra nodes.

Advantages of Cassandra: writes very fast, reads very fast, mature ecosystem with data processing tools such as Spark, high-availability, fully distributed: performs the distribution/replication for you. You don't have to deal with implementing data sharding, unless you're really pushing the beast in the multi-billions+ rows stratosphere and need to manually shard the database. In "normal" cases you will let Cassandra deal with the plumbing. Example: if you set up a 3-node Cassandra server with replication to 3, you can access any of the 3 nodes, insert, update and delete rows and the changes will automatically spread to the 2 other nodes. Because replication is 3, you can have 1 or 2 nodes dying or shutdown for maintenance and you still have all your data. Once you restart the nodes, Cassandra will synchronize them for you.

It's NoSQL so kiss goodbye to everything you learned with SQL RDBMS. No joins, no transactions, no ACID and so on. If you want to implement an app that needs transactions (e.g. invoicing system) you will want to stick to MySQL, PostgreSQL, Oracle. If you want to deal swiftly with humongous amount of data poked at by hundreds or thousands threads (the so-called "Web Scale" for instance) Cassandra will fly for you with high colors.

Free community version: https://cassandra.apache.org/. Enterprise/Commercial version: https://www.datastax.com/customers.

R&D'ing Cassandra on Odroid H2

First does it make sense to run Cassandra on an Odroid H2 from a performance viewpoint? Yes it does, with 4 cores and up to 32GB of memory @ 2400MT/s, the H2/H2+ is fully capable in terms of resources to run Cassandra. As a matter of fact, you can run Cassandra on these cheap AWS minimal instances and the Odroid H2/H2+ will provide more performance in comparison. I also know for a fact that despite the recommended minimal 8GB of memory Cassandra will run without crashing or slowing to a crawl on the 4GB memory only Odroid N2 or C4.

Let's run some basic stress testing to get a sense of the landscape. Cassandra provides the cassandra-stress tool.

Example:
...$ cassandra-stress write -mode native cql3 user=cassandra -graph file=~/Documents/cassandra-stress-write-odroid-h2-hdd.html title=Odroid-H2-HDD

This will create 1 million rows, progressively increasing the number of threads doing it and measuring ops and rows per second, latencies and garbage collector behavior. Because we use the "-graph" option, the stress test will generate an HTML report. Once you have performed such a write test, you can run the read test which reads these 1 millions in a similar manner (threads and instrumentation).

Here is a sample of the READ test:

cassandra-stress-read-h2-hdd-read.png
cassandra-stress-read-h2-hdd-read.png (148.95 KiB) Viewed 605 times

For the sake of it, I performed stress tests on three SBCs all using a flavor of the Celeron J4105: the Odroid H2/H2+, the Seed Odyssey and a Gigabyte mini-pc. Here are the consolidated results and charts:

cassandra-stress-test-celeron-J4105.png
cassandra-stress-test-celeron-J4105.png (104.86 KiB) Viewed 605 times
cassandra-stress-write-celeron-j4105.png
cassandra-stress-write-celeron-j4105.png (42.97 KiB) Viewed 605 times
cassandra-stress-read-celeron-j4105.png
cassandra-stress-read-celeron-j4105.png (44.65 KiB) Viewed 605 times

CONCLUSION: Taking into the account the margin of error, the 3 systems behave about the same with maybe the H2/H2+ and Gigabyte mini-pc slightly ahead (they both run 32GB at 2400MT/sec while the Odyssey operates "only" 8GB at 2133MT/sec).

[TO BE CONTINUED]
Last edited by domih on Sun Jul 05, 2020 8:16 am, edited 8 times in total.
These users thanked the author domih for the post (total 3):
mad_ady (Sun Jul 05, 2020 2:27 pm) • odroid (Mon Jul 06, 2020 9:28 am) • powerful owl (Sat Jul 18, 2020 3:28 pm)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 113 times
Been thanked: 121 times
Contact:

Re: Running Cassandra on H2/H2+

Post by domih »

[CONTINUED]

To put things into context, here are the same tests and results on a set of low, mid and high range PCs:

cassandra-stress-test-high-mid-range-cpu.png
cassandra-stress-test-high-mid-range-cpu.png (92.96 KiB) Viewed 604 times
cassandra-stress-write-high-mid-range-cpu.png
cassandra-stress-write-high-mid-range-cpu.png (55.97 KiB) Viewed 604 times
cassandra-stress-read-high-mid-range-cpu.png
cassandra-stress-read-high-mid-range-cpu.png (57.36 KiB) Viewed 604 times

CONCLUSION: As you may expect, a Threadripper 24C/48T rules supreme against common CPUs and we reach performance levels far above the Celeron J4105 level. The amount of $$$$ is far above too...

[TO BE CONTINUED]
Last edited by domih on Sun Jul 05, 2020 8:18 am, edited 1 time in total.

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 113 times
Been thanked: 121 times
Contact:

Re: Running Cassandra on H2/H2+

Post by domih »

[CONTINUED]

The tests above were performed with the Cassandra data on NVMe. How does it compared with the data on a rotating HDD?

I ran the same test on the Odroid H2 with a SATA HGST He10. Here are the comparative results:

cassandra-stress-read-h2-nvme-vs-hdd.png
cassandra-stress-read-h2-nvme-vs-hdd.png (36.98 KiB) Viewed 603 times
CONCLUSION: It's about the same (good news!) within margin of error. It could be because the I/O is fast enough (the HGST He10 is a fast disk) to keep pace with the compute power of the Celeron J4105. Null doubt that Cassandra is also caching and buffering.

However, do we see the same thing with much more CPU resources for the compute part? How does RAID0 2xNVMe compare with one rotating HDD on a high range PC? Here are the results with the TR 3960x:

cassandra-stress-read-tr3960x-nvme-vs-hdd.png
cassandra-stress-read-tr3960x-nvme-vs-hdd.png (39.23 KiB) Viewed 603 times
CONCLUSION: The HDD test can be as 15% slower. So the disk starts to be the bottleneck when computation power is intense.

It will be interesting to see how these numbers evolve with a multi-node server, using 3 x H2, where networking latency and speed will be part of the equation.

powerful owl
Posts: 172
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 2 x HC1, 3 x H2, 1 x H2+
Has thanked: 43 times
Been thanked: 18 times
Contact:

Re: Running Cassandra on H2/H2+

Post by powerful owl »

Fascinating. Thanks for the great write-up/intro!

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 113 times
Been thanked: 121 times
Contact:

Re: Running Cassandra on H2/H2+

Post by domih »

So what about the Odroid N2 and Odroid C4?

Here are the results:

cassandra-stree-tool-n2-c4.png
cassandra-stree-tool-n2-c4.png (118.37 KiB) Viewed 428 times

For context, here are the same results with the Odroid H2/H2+:

cassandra-stress-tool-h2-n2-c4.png
cassandra-stress-tool-h2-n2-c4.png (137.35 KiB) Viewed 428 times
Notes
1. To run correctly on relatively small ARM boards especially with "only" 4GB of RAM, it is a good idea to bump up the Cassandra server timeout settings. To do so, change the values in the cassandra.yaml file:

Code: Select all

# How long the coordinator should wait for read operations to complete
# Default: 5000
read_request_timeout_in_ms: 60000
# How long the coordinator should wait for seq or index scans to complete
# Default: 10000
range_request_timeout_in_ms: 60000
# How long the coordinator should wait for writes to complete
# Default: 2000
write_request_timeout_in_ms: 60000
# How long the coordinator should wait for counter writes to complete
# Default: 5000
counter_write_request_timeout_in_ms: 60000
# How long a coordinator should continue to retry a CAS operation
# that contends with other proposals for the same row
cas_contention_timeout_in_ms: 60000
# How long the coordinator should wait for truncates to complete
# (This can be much longer, because unless auto_snapshot is disabled
# we need to flush first so we can snapshot before removing the data.)
truncate_request_timeout_in_ms: 60000
# The default timeout for other, miscellaneous operations
# Default: 10000
request_timeout_in_ms: 60000
For simplicity, I bumped them to 60 sec.

2. USB Attached SCSI (UAS) is not active in the C4 Ubuntu Mate 20.04 kernel. So as a prerequisite, rebuild the kernel to activate it.

CONCLUSION
1. Both the N2 and C4 seem to be capable to be used in a Cassandra cluster. Given the limited resources and memory one can wonder it they can hold 10M, 100M, 500M... rows. Further testing will tell.
2. The N2 performance is much closer to the H2/H2+ than to the C4. Kudos to the N2.
3. I will not perform further tests with the C4. IMHO, a cluster of N2 will do much better.
4. I am really curious what the overclocked N2+ will show!
These users thanked the author domih for the post:
odroid (Mon Jul 20, 2020 12:14 pm)

Post Reply

Return to “Projects”

Who is online

Users browsing this forum: No registered users and 1 guest