White wood mini-tower of 3 x H2

[domih]

Little new project: a white wood mini-tower of 3 x H2 with 5-disc cage with which you do not need to disassemble the whole thing to replace a drive. I'll use only 3 hard disks, one for each H2. An ATX SFX PSU will power the 3 x H2 and the disks. It will be used for a replicated NAS and a 3-node Cassandra server. The latter is because any home must have a Cassandra server

See picture below. As the French says "y a plus qu'à"
.

20200504_161821.png (2.9 MiB) Viewed 5455 times

[rooted]

Nice project, why do you need a distributed database?

[mad_ady]

Nice project, why do you need a distributed database?

For the shopping list and cupboard items, what else?

[domih]

Nice project, why do you need a distributed database?

Mostly testing ideas, writing HOWTOs in relation to professional work.

Also I'm really interested in evaluating whether or not the algorithms used to distribute PB of data on thousands of commodity PCs are ALSO useful to distribute storage and data processing on SBCs with limited resources.

[rooted]

And here I was thinking (because of @mad_ady) you were creating a shopping list for the entire country

[domih]

Update
- Corner braces to assemble shelves with side panels,
- Velcro tape to fix PSU, H2 column and HDD cage to shelves: easy removal for maintenance without fooling around with cramped space for (un)screwing, dampened vibrations for HDD cage and resonance,
- 3.8 x 36" wood to build front mesh frame,
- Plastic back front mesh,
- Loctite to assemble front mesh frame and glue the mesh to it.

20200509_162418.png (2.86 MiB) Viewed 5246 times

[domih]

Update

See overall view of the case shown below (build in progress).
.

case_overall_view.png (1.39 MiB) Viewed 5061 times

.
I dropped the usage of corner braces (with the original goal of leaving the sides virgin of any marks), not easy to fix correctly though, result not solid enough. So I went with Philllips pan head 6x1 1/4, three on each side for each shelf. Very solid. I'll mask the screw holes with white putty or plastic round covers.

At the top there is a nice aluminum handle to make the handling/transport of the case easy. The 3 x H2 LED power button will also be on the top. I'll need a 5/8" drill bit, the closest to the 16mm button diameter. Yeah, these buttons are available again. I ordered the 3rd button (plus a few other things) from Ameridroid.com and as usual got it 2 days later with the usual Haribo Goldbears. Kudos to Ameridroid.com! Disclosure: they are not far from the San Francisco Bay Area so I'm a privileged customer in terms of delivery time
.

case_top.png (1.19 MiB) Viewed 5061 times

.
The bottom section of the tower will hold the 5-disk SATA cage (see https://www.ebay.com/sch/i.html?_nkw=Venus+TL+backplane). I'll use only 3 slots for now. 1 x HGST 10TB He10 (HUH721010ALE604) per H2. I bought them used on eBay for about the same price as a new WD Red 10TB. See https://www.ebay.com/sch/i.html?_nkw=HUH721010ALE604.
.

case_bottom.png (1.24 MiB) Viewed 5061 times

.
I'm also working on setting up the 3 x H2 and their powering. See picture shown below.
.

psu_3xh2_power_distrib.png (1.87 MiB) Viewed 5061 times

.
It's basically an expanded version of the setup I used in https://magazine.odroid.com/article/net ... odroid-h2/. Originally, I wanted to go with 2.5 Gbe using USB 3 adapters in order to pass the 1 Gbe limit. But following testing it appears that these USB 3 adapters are having teething issues, see viewtopic.php?f=172&t=38713.

So I revisited the networking solution with the following options: (a) Wait for Hard Kernel in case they would make a new SBC with 1+ Gbe NICs (b) Try 10 Gbe SolarFlare NICs (i.e. the SFN5122F which you can find for $20 or less on eBay, soon they'll give you one free when you buy detergent or toilet paper) with SFP+ DAC cables and a Microtik CRS317-1G-16S+RM switch, overall unbeatable prices compared to 10Base-T RJ-45 solutions - as of this writing (c) Continue with the Mellanox InfiniBand solution. In the Odroid Magazine article I wrote:

.../...
Note: do Mellanox ConnectX-3 cards also work on the ODROID-H2? As of this writing only the Pro version works. The non-pro versions do not. Know anyway that in terms of max speed achievable on the H2, the ConnectX-3 will provide the same result as the ConnectX-2. The ConnectX-2 provide 40 Gb/s in IB mode but only 10 GbE in Ethernet mode. The connectX-3 provide 40 Gb/s in IB mode and 40 GbE in Ethernet mode. However IP over IB provides a speed as good (if not a little bit better) with a port configured for IB as IP on a port configured for Ethernet. So really, no need to run a ConnectX3 Pro on the H2 (unless you already have a 20 or more GbE network running at home based on Ethernet).
.../...

Since then I upgraded my IB switch from a QDR (about $100) to an FDR10 (about $150) and finally an FDR (about $200) all from eBay. As a result from modern PC to modern PC my IPoIB speed jumped from about 32 Gbe, to 38 Gbe and finally 45 Gbe. However, the Connect-XB or Connect X2 (too old) cards I used on the H2 do not work well with the FDR10 or FDR switches. On the other hand, the ConnectX3 Pro on the H2 works 100% OK with these switches, still obviously with the limited speed from 10 Gbe to 14 Gbe depending on direction, which is the best the H2 + Celeron J4105 + PCIe 2.0 x4 can do. So with the ConnectX3 Pro the 3 x H2s can still be part of the IB network with the modern PCs I finally decided to adopt this solution when I found an eBay seller selling used ConnectX3 Pro cards for $45 each. See these cards having their thermal paste re-done in the image shown below.
.

connect_x3_pro.png (1.98 MiB) Viewed 5061 times

.
I also found less expensive M.2 NMVe M key to PCIe x4 adapters on eBay, see https://www.ebay.com/itm/PCIe-x4-3-0-PC ... 4398886092. Hopefully they work as well as the ones I originally bought. I'll know in a few weeks, right now they are in China...

Also note the 80mm fans on top on each H2. They are fixed with a small square of Velcro to the heat sinks. No vibration noise. The speed of the fans is set using the small fan board you can see in the picture shown earlier.

[domih]

Update

See front and back view of the case shown below. Work in progress. The LED power buttons left the top of the case to go down with the 3 x H2 shelf. One more shelf TBD for the PSU + breakout board + fan board.

Note: I dropped using a mesh for the front side of the case. I'll instead use a dark grey or blue acrylic door.
.

case_front.png (1.29 MiB) Viewed 4964 times

.

case_back.png (1.32 MiB) Viewed 4964 times

.
Note the oblong opening at the back of the shelf to allow cables to go through. The PSU shelf will have the same opening.

1 of 3 Connect X3 Pro NIC with custom fan fixed on acrylic sheet. Two bands of Velcro will allow to stick the acrylic sheet to the side of the case. Side and position will depend on the M.2 NMVe M key to PCIe x4 adapter cable which I have not received yet.
.

nic_front.png (1.8 MiB) Viewed 4964 times

.

nic_back.png (1.7 MiB) Viewed 4964 times

[venkatbo]

Nice effort!

Wondering if the fans will be effective (with airflow restricted) when the H2's are stacked. Small fans vertically mounted behind the fins may draw more air and cool better and eject the air freely at the back of the stack. Just a thought...

[domih]

.../...
I also found less expensive M.2 NMVe M key to PCIe x4 adapters on eBay, see https://www.ebay.com/itm/PCIe-x4-3-0-PC ... 4398886092. Hopefully they work as well as the ones I originally bought. I'll know in a few weeks, right now they are in China...
.../...

For the people who are interested by these adapters:

I received the 3 adapters today, so that's relatively quick from CH/HK/TW. I tested one: works AOK. There branded ADT Link. They come with the SATA to 4-pin FDD cable for power as well as the small M.2 screw. In other words, they are ready for use once received, no extra cost. I bought straight 30 cm so that I can pivot the cards 180. The pricing is much less than what I found before: $23.39 (eBay) a piece vs $30.40 or $55.10 (AliExpress). Free shipping.

[domih]

Nice effort!

Wondering if the fans will be effective (with airflow restricted) when the H2's are stacked. Small fans vertically mounted behind the fins may draw more air and cool better and eject the air freely at the back of the stack. Just a thought...

Thank you.

Yes, we'll see at usage because predicting air flow is a tall order. The fans are pushing air to the heat sinks so the air will come out by the sides. The back of the case will stay open. The front will be mesh - I reverted to mesh instead of using a blue acrylic door(*). Given the size of the H2 heat sink, it is supposed to work OK without fan. I'm adding fans because as a rule of thumb they lower the cooled CPU or IC temperature by 20 C and also to prevent #1 cooking #2 cooking #3 by convection. The 80 mm are oversized for the task, allowing me to reduce their RPM for silent operation. The main source of heat coming from the above H2 will be the 32 GB of memory. Finally there is also the 3 Mellanox NICs which will blow hot air from the side. So I have no clue what the final result will be.

(*) I experimented with small pieces of acrylic and I do not trust myself able to pull off a nice door with perfect cuts...

[domih]

Update

The wood work is quasi-complete. See picture below. Screw caps on the side to give a good enough look. Disks cage installed with Velcro taping at the bottom.

Now I'm working on the PSU, PSU break out card and fan card installation, then the front mesh panels. Last to go in will be the 3 x H2 and Mellanox cards.
.

20200606_155320.png (1.28 MiB) Viewed 4835 times

[domih]

Update

Top compartment front black mesh completed. Still working on the middle compartment one.
.

top-mesh.png (1.22 MiB) Viewed 4655 times

[domih]

Update

This is only of interest to the people who want to know how it's done.

Once the middle compartment black mesh frame is assembled and it corners smoothed, it is time to paint black mat of the 2 sides of the bars that will be visible through the mesh. Acrylic or Enamel paint will do. Acrylic is easier because you clean the brush with warm water once you are done. However I had a small bottle of Enamel handy so I used it. The brush to be cleaned with Enamel thinner. A light layer of paint is enough.
.

paint-it-black-said-mick-1.png (1.23 MiB) Viewed 4621 times

paint-it-black-2.png (1.3 MiB) Viewed 4621 times

paint-it-black-3.png (1.3 MiB) Viewed 4621 times

.
Let it dry for some time (1+ hour) and then start to glue the mesh to the 2 unpainted sides of the frame bars, one bar and one side at a time, it's just a question of patience, letting the glue cure for 15 to 30 minutes at each step. Use small clamps a sacrificial piece of wood to keep the mesh and the bar stuck together during curing.
.

mesh.png (1.35 MiB) Viewed 4621 times

.
Tip: how to get the mesh to bend cleanly at 90 degrees along a predetermined line ? It's like in paper modeling, if you want a clean bend of the paper, just trace a line with a fine point pen on the other side of the paper pressing the pen hard enough (but not too hard), then start bending the paper, the bend will exactly occur along the line, clean and perfect. For the mesh which is plastic I used a cutter, pressing it lightly enough to trace the bending line without actually fully cutting the mesh. How to know which pressure to apply: rehearse on a small piece until you obtain the intended result. Then the bend will be clean and good enough.

[domih]

Reality Check

<hopeful>Wow! The case looks as good as a commercial case.</hopeful>

Mmm... well it hopefully looks good in the pictures where photography eliminate a lot of little defects, especially because the pictures are 25% reduction of the originals.

The apparent quality is about the same if you look at it from 3 or more feet. The defects start to appear if you look closer and it is obvious this is amateur handmade (with tragi-comic mistakes) and not a work of art.

Just saying so because other forum members who want to build their own case from scratch should NOT feel intimidated to make the plunge. It's fun!

Just temper your expectations of perfection, unless you are a high quality handyman.

[mad_ady]

Don't worry - it looks better than my solution - putting a big sheet of cardboard to hide the devices/cables

[domih]

For the people tempted in using the H2 or H2+ M.2 NVMe slot as a PCI2.0 x4 slot, I just saw that Ameridroid is now selling two accessories to simplify you life:

- https://ameridroid.com/collections/new/ ... pcie-cards
- https://ameridroid.com/collections/new/ ... r-straight

[domih]

Update

The case now has its final look with the middle compartment front mesh completed.

Next step: fix the electronics inside the case.

final-front-view.png (1.45 MiB) Viewed 4420 times

[domih]

Update

Initial integration testing without the HDDs

See below the 3xH2's powered on with the 3xMellanox ConnectX-3 Pro network cards and plugged to the Mellanox 18-port FDR10 switch.

3xH2 with Mellanox network cards.png (3.62 MiB) Viewed 4306 times

2xPCs+3xH2 plugged into the FDR10 switch.png (1.94 MiB) Viewed 4306 times

For a quick introduction about the Mellanox InfiniBand networks, see https://magazine.odroid.com/article/net ... odroid-h2/

Let's check that the IB local net shows all the expected devices (it does):

Code: Select all

domih@h2a:~$ sudo iblinkinfo
CA: h2b mlx4_0:
      0xc4346bffffdc3691      7    1[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>      12    9[  ] "SwitchX -  Mellanox Technologies" ( )
CA: h2c mlx4_0:
      0xc4346bffffdc5561      5    1[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>      12   11[  ] "SwitchX -  Mellanox Technologies" ( )
CA: i5-a mlx4_0:
      0x0002c90300426771      2    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      12    3[  ] "SwitchX -  Mellanox Technologies" ( Could be FDR10 (Found link at QDR but expected speed is FDR10))
CA: ripper HCA-1:
      0xf452140300346ea1      3    1[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>      12    1[  ] "SwitchX -  Mellanox Technologies" ( )
Switch: 0xf452140300789a60 SwitchX -  Mellanox Technologies:
          12    1[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>       3    1[  ] "ripper HCA-1" ( )
          12    2[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    3[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    1[  ] "i5-a mlx4_0" ( Could be FDR10 (Found link at QDR but expected speed is FDR10))
          12    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    9[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>       7    1[  ] "h2b mlx4_0" ( )
          12   10[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>       1    1[  ] "h2a mlx4_0" ( )
          12   11[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>       5    1[  ] "h2c mlx4_0" ( )
          12   12[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   13[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   14[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   15[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   16[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   17[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   18[  ] ==(                Down/ Polling)==>             [  ] "" ( )
CA: h2a mlx4_0:
      0xc4346bffffdc55b1      1    1[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>      12   10[  ] "SwitchX -  Mellanox Technologies" ( )

Let's check that one of the H2s can ping the i5-a PC via the regular 1 GbE network and via the IPoIB network (it can):

Code: Select all

domih@h2a:~$ ping i5-a -c 10
PING i5-a (192.168.1.20) 56(84) bytes of data.
64 bytes from i5-a (192.168.1.20): icmp_seq=1 ttl=64 time=0.804 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=2 ttl=64 time=0.829 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=3 ttl=64 time=0.825 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=4 ttl=64 time=0.638 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=5 ttl=64 time=0.819 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=6 ttl=64 time=0.827 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=7 ttl=64 time=0.826 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=8 ttl=64 time=0.826 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=9 ttl=64 time=0.829 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=10 ttl=64 time=0.823 ms

--- i5-a ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9037ms
rtt min/avg/max/mdev = 0.638/0.804/0.829/0.055 ms
domih@h2a:~$ ping i5-a.ib -c 10
PING i5-a.ib (10.10.56.20) 56(84) bytes of data.
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=1 ttl=64 time=0.140 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=2 ttl=64 time=0.190 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=3 ttl=64 time=0.179 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=4 ttl=64 time=0.175 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=5 ttl=64 time=0.177 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=6 ttl=64 time=0.177 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=7 ttl=64 time=0.176 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=8 ttl=64 time=0.177 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=9 ttl=64 time=0.178 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=10 ttl=64 time=0.178 ms

--- i5-a.ib ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9207ms
rtt min/avg/max/mdev = 0.140/0.174/0.190/0.012 ms

From there let's perform iperf3 sessions over IPoIB from the 3xH2's toward the i5-a PC:

Code: Select all

domih@h2c:~$ iperf3 -c i5-a.ib --bind 10.10.56.72
Connecting to host i5-a.ib, port 5201
[  4] local 10.10.56.72 port 40529 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   1.00-2.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   2.00-3.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   3.00-4.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   4.00-5.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   5.00-6.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   6.00-7.00   sec  1.31 GBytes  11.3 Gbits/sec    0   3.18 MBytes       
[  4]   7.00-8.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   8.00-9.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   9.00-10.00  sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec                  receiver

iperf Done.
domih@h2c:~$ iperf3 -c i5-a.ib --bind 10.10.56.72 -R
Connecting to host i5-a.ib, port 5201
Reverse mode, remote host i5-a.ib is sending
[  4] local 10.10.56.72 port 49109 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   1.00-2.00   sec  1.57 GBytes  13.5 Gbits/sec                  
[  4]   2.00-3.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   3.00-4.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   4.00-5.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   5.00-6.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   6.00-7.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   7.00-8.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   8.00-9.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   9.00-10.00  sec  1.67 GBytes  14.4 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  16.6 GBytes  14.3 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  16.6 GBytes  14.3 Gbits/sec                  receiver

iperf Done.


domih@h2b:~$ iperf3 -c i5-a.ib --bind 10.10.56.71
Connecting to host i5-a.ib, port 5201
[  5] local 10.10.56.71 port 42363 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   1.00-2.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   2.00-3.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   3.00-4.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   4.00-5.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   5.00-6.00   sec  1.31 GBytes  11.3 Gbits/sec    0   3.18 MBytes       
[  5]   6.00-7.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   7.00-8.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   8.00-9.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   9.00-10.00  sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec                  receiver

iperf Done.
domih@h2b:~$ iperf3 -c i5-a.ib --bind 10.10.56.71 -R
Connecting to host i5-a.ib, port 5201
Reverse mode, remote host i5-a.ib is sending
[  5] local 10.10.56.71 port 46419 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.54 GBytes  13.2 Gbits/sec                  
[  5]   1.00-2.00   sec  1.56 GBytes  13.4 Gbits/sec                  
[  5]   2.00-3.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   3.00-4.00   sec  1.66 GBytes  14.2 Gbits/sec                  
[  5]   4.00-5.00   sec  1.67 GBytes  14.3 Gbits/sec                  
[  5]   5.00-6.00   sec  1.67 GBytes  14.3 Gbits/sec                  
[  5]   6.00-7.00   sec  1.65 GBytes  14.2 Gbits/sec                  
[  5]   7.00-8.00   sec  1.65 GBytes  14.1 Gbits/sec                  
[  5]   8.00-9.00   sec  1.67 GBytes  14.3 Gbits/sec                  
[  5]   9.00-10.00  sec  1.67 GBytes  14.3 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.4 GBytes  14.1 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  16.4 GBytes  14.1 Gbits/sec                  receiver

iperf Done.

domih@h2a:~$ iperf3 -c i5-a.ib --bind 10.10.56.70
Connecting to host i5-a.ib, port 5201
[  5] local 10.10.56.70 port 57781 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   1.00-2.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   2.00-3.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   3.00-4.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   4.00-5.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   5.00-6.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   6.00-7.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   7.00-8.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   8.00-9.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   9.00-10.00  sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec                  receiver

iperf Done.
domih@h2a:~$ iperf3 -c i5-a.ib --bind 10.10.56.70 -R
Connecting to host i5-a.ib, port 5201
Reverse mode, remote host i5-a.ib is sending
[  5] local 10.10.56.70 port 48207 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.67 GBytes  14.3 Gbits/sec                  
[  5]   1.00-2.00   sec  1.65 GBytes  14.2 Gbits/sec                  
[  5]   2.00-3.00   sec  1.67 GBytes  14.3 Gbits/sec                  
[  5]   3.00-4.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   4.00-5.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   5.00-6.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   6.00-7.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   7.00-8.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   8.00-9.00   sec  1.64 GBytes  14.1 Gbits/sec                  
[  5]   9.00-10.00  sec  1.66 GBytes  14.3 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.7 GBytes  14.3 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  16.6 GBytes  14.3 Gbits/sec                  receiver

iperf Done.

As expected we get about 11 Gbits/sec and 14 Gbits/sec depending on direction. In the reverse direction the i5-a PC is running a 9600K which is obviously faster at sending packets than the Celeron J4105.

Let's do iperf3 sessions from H2 to H2 over IPoIB:

Code: Select all

domih@h2c:~$ iperf3 -c h2a.ib --bind 10.10.56.72
Connecting to host h2a.ib, port 5201
[  4] local 10.10.56.72 port 42445 connected to 10.10.56.70 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.26 GBytes  10.8 Gbits/sec    0   3.06 MBytes       
[  4]   1.00-2.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   3.00-4.00   sec  1.30 GBytes  11.1 Gbits/sec    0   3.06 MBytes       
[  4]   4.00-5.00   sec  1.29 GBytes  11.1 Gbits/sec    0   3.06 MBytes       
[  4]   5.00-6.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   6.00-7.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   7.00-8.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   8.00-9.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   9.00-10.00  sec  1.29 GBytes  11.1 Gbits/sec    0   3.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  12.9 GBytes  11.1 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  12.9 GBytes  11.1 Gbits/sec                  receiver

iperf Done.
domih@h2c:~$ iperf3 -c h2a.ib --bind 10.10.56.72 -R
Connecting to host h2a.ib, port 5201
Reverse mode, remote host h2a.ib is sending
[  4] local 10.10.56.72 port 42283 connected to 10.10.56.70 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.30 GBytes  11.2 Gbits/sec                  
[  4]   1.00-2.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   2.00-3.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   3.00-4.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   4.00-5.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   5.00-6.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   6.00-7.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   7.00-8.00   sec  1.30 GBytes  11.2 Gbits/sec                  
[  4]   8.00-9.00   sec  1.30 GBytes  11.2 Gbits/sec                  
[  4]   9.00-10.00  sec  1.30 GBytes  11.2 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  13.0 GBytes  11.2 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  13.0 GBytes  11.2 Gbits/sec                  receiver

iperf Done.


domih@h2b:~$ iperf3 -c h2a.ib --bind 10.10.56.71
Connecting to host h2a.ib, port 5201
[  5] local 10.10.56.71 port 44087 connected to 10.10.56.70 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.30 GBytes  11.1 Gbits/sec    0   3.18 MBytes       
[  5]   1.00-2.00   sec  1.30 GBytes  11.1 Gbits/sec    0   3.18 MBytes       
[  5]   2.00-3.00   sec  1.30 GBytes  11.1 Gbits/sec    0   3.18 MBytes       
[  5]   3.00-4.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   4.00-5.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   5.00-6.00   sec  1.30 GBytes  11.1 Gbits/sec    0   3.18 MBytes       
[  5]   6.00-7.00   sec  1.23 GBytes  10.6 Gbits/sec    0   3.18 MBytes       
[  5]   7.00-8.00   sec  1.24 GBytes  10.6 Gbits/sec    0   3.18 MBytes       
[  5]   8.00-9.00   sec  1.25 GBytes  10.7 Gbits/sec    0   3.18 MBytes       
[  5]   9.00-10.00  sec  1.29 GBytes  11.1 Gbits/sec    0   3.18 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  12.8 GBytes  11.0 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  12.8 GBytes  11.0 Gbits/sec                  receiver

iperf Done.
domih@h2b:~$ iperf3 -c h2a.ib --bind 10.10.56.71 -R
Connecting to host h2a.ib, port 5201
Reverse mode, remote host h2a.ib is sending
[  5] local 10.10.56.71 port 45811 connected to 10.10.56.70 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.30 GBytes  11.1 Gbits/sec                  
[  5]   1.00-2.00   sec  1.25 GBytes  10.7 Gbits/sec                  
[  5]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec                  
[  5]   3.00-4.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  5]   4.00-5.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  5]   5.00-6.00   sec  1.30 GBytes  11.1 Gbits/sec                  
[  5]   6.00-7.00   sec  1.28 GBytes  11.0 Gbits/sec                  
[  5]   7.00-8.00   sec  1.29 GBytes  11.1 Gbits/sec                  
[  5]   8.00-9.00   sec  1.30 GBytes  11.2 Gbits/sec                  
[  5]   9.00-10.00  sec  1.31 GBytes  11.2 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec  12.9 GBytes  11.1 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  12.9 GBytes  11.1 Gbits/sec                  receiver

iperf Done.

As expected we get around 11 Gbits/sec in both directions.

CONCLUSION
The three H2s behave as expected with IPoIB, at least from the iperf3 point of view.

FINAL REMARK
IPoIB with FDR10 is able to offer about 38 GbE connections with modern PCs. We lose a lot of bandwith on the H2 due to the expected hardware limitations. So does it makes sense to even contemplate such a solution? It sounds like putting a Lotus car body around a moped engine. As a matter of fact, it does. It would be totally stupid/idiotic/crazy to spend thousands of $$$ for brand new Mellanox hardware for an H2.

However, it DOES totally makes sense with the used hardware you can find on eBay:
- 18-port FDR10 Mellanox: got one for $150.
- 36-port FDR Mellanox: got one for $200.
- Mellanox ConnectX-3 Pro cards: got for example 4 cards at $45 each. Other Pro and non Pro cards for similar prices ($45 to $65).
- FDR 56 Gbps IB cables: got for example at once 12 x 2M DAC copper for $100, that's around $8.5 per cable.

The tricks are:
- You have to know what you are looking for.
- You have to be patient for the very good deals.
- You really have to test all the hardware upon reception.

Yes we get "only" 11 GbE with H2. But on the same local net we get around 44+ GbE with FDR with modern PCs (Intel 8/9th gen. or higher, AMD Zen+ or Zen 2).

Compare this to the current RJ-45 new 10 GbE hardware. An 18 or 36-port 10 GbE switch = $$$$, 10 GbE cards are still quite expensive $$$ + CAT6-A cables $$.

Obvious conclusion: so you DO get more while spending less.

So as the French say: "Qu'importe le flacon, pourvu qu'on ait l'ivresse !" which can translate as "What matters the jug, if euphoria be within?"

Next Steps

(1) test and observe differences for Samba, NFS, GlusterFS(?) and Cassandra over:
- 1 GbE
- 2.5 GbE
- SFP+ 10 GbE
- 11+ GbE IPoIB

(2) Finalize case project:
- Put all the electronics in the tower case,
- Enjoy a 3-node server over 11+ GbE

[domih]

Update

This is only of interest to the people who want to know how it's done.

Final step of the case building proper: gluing little pieces of wood with Velcro to fix the front mesh panels while keeping them removable for easier future inside maintenance. See pictures below.

Next step: install the boards and PSU inside the case. However it will only happen in a few weeks because I'm going to do testing for 1 GbE, 2.5 GbE, 10 GbE SFP+ and 11+ GbE IPoIB with Samba and NFS, meaning a lot of configuration changes easier to make with the boards out of the case. For IPoIB the testing will be made with the FDR10 switch because I'm still working on customizing the fan set with "silent" Noctua fans instead of the crazy noisy Delta ones on the full FDR switch.

Optional ultimate step for the case: when I'm in the mood I'll try to smooth a few of the seams, fill them with some putty and paint them white insignia to make them look better. As of now I see it as optional. Once the tower is placed at its operational location nobody will look at it closely, maybe the occasional roaming house spider.

1.front-mesh-velcro-attachments-painted-white.png (1.66 MiB) Viewed 4141 times

2.gluing-them-to-the-case.png (1.33 MiB) Viewed 4141 times

3.how-it-looks-finished.png (855.51 KiB) Viewed 4141 times

4.velcro-loops-on-the-front-mesh-back.png (1.3 MiB) Viewed 4141 times

[domih]

Update

Change of plan: I can proceed with the planed testing with the assembled system, so here we are, see photos shown below. Work in progress, spaghetti cables all over the place. Middle compartment front mesh to be fixed once the LED power buttons are connected. HDDs are finally plugged in.

partially-assembled-front-view.png (1.51 MiB) Viewed 4008 times

partially-assembled-rear-view.png (1.78 MiB) Viewed 4008 times

<humor>

Thank you NZXT, Phanteks, etc(*) for the inspiration but my tower rocks compared to yours

3 x H2 computers for distributed processing
12 cores
96 GB of memory
5 x 3.5" bays
30 TB of disk storage
Enough remaining space for 3 x SATA SSD
192 GB eMMC
1 GbE, 2.5 GbE and 11+ GbE networking
Self-made & 15 μs of fame = priceless

(*) https://www.nzxt.com/product-overview/h1, http://phanteks.com/Enthoo-Evolv-Shift.html.

Long term aspiration: find a the pretext and the adequate SBCs to make something ultra-cool like this one http://www.gamerstorm.com/product/CASES ... 4840.shtml

</humor>

[domih]

Ambient temperature: 22C. 3 x H2's running with HDDs, idle: power 38.5W, CPUs temperature: 32 to 34C. H2's off but fans and HDDs still powered: 11.8W. All devices turned off: PSU alone shows 0W at the wall.

I connected the power buttons and fixed the middle compartment mesh. See picture below.

With the the middle compartment mesh installed the CPUs thermal stay the same at idle: 32 to 34C (after 30 minutes). I'm going to let them run idle for a few hours.

led-power-buttons.png (1.6 MiB) Viewed 4005 times

[domih]

Update

Testing the network cards. Some definitive cable management has to be done before I stick the cards inside the case.

cable-management-needed.png (1.82 MiB) Viewed 3945 times

I tested running 3 concurrent iperf3 sessions from a PC toward the 3 x H2. Because the 3 connections were going through the same port on the PC side I reached the max FDR10 IPoIB speed of about 38 GbE. So I ended up with slightly more than 12 GbE throughput on each H2. I guess I'll reach 14+ GbE with the FDR switch.

On the PC side the System Monitor showed an outgoing network throughput of 4.7GB/sec

parallel-3x12GbE.png (1.69 MiB) Viewed 3945 times

[misaz]

4.7 GB is correct. Note that lowercase b is bits and uppercase B is bytes. Every byte have 8 bits. So if you want to calculate speed in bits per second instead of bytes per second just multiple value by 8.

4.7 GBps = 4.7 * 8 Gbps = 37.6 Gbps.

Alternatively you can use google gadget to convert just type "4.7 GB to Gb" (https://www.google.com/search?q=4.7+GB+to+Gb)

[powerful owl]

@domih, thanks for the updates I need to make something similar...

[domih]

4.7 GB is correct. Note that lowercase b is bits and uppercase B is bytes. Every byte have 8 bits. So if you want to calculate speed in bits per second instead of bytes per second just multiple value by 8.

4.7 GBps = 4.7 * 8 Gbps = 37.6 Gbps.

Alternatively you can use google gadget to convert just type "4.7 GB to Gb" (https://www.google.com/search?q=4.7+GB+to+Gb)

Yes, it is

Also for converting <whatever>bytes to <whatever>bits, it also depends on who or what is counting, meaning using mega=1,000 or mega=1,024. Usually network bandwidth is expressed using mega=1,000. I assume that is what the System Monitor does.

On the other hand, if you were to express the "resulting" bandwidth by measuring the overall time it takes to transfer a file of N x GB in size, then you have to use mega=1,024 to get the right size in bytes (an x8 for bits) and then use mega=1,000 to get the bandwidth in Gb/sec.

[domih]

Update

Testing the network cards but this time the SolarFlare 10G SFP+ instead of the Mellanox ConnectX-3 Pro 56Gbps.

3xH2 with SolarFlare 10G SFP+.png (1.79 MiB) Viewed 3892 times

Connections to MicroTik Switch.png (1.52 MiB) Viewed 3892 times

Note: The white RJ-45 CAT6-A cable connects a PC with a 10GbE Aquantia onboard NIC. It plugs into the switch via an Ipolex SFP+ <-> RJ45 transceiver.

Tip
In order to work the SolarFlare NIC requires PCI Express Clock Gating to be Disabled and Compliance Mode to be Enabled in the BIOS. Once in the BIOS to go to Chipset then South Cluster Configuration then PCI Express Configuration.

BIOS.png (1.22 MiB) Viewed 3892 times

As we should expect iperf3 shows a 10Gbe connection between the PC and one of the H2's:

Code: Select all

domih@trx:~$ iperf3 -c h2a.25
Connecting to host h2a.25, port 5201
[  5] local 172.16.25.36 port 40518 connected to 172.16.25.70 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.79 MBytes       
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.88 MBytes       
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.98 MBytes       
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.08 MBytes       
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.08 MBytes       
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.19 MBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.41 Gbits/sec    0   2.19 MBytes       
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.19 MBytes       
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.19 MBytes       
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec    0   2.19 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec    0             sender
[  5]   0.00-10.15  sec  11.0 GBytes  9.27 Gbits/sec                  receiver

Now, because the Aquantia onboard NIC on the PC tops at 10GbE if we run 3 concurrent iperf3 sessions from the PC to each of the 3 x H2's, each session tops at about 10/3 = 3.3 GbE. This is indeed what we see in the screenshot shown below:

3 x H2 10GbE SFP+.png (1.78 MiB) Viewed 3892 times

The overall bandwidth as shown in the System Monitor is 1.3 GB/sec (roughly 10Gbe)

So we are obviously far below the 3 x 12.5 (roughly 38 Gb/sec) we got with the Mellanox cards IPoIB. Choose your network solution depending on what you are planing to do with it and... your budget. In the example above, if we were to have one high-performance PC workstation doing multi-thread data processing from data coming from the 3 x H2's it makes quite a lot of sense to prefer the IPoIB Mellanox solution because the overall available network bandwidth is about 4 times larger (even more once the FDR10 switch is replaced with the FDR one). On the other hand, if we had 3 clients each talking to one H2, the difference would be much smaller (11+ to 14+ Gb/sec instead of 9.4 Gb/sec with 10 GbE SFP+ or 10GBaseT).

Given the fact that the cost of used Mellanox hardware is not obscenely higher than the cost of used/new SFP+ hardware (maybe less than twice) I believe that if you chose to go "exotic" instead of 10GBaseT the Mellanox hardware is a better solution. However the Mellanox solution will require you to do some studies (read documentation, knowing what to buy) and customizing (involves soldering) the fan set of the switch (if to be used in an apartment) because these switches are extreme noisy beasts. The currently in progress FDR switch fan set customization is shown below. It consists in opening the box, remove the existing fan, put silent Noctua fans (top panel with not be put back) and soldering the cables (which means retrieving the original cabling online, not on the Mellanox site but on the fans manufacturer site). Not transcendental but requires time and patience.

FDR switch fan set customization.png (1.89 MiB) Viewed 3892 times

Note: the silent Noctua fans add to the overall cost, so the switch ends up at about $400, but that's a 36-port 56Gb/ps. And you do not need to change the fans if you have an isolated room (e.g. a sound proof attic or garage).

CONCLUSION
Depending on budget and needs you can go 1 GbE, 2.5 GbE, 10GBaseT (RJ-45), 10G SFP+ or 11-14 Gbps with IPoIB. Personally I think Hardkernel did very well in upgrading to 2.5 GbE with the H2+ for basically the same price. Hopefully there will be manufacturers who will make cheap 2.5 GbE switches given the fact that 2.5 GbE chipset are pretty close in price to the 1 GbE chipset. An 8 x 2.5GbE switch should NOT cost $400. That's really taking us the users for suckers.

[domih]

Update

The Mellanox network cards are now inside the case in their final location (but using the Velcro allowing for removal without breaking anything).

network-cards-are-in-front-view.png (1.49 MiB) Viewed 3549 times

network-cards-are-in-rear-view.png (1.77 MiB) Viewed 3549 times

Everything is now connected and up and running. I did proceed with some cable management. See image below. I'll make it better when I'm in the mood. Also there will be 6 less cables (HDMI and USB to KVM) once I stopped using them via the Mate GUI and use SSH instead.

all-connected-up-and-running.png (1.8 MiB) Viewed 3549 times

Additional Information

Idle power with 3 x H2 + 3 x 10TB disk + 4 x 80mm fans + 3 x ConnectX3 Pro + 3 x 50mm fans + LED Power Buttons : 50.2 W

Thermals

Ambient temperature: 24C

Left idle for 45 minutes

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
34000
34000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
33000
34000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
33000
33000

Left idle for 90 minutes

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
37000
36000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
36000
35000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
35000
35000

Note: h2a is the one at the bottom, h2b the one in the middle, h2c the one at the top.

The whole tower is virtually silent during the day. By night you have to be close to it to hear the fans.

CONCLUSION : The thermals are good when idle. We will see how it goes when the 3 x H2 do heavy compute and disk access.

[domih]

Update: New Numbers

Note: no more keyboard and mouse via KVM. No more desktop, booting in multi-user.target. The H2 are now used and accessed via SSH.

Power Consumption

Booting 3 x H2 all at same time, max wattage: 77.4W

Idle power with 3 x H2 + 3 x 10TB disk + 4 x 80mm fans + 3 x ConnectX3 Pro + 3 x 50mm fans + LED Power Buttons : 45.3W

Thermals

Ambient temperature: 26.5C

Left Idle for 10 minutes

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
36000
36000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
35000
35000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
35000
34000

Running iperf3 as server on each H2 @ around 12Gbits/sec

Total wattage: around 67W

After two minutes:

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
52000
50000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
47000
47000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
50000
50000

After 10 minutes:

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
55000
54000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
49000
50000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
52000
54000

After 30 minutes:

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
56000
57000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
48000
50000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
52000
52000

After 45 minutes:
domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
55000
54000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
50000
50000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
51000
54000

After 55 minutes:
domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
55000
54000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
49000
51000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
51000
52000

CONCLUSION:
- Thermals: from 51C to 57C, we're fine.
- Total segment retries: 2 (h2a) + 2 (h2b) + 5 (h2c)
- Average throughput: 11.4 Gbits/sec (h2a), 11.8 Gbits/sec (h2b), 11.7 Gbits/sec (h2c) Note: should have been 14+ but the 3 sessions ran from the same PC so we topped at the FDR10 limit.
- Total data transferred: 4.67 TBytes (h2a), 4.83 TBytes (h2b), 4.80 TBytes (h2c)


Next (upcoming in the next few weeks)
- Same thing in reverse (will be much more taxing on the H2/H2+ cores).
- Running Cassandra stress tool (isolated) on each of the 3 x H2 (for power and thermal analysis; should be even more taxing and pushing all cores to 100%).
- NFS async testing.
- Samba sync and async testing.
- Running Cassandra stress tool in distributed mode (including 1Gbe vs. 11+GbE IPoIB comparison). I will do only do 2.5Gbe when there are 100% working USB 3 adapter because the tower now includes 2 x H2 and one H2+.
- Possibly glusterfs for 1Gbe vs. 2.5Gbe vs. 11+GbE IPoIB comparison (but I need to refresh my knowledge about it).

[domih]

Update
Testing the Cassandra Stress tool running (isolated) on the 3 x H2. The main purpose being to measure power and thermals while the 3 x 4 x cores are at >85 to 100%(*).

Power
Write test: variable from 68W to 90W
Read test: pretty stable at 80W

Thermals
Write & Read: 54C to 67C (Ambient 27C)

(*) Using htop to confirm all cores are pushed to >85 to 100% (if htop is to be trusted). Note: screenshot shows high 80's but I saw a lot of 100% during the several minutes test.

htop-cass-h2a-h2b-h2c-100.png (851.61 KiB) Viewed 3217 times

For the Cassandra Stress results (which were not the main purpose) there was a little surprise in store:

cassandra-stress-h2a-h2b-h2c.png (108.28 KiB) Viewed 3217 times

h2a, h2b, h2c (identical hardware and software, except h2a is a H2+ while b and c are H2's) were pretty much within 5% of each other during the write test.
On the other hand h2c decided to act up way above h2 a and h2b during the read test. Not sure why, I'll have to check the configuration.

CONCLUSION: we are OK in terms of power and thermals. We can move forward to the other tests.

[rupy]

@domih Neat, you could run my software instead to get better performance: https://github.com/tinspin/rupy

Here is my latest cluster that will soon be upgraded with RPi 4 with custom heatsink:

http://sprout.rupy.se/article?id=290
http://move.rupy.se/file/pi_4.jpg

[domih]

Samba Testing (Part I)

Client PC
AMD Threadripper 3960X

NAS PC 1...: AMD V1605B, 32GB @ 2400MT/s, WD Red 10TB
NAS PC 2...: AMD R5 3400G, 32GB mem @ 3200MT/s, WD Red 10TB
NAS PC 3...: Odroid H2, Intel Celeron J4105, 32GB @ 2400MT/s, HSGT He10
NAS PC 4...: Odroid H2+, Intel Celeron J4115, 32GB @ 2400MT/s, HSGT He10

Notes
1. On each NAS PC the hard disk was already about 25% full.
2. The Odroid H2 was used for the 1 GbE and IPoIB tests. The Odroid H2+ was used for the 2.5 GbE tests.
3. I included two low mid-range PC for context.
4. All machines running Ubuntu 18.04 or 20.04.

Tests
- Copy a 1 GB file (uploading)
- Copy a 2 GB file (uploading)
- Copy a 4 GB file (uploading)
- Copy a folder of 500 files (100 of 1, 2, 3, 4 and 5 MB) for a total of 1.6 GB.

The files were created this way:

Code: Select all

#!/bin/bash
for i in {1..100}
do
   dd if=/dev/urandom of=./test_dir/"$i 1MB.txt" count=1 bs=1048576
   dd if=/dev/urandom of=./test_dir/"$i 2MB.txt" count=2 bs=1048576
   dd if=/dev/urandom of=./test_dir/"$i 3MB.txt" count=3 bs=1048576
   dd if=/dev/urandom of=./test_dir/"$i 4MB.txt" count=4 bs=1048576
   dd if=/dev/urandom of=./test_dir/"$i 5MB.txt" count=5 bs=1048576
done

dd if=/dev/urandom of=~/Downloads/1GB.txt count=1024 bs=1048576
dd if=/dev/urandom of=~/Downloads/2GB.txt count=2048 bs=1048576
dd if=/dev/urandom of=~/Downloads/4GB.txt count=4096 bs=1048576

The measurements were made this way:

Code: Select all

# 1, 2 and 4 GB file
sudo smbclient '//xxx/xxx/' xxxxxxxx -U domih  -c "prompt OFF;rm 1GB-hdd-2.5GbE.txt" ; \
time sudo smbclient '//xxx/xxx/' xxxxxxxx -U domih  -c "put /mnt/raid/test/1GB.txt 1GB-hdd-2.5GbE.txt"

# Folder of 500 files
sudo smbclient '//xxx/xxx/' xxxxxxxx -U domih  -c "prompt OFF;deltree test_dir_1.6_GB_500_files_2.5GbE" ; \
sleep 30 ; \
time sudo smbclient '//h2a.25/h2a/' dev.2019 -U domih  -c "prompt OFF;recurse ON;mkdir test_dir_1.6_GB_500_files_2.5GbE; cd test_dir_1.6_GB_500_files_2.5GbE; lcd /mnt/raid/test/test_dir_1.6_GB_500_files;mput *"

iperf3 Testing
Let's do iperf3 tests to get a view of the landscape:

iperf3-results.png (45.95 KiB) Viewed 2473 times

Samba Testing

Note about Samba testing: in order to eliminate variations due from caching I always run the a 1 GbE test once or twice before registering the next 6 results, no need for the 2.5 GbE tests which always follow the 1 GbE tests. I also eliminate outliers (e.g. variations due to background execution like CRON).

1GB.png (71.33 KiB) Viewed 2473 times

2GB.png (71.49 KiB) Viewed 2473 times

4GB.png (71.37 KiB) Viewed 2473 times

folder.png (75 KiB) Viewed 2473 times

CONCLUSIONS

1. Note that the 1 GbE results are basically the same across the 4 NAS PC! So yes you can run a NAS with an H2 at 1 GbE for home or SOHO usage, it will do as good and consume less over the years.

2. The Odroid H2+ with 2.5 GbE being 2.464, 2.487, 2.485 times faster than 1 GbE for the 1, 2 and 4 GB file copy (pure sequential writing) is extremely close to the 2.5 times faster from iperf3. Only for the 500 files folder the ratio falls to 1.7 faster. 2.5 GbE is not as demanding at 10+ GbE for the internal components (e.g. disk IO, power of the CPU, etc). It is therefore probably that 2.5 GbE with a more powerful PC would provide similar performance. So yes you can run a NAS with an H2+ at 2.5 GbE for home or SOHO usage, it will do as good and consume less over the years.

3. For IPoIB, the internal "oomph" of the PC greatly affects the throughput. At iperf3 34.4 Gbits/sec the 3400G is approaching the expected practical maximal throughput of IPoIB over FDR10. At around 20.0 Gbits/sec the V1605B already reaches its limit (the CPU was made with 10 GbE). At 11 Gbits/sec the Odroid H2's Celeron J4105 does as best as it can. Consequently it is no wonder that IPoIB does not report ratios for Samba tasks as high as iperf3 ratios. This where you can see the difference between Enterprise/Data Center and Desktop/Consumer hardware. With the former, the internal components (e.g. beefy CPUs, offloading disk IO to RAID components) would be much more able to face the throughput of the network. It's just a question of $$$.

[domih]

To (partially) illustrate point 3 of the conclusions in the previous post, I proceeded with a test with the TRX as Samba server with a share located on a 2 x NVMe RAID 0 (using mdadm). Here are the results:

trx.png (85.56 KiB) Viewed 2457 times

If we compare the throughput ratios, we can see that the TRX samba server flies high above the other servers:

ratio-summary.png (78.25 KiB) Viewed 2457 times

Note: note the quirk on copying the 500 files folder (which I have not looked into yet).

So if you really need an ultra-fast beefy NAS and IF money and energy consumption are not an issue, use:
- AMD EPYC (for the 128 PCIe 4 lanes, 8 channels memory)
- A beefy RAID card (a real one with real offloading of the RAID functions from the CPU to the card)
- 16 or 24 SAS 12Gbps (or 24 Gbps) SSD configured in RAID 10
- 100GbE NICs.

Then you'll have at home what they use in Enterprise or Data Centers. If it is still not enough, visit the site of the main players of this field (tip: find a Gartner report for the names, https://www.bing.com/images/search?q=en ... er&first=1).

Landing back on Earth : For Home and SOHO, an H2+ based NAS over 2.5 GbE works pretty well.

If 2.5 GbE is not enough, you can scale up your H2 or H2+ with 10 GbE RJ-45 or SFP+ or ConnectX 3 Pro (with IPoIB and/or RDMA).

But do not forget that the numbers returned by iperf3, while real, are also theoretical once you involve the other components of the system (CPU, Memory, Disk IO).

[domih]

To complete the Copy (upload) Samba testing, here are the SFP+ tests.

iperf3

iperf3-with-sfp+.png (35.75 KiB) Viewed 2375 times

Samba

samba-sfp+.png (91.29 KiB) Viewed 2375 times

Optimizations (ratios) over 1 GbE

ratios-comparison.png (95.76 KiB) Viewed 2375 times

Note how the 10 SFP+ and IPoIB are basically identical within margins of error for copying the 1, 2 and 4 GB files.

The SFP+ acceleration only starts to collapse in comparison to that of IPoIB when copying the 500 files folder.

I suspect an 10 GbE RJ-45 NIC would show the same behavior.

CONCLUSION (mostly a repeat of what I posted in this or other threads)
1. If you want your H2+(s) and other computers to talk 10 GbE, you can go the RJ-45 or SFP+ way, the latter being significantly less expensive (as of this writing) with eBay used hardware in comparison to the new 10 GbE consumer hardware.
2. It only makes sense to use IPoIB if you already had the infrastructure and want to insert your H2+(s) into it or if you plan to use applications leveraging RDMA (e.g. NFS). InfiniBand eBay used hardware is more expensive than SFP+, but still less expensive than the new 10 GbE consumer hardware.

[domih]

For similar Samba testing results using an Odroid N2+ see viewtopic.php?p=305819#p305819.

And for the Odroid C4 see: viewtopic.php?p=306645#p306645

[domih]

[ reserved ]

I'm done testing the N2+ and C4 with iperf3 and Samba upload and download over 1 and 2.5 GbE. For the H2/H2+, I still need to perform the Samba download tests over 1 and 2.5 GbE as well as 10G SFP+ and IPoIB. I'm currently replacing the 2 x H2 in the white wood mini-tower with 2 x H2+. I'll perform the remaining H2/H2+ Samba tests once I'm done with the overhaul of the tower.

[ reserved ]

[domih]

Do 2.5 GbE or 10+ GbE still bring acceleration when using rsync via an SSH tunnel?

This is a valid question:
1. The SSL tunnel is going to tax the CPU more even though all modern CPUs include SSL/TLS primitives acceleration. In other words the time spent in computation is going to lower the benefits of faster networking.
2. rsync brings itself it own tremendous optimization by only copying new or changed files and optionally deleting disappeared ones if configured this way. In other words will the time saved by faster networking still matters compared to the time saved by rsync.

I tried to answer these two questions using testing between two H2+:

rsync-1-GbE-vs-2.5-GbE-vs-IPoIB-Odroid H2+.png (121.35 KiB) Viewed 730 times

The 2 x H2+ are running Ubuntu 20.04 stock kernel 5.7 with the manually installed Realtek driver version 9.003.05 (see note below):

Code: Select all

domih@h2a:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.1 LTS
Release:	20.04
Codename:	focal
domih@h2a:~$ uname -a
Linux h2a 5.7.0-050700-generic #202006082127 SMP Mon Jun 8 21:30:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
domih@h2a:~$ modinfo r8125
filename:       /lib/modules/5.7.0-050700-generic/kernel/drivers/net/ethernet/realtek/r8125.ko
version:        9.003.05-NAPI
license:        GPL
description:    Realtek RTL8125 2.5Gigabit Ethernet driver

Note: I'm currently staying using the distribution stock kernel and the Realtek driver version 9.003.05 because (a) version 9.004.01 has fantasy behavior when you run multiple concurrent TCP (e.g. iperf3) sessions through several NICs (b) Upstream kernels such 5.10 rcX eliminate the fantasy behavior but still degrade the 2.5 GbE speed a bit and last but not least consume way too much CPU resources.

Because stock kernel + 9.003.05 still has the 1 GbE speed issue, I just added a 1 GbE USB adapter to the two H2+. As a control here are the iperf3 test results:

IPoIB

Code: Select all

iperf3 -c  h2a.ib
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  12.8 GBytes  11.0 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  12.8 GBytes  11.0 Gbits/sec                  receiver
iperf3 -c  h2a.ib -R
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  13.0 GBytes  11.2 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  13.0 GBytes  11.2 Gbits/sec                  receiver

2.5 GbE

Code: Select all

iperf3 -c  h2a.25_1
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.73 GBytes  2.34 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.73 GBytes  2.34 Gbits/sec                  receiver
iperf3 -c  h2a.25_1 -R
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.73 GBytes  2.34 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.73 GBytes  2.34 Gbits/sec                  receiver

1 GbE

Code: Select all

iperf3 -c h2a -t 20
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.14 GBytes   918 Mbits/sec   10             sender
[  5]   0.00-20.00  sec  2.14 GBytes   918 Mbits/sec                  receiver
iperf3 -c h2a -R
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.07 GBytes   916 Mbits/sec   25             sender
[  5]   0.00-10.00  sec  1.06 GBytes   914 Mbits/sec                  receiver

Note: 1 GbE over the USB-based NIC is slightly slower than what you would expect from a PCIe-based NIC, meaning 0.915 vs. 0.945, which is about 3%. Given the less than 5% difference I did not adjust the 1 GbE measurements to simulate a PCIe-based result.

CONCLUSION
Question #1: Yes 2.5 and 10+ GbE are about twice faster.
Question #2: Yes, you can potentially save a few minutes.
More refined conclusions:
1. Note that increasing the networking speed has diminishing returns for a protocol such as rsync: 2.5 GbE provides substantial optimization over 1 GbE, 10+ GbE does too but the difference between 2.5 GbE and 10+ GbE is only 1.14 times faster. This is with consumer hardware, data center hardware would show a different behavior with accelerators at all level of the chain, hence the $$$ difference.
2. The optimization depends entirely on what rsync ends up actually doing during a sync operation. In the testing above, the rsync client received 18.19G bytes for a total size of 2.85TB so rsync provided a 156.77 speed up. Best timings were 2m46.252s for IPoIB, 3m11.181s for 2.5GbE and 5m29.356s for 1 GbE. This leads to additional questions:
(a) How many times do you transfer batch of files amounting to tens or hundreds of GB? In normal usage: very rarely. In a more professional usage (multimedia creation, large database or VMs snaphots, etc): quite often. A user in the former category will probably be happy with the optimization from the two H2+ 2.5 GbE NICs. A user from the latter category will want to shave time as much as possible and invest in 10+ GbE capability, especially if it also allows to integrate the H2/H2++ into an already existing 10+ GbE intranet.

[domih]

To celebrate the end of the year with plenty of green lights, I finally completed the customization of the 56Gbps switch

fdr-switch.png (1.9 MiB) Viewed 306 times

connections.png (1.92 MiB) Viewed 306 times

The PCs show up with 4 x 14.06425 = 56 Gbps connections, the Odroid H2+ do too. The PCs run ConnectX-3 cards, the Odroid H2+ ConnectX-3 Pro.

iblinkinfo.png (212.02 KiB) Viewed 306 times

Let's compare FDR10 to FDR:

results.png (19.48 KiB) Viewed 306 times

CONCLUSION
1) As we already knew the Odroid H2 maxes out at 14.4 / 11.2 GbE depending on direction.
2) An AMD R3 3200g maxes out around 30 GbE.
3) The AMD R5 3400g and INTEL i5-9660K can push beyond 45 GbE.

Using the 56Gbps switch provides the same throughput as directly connecting two machines with ConnectX-3 or ConnectX-3 Pro cards. I have at the office and AMD 3900X talking directly to an INTEL i5-9660K and throughput also tops at around 45 GbE which is bliss when I transfer a complete multi-GB project from the former to the later. On Linux, the progress bar has barely the time to appear before the copy is finished.

For the hypothetical H3 (see viewtopic.php?f=55&t=35941), if Hardkernel provides PCIe Gen3 x4, I think the H3 should max out at around 20+ GbE, with Gen3 x8 it should max out at 30+ GbE. Help me dear readers, go to the H3 thread and say you want PCIe Gen3 x8

[mad_ady]

Are those noctua fans in your switch chassis? I bet they cost more than the switch itself

[domih]

Are those noctua fans in your switch chassis? I bet they cost more than the switch itself

Yes, they are. Switch on eBay: $200. Fans: $200. Total $400 for 36 x 56 Gbps ports

I currently do not have a garage or attic and the switch is in one corner of the room, so no way I can use the original 20K RPM / 70 db Delta fans except the one in the PSU which does not fully boot up without the original fan and the switch main board and PSU talk during boot: the PSU needs to be told to boot by the main board and then the main board shutdowns 30 seconds later if the PSU tells it something is wrong. So I had to put back the Delta in the PSU, fortunately the fan doesn't start with the case staying open and the PSU top cover removed, cold enough. I'll see what happens in summer...

[mad_ady]

Damn... I lost my bet...

[domih]

FDR switch with 8 x Noctua fans + 8 x NA-RC7 cables which are fan speed reducer (just a cable with a 12V / 80 ohm resistor) for fans always at full speed make it a quasi silent switch.

Power consumption idle: 31W (*). Idle will probably be 80+% of the time. So this switch which probably has seen glorious days serving up to 2 TB/sec (36 x 56 Gbps) is now a happy camper enjoying its retirement in my living room

(*) Mostly because the 20K RPM Delta fans are gone.

Bye bye FDR10 switch you served me well for 1+ year, the FDR switch is now taking your place. But do not worry you might serve again if/when the FDR switch dies on me.

[domih]

Following the experience with the NA-RC7 fan speed reducers with the FDR switch, I revised the configuration of the 3 x H2+ mini-tower. I added an NA-RC7 for the 80mm fan at the back of the hard disk cage as well as each fan on the 3 network cards. I also replaced the Noctua 80mm fan on each H2+ with an original Hard Kernel 92mm one, It appears I was kind of heavy handed with the fans because I was afraid of the thermal budget for 3 x H2+ with 3 x high-speed NICs in a same case. Testing showed it was not an issue given that the back of the tower stays open. The 92mm Odroid fans are connected to the H2+ mobos and therefore run only when the J4115 starts to heat, so there are mostly silent and only start to really make noise when they are running full speed, even rarer. The little 40x10mm fans on the NICs and 80mm fan for the disk cage are silent with the NA-RC7 reducing their speed.

CONCLUSION: the tower is particularly silent now, beyond expectation. In addition, the overall power consumption of the whole tower when idle went from 50.2W to 42W, which is good news too.

In the next few weeks I will publish (here) the HOWTO for setting up 3 replicated NAS: 3 copies of up to 10TB storage with automatic replication.

Meanwhile: Happy New Vaccine Year!