White wood mini-tower of 3 x H2

Post Reply
domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

White wood mini-tower of 3 x H2

Post by domih »

Little new project: a white wood mini-tower of 3 x H2 with 5-disc cage with which you do not need to disassemble the whole thing to replace a drive. I'll use only 3 hard disks, one for each H2. An ATX SFX PSU will power the 3 x H2 and the disks. It will be used for a replicated NAS and a 3-node Cassandra server. The latter is because any home must have a Cassandra server :-)

See picture below. As the French says "y a plus qu'à"
.
20200504_161821.png
20200504_161821.png (2.9 MiB) Viewed 4663 times
These users thanked the author domih for the post:
ecogit (Fri Jul 24, 2020 1:01 pm)

User avatar
rooted
Posts: 8110
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 727 times
Been thanked: 270 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by rooted »

Nice project, why do you need a distributed database?
These users thanked the author rooted for the post:
domih (Tue May 05, 2020 3:50 pm)

mad_ady
Posts: 8821
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 586 times
Been thanked: 529 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by mad_ady »

rooted wrote:Nice project, why do you need a distributed database?
For the shopping list and cupboard items, what else?
These users thanked the author mad_ady for the post:
domih (Tue May 05, 2020 3:50 pm)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

rooted wrote:
Tue May 05, 2020 10:28 am
Nice project, why do you need a distributed database?
Mostly testing ideas, writing HOWTOs in relation to professional work.

Also I'm really interested in evaluating whether or not the algorithms used to distribute PB of data on thousands of commodity PCs are ALSO useful to distribute storage and data processing on SBCs with limited resources.
These users thanked the author domih for the post:
rooted (Thu May 07, 2020 11:01 pm)

User avatar
rooted
Posts: 8110
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 727 times
Been thanked: 270 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by rooted »

And here I was thinking (because of @mad_ady) you were creating a shopping list for the entire country
These users thanked the author rooted for the post:
domih (Sun May 10, 2020 8:36 am)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update
- Corner braces to assemble shelves with side panels,
- Velcro tape to fix PSU, H2 column and HDD cage to shelves: easy removal for maintenance without fooling around with cramped space for (un)screwing, dampened vibrations for HDD cage and resonance,
- 3.8 x 36" wood to build front mesh frame,
- Plastic back front mesh,
- Loctite to assemble front mesh frame and glue the mesh to it.
20200509_162418.png
20200509_162418.png (2.86 MiB) Viewed 4454 times
These users thanked the author domih for the post:
odroid (Mon May 11, 2020 10:49 am)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

See overall view of the case shown below (build in progress).
.
case_overall_view.png
case_overall_view.png (1.39 MiB) Viewed 4269 times
.
I dropped the usage of corner braces (with the original goal of leaving the sides virgin of any marks), not easy to fix correctly though, result not solid enough. So I went with Philllips pan head 6x1 1/4, three on each side for each shelf. Very solid. I'll mask the screw holes with white putty or plastic round covers.

At the top there is a nice aluminum handle to make the handling/transport of the case easy. The 3 x H2 LED power button will also be on the top. I'll need a 5/8" drill bit, the closest to the 16mm button diameter. Yeah, these buttons are available again. I ordered the 3rd button (plus a few other things) from Ameridroid.com and as usual got it 2 days later with the usual Haribo Goldbears. Kudos to Ameridroid.com! Disclosure: they are not far from the San Francisco Bay Area so I'm a privileged customer in terms of delivery time :-)
.
case_top.png
case_top.png (1.19 MiB) Viewed 4269 times
.
The bottom section of the tower will hold the 5-disk SATA cage (see https://www.ebay.com/sch/i.html?_nkw=Venus+TL+backplane). I'll use only 3 slots for now. 1 x HGST 10TB He10 (HUH721010ALE604) per H2. I bought them used on eBay for about the same price as a new WD Red 10TB. See https://www.ebay.com/sch/i.html?_nkw=HUH721010ALE604.
.
case_bottom.png
case_bottom.png (1.24 MiB) Viewed 4269 times
.
I'm also working on setting up the 3 x H2 and their powering. See picture shown below.
.
psu_3xh2_power_distrib.png
psu_3xh2_power_distrib.png (1.87 MiB) Viewed 4269 times
.
It's basically an expanded version of the setup I used in https://magazine.odroid.com/article/net ... odroid-h2/. Originally, I wanted to go with 2.5 Gbe using USB 3 adapters in order to pass the 1 Gbe limit. But following testing it appears that these USB 3 adapters are having teething issues, see viewtopic.php?f=172&t=38713.

So I revisited the networking solution with the following options: (a) Wait for Hard Kernel in case they would make a new SBC with 1+ Gbe NICs (b) Try 10 Gbe SolarFlare NICs (i.e. the SFN5122F which you can find for $20 or less on eBay, soon they'll give you one free when you buy detergent or toilet paper) with SFP+ DAC cables and a Microtik CRS317-1G-16S+RM switch, overall unbeatable prices compared to 10Base-T RJ-45 solutions - as of this writing (c) Continue with the Mellanox InfiniBand solution. In the Odroid Magazine article I wrote:

.../...
Note: do Mellanox ConnectX-3 cards also work on the ODROID-H2? As of this writing only the Pro version works. The non-pro versions do not. Know anyway that in terms of max speed achievable on the H2, the ConnectX-3 will provide the same result as the ConnectX-2. The ConnectX-2 provide 40 Gb/s in IB mode but only 10 GbE in Ethernet mode. The connectX-3 provide 40 Gb/s in IB mode and 40 GbE in Ethernet mode. However IP over IB provides a speed as good (if not a little bit better) with a port configured for IB as IP on a port configured for Ethernet. So really, no need to run a ConnectX3 Pro on the H2 (unless you already have a 20 or more GbE network running at home based on Ethernet).
.../...

Since then I upgraded my IB switch from a QDR (about $100) to an FDR10 (about $150) and finally an FDR (about $200) all from eBay. As a result from modern PC to modern PC my IPoIB speed jumped from about 32 Gbe, to 38 Gbe and finally 45 Gbe. However, the Connect-XB or Connect X2 (too old) cards I used on the H2 do not work well with the FDR10 or FDR switches. On the other hand, the ConnectX3 Pro on the H2 works 100% OK with these switches, still obviously with the limited speed from 10 Gbe to 14 Gbe depending on direction, which is the best the H2 + Celeron J4105 + PCIe 2.0 x4 can do. So with the ConnectX3 Pro the 3 x H2s can still be part of the IB network with the modern PCs :-) I finally decided to adopt this solution when I found an eBay seller selling used ConnectX3 Pro cards for $45 each. See these cards having their thermal paste re-done in the image shown below.
.
connect_x3_pro.png
connect_x3_pro.png (1.98 MiB) Viewed 4269 times
.
I also found less expensive M.2 NMVe M key to PCIe x4 adapters on eBay, see https://www.ebay.com/itm/PCIe-x4-3-0-PC ... 4398886092. Hopefully they work as well as the ones I originally bought. I'll know in a few weeks, right now they are in China...

Also note the 80mm fans on top on each H2. They are fixed with a small square of Velcro to the heat sinks. No vibration noise. The speed of the fans is set using the small fan board you can see in the picture shown earlier.
These users thanked the author domih for the post:
hominoid (Thu May 28, 2020 10:07 pm)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

See front and back view of the case shown below. Work in progress. The LED power buttons left the top of the case to go down with the 3 x H2 shelf. One more shelf TBD for the PSU + breakout board + fan board.

Note: I dropped using a mesh for the front side of the case. I'll instead use a dark grey or blue acrylic door.
.
case_front.png
case_front.png (1.29 MiB) Viewed 4172 times
.
case_back.png
case_back.png (1.32 MiB) Viewed 4172 times
.
Note the oblong opening at the back of the shelf to allow cables to go through. The PSU shelf will have the same opening.

1 of 3 Connect X3 Pro NIC with custom fan fixed on acrylic sheet. Two bands of Velcro will allow to stick the acrylic sheet to the side of the case. Side and position will depend on the M.2 NMVe M key to PCIe x4 adapter cable which I have not received yet.
.
nic_front.png
nic_front.png (1.8 MiB) Viewed 4172 times
.
nic_back.png
nic_back.png (1.7 MiB) Viewed 4172 times

venkatbo
Posts: 539
Joined: Mon Feb 25, 2013 3:10 pm
languages_spoken: english
ODROIDs: C0/C1+/C2; U3+; XU3-Lite/XU4; H2; VU/VU7+; Touchscreen Show TFT Cloudshell SmartPower mAHRS DAC GPS Multiscope Ocam
Location: Bay Area, California
Has thanked: 2 times
Been thanked: 5 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by venkatbo »

Nice effort!

Wondering if the fans will be effective (with airflow restricted) when the H2's are stacked. Small fans vertically mounted behind the fins may draw more air and cool better and eject the air freely at the back of the stack. Just a thought...

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

domih wrote:
Wed May 27, 2020 2:46 pm
.../...
I also found less expensive M.2 NMVe M key to PCIe x4 adapters on eBay, see https://www.ebay.com/itm/PCIe-x4-3-0-PC ... 4398886092. Hopefully they work as well as the ones I originally bought. I'll know in a few weeks, right now they are in China...
.../...
For the people who are interested by these adapters:

I received the 3 adapters today, so that's relatively quick from CH/HK/TW. I tested one: works AOK. There branded ADT Link. They come with the SATA to 4-pin FDD cable for power as well as the small M.2 screw. In other words, they are ready for use once received, no extra cost. I bought straight 30 cm so that I can pivot the cards 180. The pricing is much less than what I found before: $23.39 (eBay) a piece vs $30.40 or $55.10 (AliExpress). Free shipping.

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

venkatbo wrote:
Mon Jun 01, 2020 10:14 pm
Nice effort!

Wondering if the fans will be effective (with airflow restricted) when the H2's are stacked. Small fans vertically mounted behind the fins may draw more air and cool better and eject the air freely at the back of the stack. Just a thought...
Thank you.

Yes, we'll see at usage because predicting air flow is a tall order. The fans are pushing air to the heat sinks so the air will come out by the sides. The back of the case will stay open. The front will be mesh - I reverted to mesh instead of using a blue acrylic door(*). Given the size of the H2 heat sink, it is supposed to work OK without fan. I'm adding fans because as a rule of thumb they lower the cooled CPU or IC temperature by 20 C and also to prevent #1 cooking #2 cooking #3 by convection. The 80 mm are oversized for the task, allowing me to reduce their RPM for silent operation. The main source of heat coming from the above H2 will be the 32 GB of memory. Finally there is also the 3 Mellanox NICs which will blow hot air from the side. So I have no clue what the final result will be.

(*) I experimented with small pieces of acrylic and I do not trust myself able to pull off a nice door with perfect cuts...

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

The wood work is quasi-complete. See picture below. Screw caps on the side to give a good enough look. Disks cage installed with Velcro taping at the bottom.

Now I'm working on the PSU, PSU break out card and fan card installation, then the front mesh panels. Last to go in will be the 3 x H2 and Mellanox cards.
.
20200606_155320.png
20200606_155320.png (1.28 MiB) Viewed 4043 times
These users thanked the author domih for the post:
odroid (Mon Jun 08, 2020 9:40 am)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

Top compartment front black mesh completed. Still working on the middle compartment one.
.
top-mesh.png
top-mesh.png (1.22 MiB) Viewed 3863 times
These users thanked the author domih for the post (total 2):
odroid (Tue Jun 16, 2020 11:00 am) • Luke.go (Tue Jun 16, 2020 11:46 am)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

This is only of interest to the people who want to know how it's done.

Once the middle compartment black mesh frame is assembled and it corners smoothed, it is time to paint black mat of the 2 sides of the bars that will be visible through the mesh. Acrylic or Enamel paint will do. Acrylic is easier because you clean the brush with warm water once you are done. However I had a small bottle of Enamel handy so I used it. The brush to be cleaned with Enamel thinner. A light layer of paint is enough.
.
paint-it-black-said-mick-1.png
paint-it-black-said-mick-1.png (1.23 MiB) Viewed 3829 times
paint-it-black-2.png
paint-it-black-2.png (1.3 MiB) Viewed 3829 times
paint-it-black-3.png
paint-it-black-3.png (1.3 MiB) Viewed 3829 times
.
Let it dry for some time (1+ hour) and then start to glue the mesh to the 2 unpainted sides of the frame bars, one bar and one side at a time, it's just a question of patience, letting the glue cure for 15 to 30 minutes at each step. Use small clamps a sacrificial piece of wood to keep the mesh and the bar stuck together during curing.
.
mesh.png
mesh.png (1.35 MiB) Viewed 3829 times
.
Tip: how to get the mesh to bend cleanly at 90 degrees along a predetermined line ? It's like in paper modeling, if you want a clean bend of the paper, just trace a line with a fine point pen on the other side of the paper pressing the pen hard enough (but not too hard), then start bending the paper, the bend will exactly occur along the line, clean and perfect. For the mesh which is plastic I used a cutter, pressing it lightly enough to trace the bending line without actually fully cutting the mesh. How to know which pressure to apply: rehearse on a small piece until you obtain the intended result. Then the bend will be clean and good enough.

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Reality Check

<hopeful>Wow! The case looks as good as a commercial case.</hopeful>

Mmm... well it hopefully looks good in the pictures where photography eliminate a lot of little defects, especially because the pictures are 25% reduction of the originals.

The apparent quality is about the same if you look at it from 3 or more feet. The defects start to appear if you look closer and it is obvious this is amateur handmade (with tragi-comic mistakes) and not a work of art.

Just saying so because other forum members who want to build their own case from scratch should NOT feel intimidated to make the plunge. It's fun!

Just temper your expectations of perfection, unless you are a high quality handyman.

mad_ady
Posts: 8821
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 586 times
Been thanked: 529 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by mad_ady »

Don't worry - it looks better than my solution - putting a big sheet of cardboard to hide the devices/cables :)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

For the people tempted in using the H2 or H2+ M.2 NVMe slot as a PCI2.0 x4 slot, I just saw that Ameridroid is now selling two accessories to simplify you life:

- https://ameridroid.com/collections/new/ ... pcie-cards
- https://ameridroid.com/collections/new/ ... r-straight
These users thanked the author domih for the post (total 2):
odroid (Fri Jun 19, 2020 9:40 am) • Ameridroid (Sat Jun 20, 2020 12:38 am)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

The case now has its final look with the middle compartment front mesh completed.

Next step: fix the electronics inside the case.

final-front-view.png
final-front-view.png (1.45 MiB) Viewed 3628 times

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

Initial integration testing without the HDDs

See below the 3xH2's powered on with the 3xMellanox ConnectX-3 Pro network cards and plugged to the Mellanox 18-port FDR10 switch.

3xH2 with Mellanox network cards.png
3xH2 with Mellanox network cards.png (3.62 MiB) Viewed 3514 times
2xPCs+3xH2 plugged into the FDR10 switch.png
2xPCs+3xH2 plugged into the FDR10 switch.png (1.94 MiB) Viewed 3514 times

For a quick introduction about the Mellanox InfiniBand networks, see https://magazine.odroid.com/article/net ... odroid-h2/

Let's check that the IB local net shows all the expected devices (it does):

Code: Select all

domih@h2a:~$ sudo iblinkinfo
CA: h2b mlx4_0:
      0xc4346bffffdc3691      7    1[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>      12    9[  ] "SwitchX -  Mellanox Technologies" ( )
CA: h2c mlx4_0:
      0xc4346bffffdc5561      5    1[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>      12   11[  ] "SwitchX -  Mellanox Technologies" ( )
CA: i5-a mlx4_0:
      0x0002c90300426771      2    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      12    3[  ] "SwitchX -  Mellanox Technologies" ( Could be FDR10 (Found link at QDR but expected speed is FDR10))
CA: ripper HCA-1:
      0xf452140300346ea1      3    1[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>      12    1[  ] "SwitchX -  Mellanox Technologies" ( )
Switch: 0xf452140300789a60 SwitchX -  Mellanox Technologies:
          12    1[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>       3    1[  ] "ripper HCA-1" ( )
          12    2[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    3[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    1[  ] "i5-a mlx4_0" ( Could be FDR10 (Found link at QDR but expected speed is FDR10))
          12    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12    9[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>       7    1[  ] "h2b mlx4_0" ( )
          12   10[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>       1    1[  ] "h2a mlx4_0" ( )
          12   11[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>       5    1[  ] "h2c mlx4_0" ( )
          12   12[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   13[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   14[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   15[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   16[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   17[  ] ==(                Down/ Polling)==>             [  ] "" ( )
          12   18[  ] ==(                Down/ Polling)==>             [  ] "" ( )
CA: h2a mlx4_0:
      0xc4346bffffdc55b1      1    1[  ] ==( 4X  10.0 Gbps (FDR10) Active/  LinkUp)==>      12   10[  ] "SwitchX -  Mellanox Technologies" ( )
Let's check that one of the H2s can ping the i5-a PC via the regular 1 GbE network and via the IPoIB network (it can):

Code: Select all

domih@h2a:~$ ping i5-a -c 10
PING i5-a (192.168.1.20) 56(84) bytes of data.
64 bytes from i5-a (192.168.1.20): icmp_seq=1 ttl=64 time=0.804 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=2 ttl=64 time=0.829 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=3 ttl=64 time=0.825 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=4 ttl=64 time=0.638 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=5 ttl=64 time=0.819 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=6 ttl=64 time=0.827 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=7 ttl=64 time=0.826 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=8 ttl=64 time=0.826 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=9 ttl=64 time=0.829 ms
64 bytes from i5-a (192.168.1.20): icmp_seq=10 ttl=64 time=0.823 ms

--- i5-a ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9037ms
rtt min/avg/max/mdev = 0.638/0.804/0.829/0.055 ms
domih@h2a:~$ ping i5-a.ib -c 10
PING i5-a.ib (10.10.56.20) 56(84) bytes of data.
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=1 ttl=64 time=0.140 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=2 ttl=64 time=0.190 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=3 ttl=64 time=0.179 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=4 ttl=64 time=0.175 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=5 ttl=64 time=0.177 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=6 ttl=64 time=0.177 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=7 ttl=64 time=0.176 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=8 ttl=64 time=0.177 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=9 ttl=64 time=0.178 ms
64 bytes from i5-a.ib (10.10.56.20): icmp_seq=10 ttl=64 time=0.178 ms

--- i5-a.ib ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9207ms
rtt min/avg/max/mdev = 0.140/0.174/0.190/0.012 ms
From there let's perform iperf3 sessions over IPoIB from the 3xH2's toward the i5-a PC:

Code: Select all

domih@h2c:~$ iperf3 -c i5-a.ib --bind 10.10.56.72
Connecting to host i5-a.ib, port 5201
[  4] local 10.10.56.72 port 40529 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   1.00-2.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   2.00-3.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   3.00-4.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   4.00-5.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   5.00-6.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   6.00-7.00   sec  1.31 GBytes  11.3 Gbits/sec    0   3.18 MBytes       
[  4]   7.00-8.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   8.00-9.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  4]   9.00-10.00  sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec                  receiver

iperf Done.
domih@h2c:~$ iperf3 -c i5-a.ib --bind 10.10.56.72 -R
Connecting to host i5-a.ib, port 5201
Reverse mode, remote host i5-a.ib is sending
[  4] local 10.10.56.72 port 49109 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   1.00-2.00   sec  1.57 GBytes  13.5 Gbits/sec                  
[  4]   2.00-3.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   3.00-4.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   4.00-5.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   5.00-6.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   6.00-7.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   7.00-8.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   8.00-9.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  4]   9.00-10.00  sec  1.67 GBytes  14.4 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  16.6 GBytes  14.3 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  16.6 GBytes  14.3 Gbits/sec                  receiver

iperf Done.


domih@h2b:~$ iperf3 -c i5-a.ib --bind 10.10.56.71
Connecting to host i5-a.ib, port 5201
[  5] local 10.10.56.71 port 42363 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   1.00-2.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   2.00-3.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   3.00-4.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   4.00-5.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   5.00-6.00   sec  1.31 GBytes  11.3 Gbits/sec    0   3.18 MBytes       
[  5]   6.00-7.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   7.00-8.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   8.00-9.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   9.00-10.00  sec  1.31 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec                  receiver

iperf Done.
domih@h2b:~$ iperf3 -c i5-a.ib --bind 10.10.56.71 -R
Connecting to host i5-a.ib, port 5201
Reverse mode, remote host i5-a.ib is sending
[  5] local 10.10.56.71 port 46419 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.54 GBytes  13.2 Gbits/sec                  
[  5]   1.00-2.00   sec  1.56 GBytes  13.4 Gbits/sec                  
[  5]   2.00-3.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   3.00-4.00   sec  1.66 GBytes  14.2 Gbits/sec                  
[  5]   4.00-5.00   sec  1.67 GBytes  14.3 Gbits/sec                  
[  5]   5.00-6.00   sec  1.67 GBytes  14.3 Gbits/sec                  
[  5]   6.00-7.00   sec  1.65 GBytes  14.2 Gbits/sec                  
[  5]   7.00-8.00   sec  1.65 GBytes  14.1 Gbits/sec                  
[  5]   8.00-9.00   sec  1.67 GBytes  14.3 Gbits/sec                  
[  5]   9.00-10.00  sec  1.67 GBytes  14.3 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.4 GBytes  14.1 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  16.4 GBytes  14.1 Gbits/sec                  receiver

iperf Done.

domih@h2a:~$ iperf3 -c i5-a.ib --bind 10.10.56.70
Connecting to host i5-a.ib, port 5201
[  5] local 10.10.56.70 port 57781 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   1.00-2.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   2.00-3.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   3.00-4.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   4.00-5.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   5.00-6.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   6.00-7.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   7.00-8.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   8.00-9.00   sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  5]   9.00-10.00  sec  1.31 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  13.1 GBytes  11.2 Gbits/sec                  receiver

iperf Done.
domih@h2a:~$ iperf3 -c i5-a.ib --bind 10.10.56.70 -R
Connecting to host i5-a.ib, port 5201
Reverse mode, remote host i5-a.ib is sending
[  5] local 10.10.56.70 port 48207 connected to 10.10.56.20 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.67 GBytes  14.3 Gbits/sec                  
[  5]   1.00-2.00   sec  1.65 GBytes  14.2 Gbits/sec                  
[  5]   2.00-3.00   sec  1.67 GBytes  14.3 Gbits/sec                  
[  5]   3.00-4.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   4.00-5.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   5.00-6.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   6.00-7.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   7.00-8.00   sec  1.67 GBytes  14.4 Gbits/sec                  
[  5]   8.00-9.00   sec  1.64 GBytes  14.1 Gbits/sec                  
[  5]   9.00-10.00  sec  1.66 GBytes  14.3 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.7 GBytes  14.3 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  16.6 GBytes  14.3 Gbits/sec                  receiver

iperf Done.
As expected we get about 11 Gbits/sec and 14 Gbits/sec depending on direction. In the reverse direction the i5-a PC is running a 9600K which is obviously faster at sending packets than the Celeron J4105.

Let's do iperf3 sessions from H2 to H2 over IPoIB:

Code: Select all

domih@h2c:~$ iperf3 -c h2a.ib --bind 10.10.56.72
Connecting to host h2a.ib, port 5201
[  4] local 10.10.56.72 port 42445 connected to 10.10.56.70 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.26 GBytes  10.8 Gbits/sec    0   3.06 MBytes       
[  4]   1.00-2.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   3.00-4.00   sec  1.30 GBytes  11.1 Gbits/sec    0   3.06 MBytes       
[  4]   4.00-5.00   sec  1.29 GBytes  11.1 Gbits/sec    0   3.06 MBytes       
[  4]   5.00-6.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   6.00-7.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   7.00-8.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   8.00-9.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.06 MBytes       
[  4]   9.00-10.00  sec  1.29 GBytes  11.1 Gbits/sec    0   3.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  12.9 GBytes  11.1 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  12.9 GBytes  11.1 Gbits/sec                  receiver

iperf Done.
domih@h2c:~$ iperf3 -c h2a.ib --bind 10.10.56.72 -R
Connecting to host h2a.ib, port 5201
Reverse mode, remote host h2a.ib is sending
[  4] local 10.10.56.72 port 42283 connected to 10.10.56.70 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.30 GBytes  11.2 Gbits/sec                  
[  4]   1.00-2.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   2.00-3.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   3.00-4.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   4.00-5.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   5.00-6.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   6.00-7.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  4]   7.00-8.00   sec  1.30 GBytes  11.2 Gbits/sec                  
[  4]   8.00-9.00   sec  1.30 GBytes  11.2 Gbits/sec                  
[  4]   9.00-10.00  sec  1.30 GBytes  11.2 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  13.0 GBytes  11.2 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  13.0 GBytes  11.2 Gbits/sec                  receiver

iperf Done.


domih@h2b:~$ iperf3 -c h2a.ib --bind 10.10.56.71
Connecting to host h2a.ib, port 5201
[  5] local 10.10.56.71 port 44087 connected to 10.10.56.70 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.30 GBytes  11.1 Gbits/sec    0   3.18 MBytes       
[  5]   1.00-2.00   sec  1.30 GBytes  11.1 Gbits/sec    0   3.18 MBytes       
[  5]   2.00-3.00   sec  1.30 GBytes  11.1 Gbits/sec    0   3.18 MBytes       
[  5]   3.00-4.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   4.00-5.00   sec  1.30 GBytes  11.2 Gbits/sec    0   3.18 MBytes       
[  5]   5.00-6.00   sec  1.30 GBytes  11.1 Gbits/sec    0   3.18 MBytes       
[  5]   6.00-7.00   sec  1.23 GBytes  10.6 Gbits/sec    0   3.18 MBytes       
[  5]   7.00-8.00   sec  1.24 GBytes  10.6 Gbits/sec    0   3.18 MBytes       
[  5]   8.00-9.00   sec  1.25 GBytes  10.7 Gbits/sec    0   3.18 MBytes       
[  5]   9.00-10.00  sec  1.29 GBytes  11.1 Gbits/sec    0   3.18 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  12.8 GBytes  11.0 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  12.8 GBytes  11.0 Gbits/sec                  receiver

iperf Done.
domih@h2b:~$ iperf3 -c h2a.ib --bind 10.10.56.71 -R
Connecting to host h2a.ib, port 5201
Reverse mode, remote host h2a.ib is sending
[  5] local 10.10.56.71 port 45811 connected to 10.10.56.70 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.30 GBytes  11.1 Gbits/sec                  
[  5]   1.00-2.00   sec  1.25 GBytes  10.7 Gbits/sec                  
[  5]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec                  
[  5]   3.00-4.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  5]   4.00-5.00   sec  1.31 GBytes  11.2 Gbits/sec                  
[  5]   5.00-6.00   sec  1.30 GBytes  11.1 Gbits/sec                  
[  5]   6.00-7.00   sec  1.28 GBytes  11.0 Gbits/sec                  
[  5]   7.00-8.00   sec  1.29 GBytes  11.1 Gbits/sec                  
[  5]   8.00-9.00   sec  1.30 GBytes  11.2 Gbits/sec                  
[  5]   9.00-10.00  sec  1.31 GBytes  11.2 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec  12.9 GBytes  11.1 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  12.9 GBytes  11.1 Gbits/sec                  receiver

iperf Done.
As expected we get around 11 Gbits/sec in both directions.

CONCLUSION
The three H2s behave as expected with IPoIB, at least from the iperf3 point of view.

FINAL REMARK
IPoIB with FDR10 is able to offer about 38 GbE connections with modern PCs. We lose a lot of bandwith on the H2 due to the expected hardware limitations. So does it makes sense to even contemplate such a solution? It sounds like putting a Lotus car body around a moped engine. As a matter of fact, it does. It would be totally stupid/idiotic/crazy to spend thousands of $$$ for brand new Mellanox hardware for an H2.

However, it DOES totally makes sense with the used hardware you can find on eBay:
- 18-port FDR10 Mellanox: got one for $150.
- 36-port FDR Mellanox: got one for $200.
- Mellanox ConnectX-3 Pro cards: got for example 4 cards at $45 each. Other Pro and non Pro cards for similar prices ($45 to $65).
- FDR 56 Gbps IB cables: got for example at once 12 x 2M DAC copper for $100, that's around $8.5 per cable.

The tricks are:
- You have to know what you are looking for.
- You have to be patient for the very good deals.
- You really have to test all the hardware upon reception.

Yes we get "only" 11 GbE with H2. But on the same local net we get around 44+ GbE with FDR with modern PCs (Intel 8/9th gen. or higher, AMD Zen+ or Zen 2).

Compare this to the current RJ-45 new 10 GbE hardware. An 18 or 36-port 10 GbE switch = $$$$, 10 GbE cards are still quite expensive $$$ + CAT6-A cables $$.

Obvious conclusion: so you DO get more while spending less.

So as the French say: "Qu'importe le flacon, pourvu qu'on ait l'ivresse !" which can translate as "What matters the jug, if euphoria be within?"

Next Steps

(1) test and observe differences for Samba, NFS, GlusterFS(?) and Cassandra over:
- 1 GbE
- 2.5 GbE
- SFP+ 10 GbE
- 11+ GbE IPoIB

(2) Finalize case project:
- Put all the electronics in the tower case,
- Enjoy a 3-node server over 11+ GbE :-)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

This is only of interest to the people who want to know how it's done.

Final step of the case building proper: gluing little pieces of wood with Velcro to fix the front mesh panels while keeping them removable for easier future inside maintenance. See pictures below.

Next step: install the boards and PSU inside the case. However it will only happen in a few weeks because I'm going to do testing for 1 GbE, 2.5 GbE, 10 GbE SFP+ and 11+ GbE IPoIB with Samba and NFS, meaning a lot of configuration changes easier to make with the boards out of the case. For IPoIB the testing will be made with the FDR10 switch because I'm still working on customizing the fan set with "silent" Noctua fans instead of the crazy noisy Delta ones on the full FDR switch.

Optional ultimate step for the case: when I'm in the mood I'll try to smooth a few of the seams, fill them with some putty and paint them white insignia to make them look better. As of now I see it as optional. Once the tower is placed at its operational location nobody will look at it closely, maybe the occasional roaming house spider.


1.front-mesh-velcro-attachments-painted-white.png
1.front-mesh-velcro-attachments-painted-white.png (1.66 MiB) Viewed 3349 times
2.gluing-them-to-the-case.png
2.gluing-them-to-the-case.png (1.33 MiB) Viewed 3349 times
3.how-it-looks-finished.png
3.how-it-looks-finished.png (855.51 KiB) Viewed 3349 times
4.velcro-loops-on-the-front-mesh-back.png
4.velcro-loops-on-the-front-mesh-back.png (1.3 MiB) Viewed 3349 times

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

Change of plan: I can proceed with the planed testing with the assembled system, so here we are, see photos shown below. Work in progress, spaghetti cables all over the place. Middle compartment front mesh to be fixed once the LED power buttons are connected. HDDs are finally plugged in.

partially-assembled-front-view.png
partially-assembled-front-view.png (1.51 MiB) Viewed 3216 times
partially-assembled-rear-view.png
partially-assembled-rear-view.png (1.78 MiB) Viewed 3216 times
<humor>
Thank you NZXT, Phanteks, etc(*) for the inspiration but my tower rocks compared to yours :-)

3 x H2 computers for distributed processing
12 cores
96 GB of memory
5 x 3.5" bays
30 TB of disk storage
Enough remaining space for 3 x SATA SSD
192 GB eMMC
1 GbE, 2.5 GbE and 11+ GbE networking
Self-made & 15 μs of fame = priceless

(*) https://www.nzxt.com/product-overview/h1, http://phanteks.com/Enthoo-Evolv-Shift.html.

Long term aspiration: find a the pretext and the adequate SBCs to make something ultra-cool like this one http://www.gamerstorm.com/product/CASES ... 4840.shtml
</humor>

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Ambient temperature: 22C. 3 x H2's running with HDDs, idle: power 38.5W, CPUs temperature: 32 to 34C. H2's off but fans and HDDs still powered: 11.8W. All devices turned off: PSU alone shows 0W at the wall.

I connected the power buttons and fixed the middle compartment mesh. See picture below.

With the the middle compartment mesh installed the CPUs thermal stay the same at idle: 32 to 34C (after 30 minutes). I'm going to let them run idle for a few hours.

led-power-buttons.png
led-power-buttons.png (1.6 MiB) Viewed 3213 times

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

Testing the network cards. Some definitive cable management has to be done before I stick the cards inside the case.

cable-management-needed.png
cable-management-needed.png (1.82 MiB) Viewed 3153 times
I tested running 3 concurrent iperf3 sessions from a PC toward the 3 x H2. Because the 3 connections were going through the same port on the PC side I reached the max FDR10 IPoIB speed of about 38 GbE. So I ended up with slightly more than 12 GbE throughput on each H2. I guess I'll reach 14+ GbE with the FDR switch.

On the PC side the System Monitor showed an outgoing network throughput of 4.7GB/sec :-)

parallel-3x12GbE.png
parallel-3x12GbE.png (1.69 MiB) Viewed 3153 times
These users thanked the author domih for the post:
powerful owl (Mon Jul 06, 2020 12:53 am)

misaz
Posts: 79
Joined: Sat Jul 27, 2019 3:58 am
languages_spoken: english, czech
ODROIDs: H2+
Has thanked: 7 times
Been thanked: 15 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by misaz »

4.7 GB is correct. Note that lowercase b is bits and uppercase B is bytes. Every byte have 8 bits. So if you want to calculate speed in bits per second instead of bytes per second just multiple value by 8.

4.7 GBps = 4.7 * 8 Gbps = 37.6 Gbps.

Alternatively you can use google gadget to convert just type "4.7 GB to Gb" (https://www.google.com/search?q=4.7+GB+to+Gb)

powerful owl
Posts: 172
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 2 x HC1, 3 x H2, 1 x H2+
Has thanked: 43 times
Been thanked: 18 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by powerful owl »

@domih, thanks for the updates :) I need to make something similar...

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

misaz wrote:
Sun Jul 05, 2020 9:25 pm
4.7 GB is correct. Note that lowercase b is bits and uppercase B is bytes. Every byte have 8 bits. So if you want to calculate speed in bits per second instead of bytes per second just multiple value by 8.

4.7 GBps = 4.7 * 8 Gbps = 37.6 Gbps.

Alternatively you can use google gadget to convert just type "4.7 GB to Gb" (https://www.google.com/search?q=4.7+GB+to+Gb)
Yes, it is :)

Also for converting <whatever>bytes to <whatever>bits, it also depends on who or what is counting, meaning using mega=1,000 or mega=1,024. Usually network bandwidth is expressed using mega=1,000. I assume that is what the System Monitor does.

On the other hand, if you were to express the "resulting" bandwidth by measuring the overall time it takes to transfer a file of N x GB in size, then you have to use mega=1,024 to get the right size in bytes (an x8 for bits) and then use mega=1,000 to get the bandwidth in Gb/sec.

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

Testing the network cards but this time the SolarFlare 10G SFP+ instead of the Mellanox ConnectX-3 Pro 56Gbps.

3xH2 with SolarFlare 10G SFP+.png
3xH2 with SolarFlare 10G SFP+.png (1.79 MiB) Viewed 3100 times
Connections to MicroTik Switch.png
Connections to MicroTik Switch.png (1.52 MiB) Viewed 3100 times

Note: The white RJ-45 CAT6-A cable connects a PC with a 10GbE Aquantia onboard NIC. It plugs into the switch via an Ipolex SFP+ <-> RJ45 transceiver.

Tip
In order to work the SolarFlare NIC requires PCI Express Clock Gating to be Disabled and Compliance Mode to be Enabled in the BIOS. Once in the BIOS to go to Chipset then South Cluster Configuration then PCI Express Configuration.

BIOS.png
BIOS.png (1.22 MiB) Viewed 3100 times
As we should expect iperf3 shows a 10Gbe connection between the PC and one of the H2's:

Code: Select all

domih@trx:~$ iperf3 -c h2a.25
Connecting to host h2a.25, port 5201
[  5] local 172.16.25.36 port 40518 connected to 172.16.25.70 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.79 MBytes       
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.88 MBytes       
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.98 MBytes       
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.08 MBytes       
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.08 MBytes       
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.19 MBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.41 Gbits/sec    0   2.19 MBytes       
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.19 MBytes       
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.19 MBytes       
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec    0   2.19 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec    0             sender
[  5]   0.00-10.15  sec  11.0 GBytes  9.27 Gbits/sec                  receiver
Now, because the Aquantia onboard NIC on the PC tops at 10GbE if we run 3 concurrent iperf3 sessions from the PC to each of the 3 x H2's, each session tops at about 10/3 = 3.3 GbE. This is indeed what we see in the screenshot shown below:

3 x H2 10GbE SFP+.png
3 x H2 10GbE SFP+.png (1.78 MiB) Viewed 3100 times
The overall bandwidth as shown in the System Monitor is 1.3 GB/sec (roughly 10Gbe)

So we are obviously far below the 3 x 12.5 (roughly 38 Gb/sec) we got with the Mellanox cards IPoIB. Choose your network solution depending on what you are planing to do with it and... your budget. In the example above, if we were to have one high-performance PC workstation doing multi-thread data processing from data coming from the 3 x H2's it makes quite a lot of sense to prefer the IPoIB Mellanox solution because the overall available network bandwidth is about 4 times larger (even more once the FDR10 switch is replaced with the FDR one). On the other hand, if we had 3 clients each talking to one H2, the difference would be much smaller (11+ to 14+ Gb/sec instead of 9.4 Gb/sec with 10 GbE SFP+ or 10GBaseT).

Given the fact that the cost of used Mellanox hardware is not obscenely higher than the cost of used/new SFP+ hardware (maybe less than twice) I believe that if you chose to go "exotic" instead of 10GBaseT the Mellanox hardware is a better solution. However the Mellanox solution will require you to do some studies (read documentation, knowing what to buy) and customizing (involves soldering) the fan set of the switch (if to be used in an apartment) because these switches are extreme noisy beasts. The currently in progress FDR switch fan set customization is shown below. It consists in opening the box, remove the existing fan, put silent Noctua fans (top panel with not be put back) and soldering the cables (which means retrieving the original cabling online, not on the Mellanox site but on the fans manufacturer site). Not transcendental but requires time and patience.

FDR switch fan set customization.png
FDR switch fan set customization.png (1.89 MiB) Viewed 3100 times
Note: the silent Noctua fans add to the overall cost, so the switch ends up at about $400, but that's a 36-port 56Gb/ps. And you do not need to change the fans if you have an isolated room (e.g. a sound proof attic or garage).

CONCLUSION
Depending on budget and needs you can go 1 GbE, 2.5 GbE, 10GBaseT (RJ-45), 10G SFP+ or 11-14 Gbps with IPoIB. Personally I think Hardkernel did very well in upgrading to 2.5 GbE with the H2+ for basically the same price. Hopefully there will be manufacturers who will make cheap 2.5 GbE switches given the fact that 2.5 GbE chipset are pretty close in price to the 1 GbE chipset. An 8 x 2.5GbE switch should NOT cost $400. That's really taking us the users for suckers.
These users thanked the author domih for the post:
odroid (Tue Jul 14, 2020 9:53 am)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update

The Mellanox network cards are now inside the case in their final location (but using the Velcro allowing for removal without breaking anything).

network-cards-are-in-front-view.png
network-cards-are-in-front-view.png (1.49 MiB) Viewed 2757 times
network-cards-are-in-rear-view.png
network-cards-are-in-rear-view.png (1.77 MiB) Viewed 2757 times
Everything is now connected and up and running. I did proceed with some cable management. See image below. I'll make it better when I'm in the mood. Also there will be 6 less cables (HDMI and USB to KVM) once I stopped using them via the Mate GUI and use SSH instead.

all-connected-up-and-running.png
all-connected-up-and-running.png (1.8 MiB) Viewed 2757 times
Additional Information

Idle power with 3 x H2 + 3 x 10TB disk + 4 x 80mm fans + 3 x ConnectX3 Pro + 3 x 50mm fans + LED Power Buttons : 50.2 W

Thermals

Ambient temperature: 24C

Left idle for 45 minutes

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
34000
34000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
33000
34000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
33000
33000

Left idle for 90 minutes

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
37000
36000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
36000
35000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
35000
35000

Note: h2a is the one at the bottom, h2b the one in the middle, h2c the one at the top.

The whole tower is virtually silent during the day. By night you have to be close to it to hear the fans.

CONCLUSION : The thermals are good when idle. We will see how it goes when the 3 x H2 do heavy compute and disk access.
These users thanked the author domih for the post:
Luke.go (Fri Jul 17, 2020 12:17 pm)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update: New Numbers

Note: no more keyboard and mouse via KVM. No more desktop, booting in multi-user.target. The H2 are now used and accessed via SSH.

Power Consumption

Booting 3 x H2 all at same time, max wattage: 77.4W

Idle power with 3 x H2 + 3 x 10TB disk + 4 x 80mm fans + 3 x ConnectX3 Pro + 3 x 50mm fans + LED Power Buttons : 45.3W

Thermals

Ambient temperature: 26.5C

Left Idle for 10 minutes

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
36000
36000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
35000
35000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
35000
34000

Running iperf3 as server on each H2 @ around 12Gbits/sec

Total wattage: around 67W

After two minutes:

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
52000
50000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
47000
47000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
50000
50000

After 10 minutes:

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
55000
54000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
49000
50000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
52000
54000

After 30 minutes:

domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
56000
57000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
48000
50000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
52000
52000

After 45 minutes:
domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
55000
54000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
50000
50000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
51000
54000

After 55 minutes:
domih@h2a:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
55000
54000

domih@h2b:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
49000
51000

domih@h2c:~$ cat /sys/class/thermal/thermal_zone0/temp && cat /sys/class/thermal/thermal_zone1/temp
51000
52000

CONCLUSION:
- Thermals: from 51C to 57C, we're fine.
- Total segment retries: 2 (h2a) + 2 (h2b) + 5 (h2c)
- Average throughput: 11.4 Gbits/sec (h2a), 11.8 Gbits/sec (h2b), 11.7 Gbits/sec (h2c) Note: should have been 14+ but the 3 sessions ran from the same PC so we topped at the FDR10 limit.
- Total data transferred: 4.67 TBytes (h2a), 4.83 TBytes (h2b), 4.80 TBytes (h2c)


Next (upcoming in the next few weeks)
- Same thing in reverse (will be much more taxing on the H2/H2+ cores).
- Running Cassandra stress tool (isolated) on each of the 3 x H2 (for power and thermal analysis; should be even more taxing and pushing all cores to 100%).
- NFS async testing.
- Samba sync and async testing.
- Running Cassandra stress tool in distributed mode (including 1Gbe vs. 11+GbE IPoIB comparison). I will do only do 2.5Gbe when there are 100% working USB 3 adapter because the tower now includes 2 x H2 and one H2+.
- Possibly glusterfs for 1Gbe vs. 2.5Gbe vs. 11+GbE IPoIB comparison (but I need to refresh my knowledge about it).

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Update
Testing the Cassandra Stress tool running (isolated) on the 3 x H2. The main purpose being to measure power and thermals while the 3 x 4 x cores are at >85 to 100%(*).

Power
Write test: variable from 68W to 90W
Read test: pretty stable at 80W

Thermals
Write & Read: 54C to 67C (Ambient 27C)

(*) Using htop to confirm all cores are pushed to >85 to 100% (if htop is to be trusted). Note: screenshot shows high 80's but I saw a lot of 100% during the several minutes test.

htop-cass-h2a-h2b-h2c-100.png
htop-cass-h2a-h2b-h2c-100.png (851.61 KiB) Viewed 2425 times
For the Cassandra Stress results (which were not the main purpose) there was a little surprise in store:

cassandra-stress-h2a-h2b-h2c.png
cassandra-stress-h2a-h2b-h2c.png (108.28 KiB) Viewed 2425 times
h2a, h2b, h2c (identical hardware and software, except h2a is a H2+ while b and c are H2's) were pretty much within 5% of each other during the write test.
On the other hand h2c decided to act up way above h2 a and h2b during the read test. Not sure why, I'll have to check the configuration.

CONCLUSION: we are OK in terms of power and thermals. We can move forward to the other tests.
These users thanked the author domih for the post (total 2):
hominoid (Sun Jul 26, 2020 10:55 pm) • odroid (Mon Jul 27, 2020 11:34 am)

rupy
Posts: 48
Joined: Fri Mar 04, 2016 7:38 am
languages_spoken: english
Has thanked: 0
Been thanked: 0
Contact:

Re: White wood mini-tower of 3 x H2

Post by rupy »

@domih Neat, you could run my software instead to get better performance: https://github.com/tinspin/rupy

Here is my latest cluster that will soon be upgraded with RPi 4 with custom heatsink:

http://sprout.rupy.se/article?id=290
http://move.rupy.se/file/pi_4.jpg

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

Samba Testing (Part I)

Client PC
AMD Threadripper 3960X

NAS PC 1...: AMD V1605B, 32GB @ 2400MT/s, WD Red 10TB
NAS PC 2...: AMD R5 3400G, 32GB mem @ 3200MT/s, WD Red 10TB
NAS PC 3...: Odroid H2, Intel Celeron J4105, 32GB @ 2400MT/s, HSGT He10
NAS PC 4...: Odroid H2+, Intel Celeron J4115, 32GB @ 2400MT/s, HSGT He10

Notes
1. On each NAS PC the hard disk was already about 25% full.
2. The Odroid H2 was used for the 1 GbE and IPoIB tests. The Odroid H2+ was used for the 2.5 GbE tests.
3. I included two low mid-range PC for context.
4. All machines running Ubuntu 18.04 or 20.04.

Tests
- Copy a 1 GB file (uploading)
- Copy a 2 GB file (uploading)
- Copy a 4 GB file (uploading)
- Copy a folder of 500 files (100 of 1, 2, 3, 4 and 5 MB) for a total of 1.6 GB.

The files were created this way:

Code: Select all

#!/bin/bash
for i in {1..100}
do
   dd if=/dev/urandom of=./test_dir/"$i 1MB.txt" count=1 bs=1048576
   dd if=/dev/urandom of=./test_dir/"$i 2MB.txt" count=2 bs=1048576
   dd if=/dev/urandom of=./test_dir/"$i 3MB.txt" count=3 bs=1048576
   dd if=/dev/urandom of=./test_dir/"$i 4MB.txt" count=4 bs=1048576
   dd if=/dev/urandom of=./test_dir/"$i 5MB.txt" count=5 bs=1048576
done

dd if=/dev/urandom of=~/Downloads/1GB.txt count=1024 bs=1048576
dd if=/dev/urandom of=~/Downloads/2GB.txt count=2048 bs=1048576
dd if=/dev/urandom of=~/Downloads/4GB.txt count=4096 bs=1048576
The measurements were made this way:

Code: Select all

# 1, 2 and 4 GB file
sudo smbclient '//xxx/xxx/' xxxxxxxx -U domih  -c "prompt OFF;rm 1GB-hdd-2.5GbE.txt" ; \
time sudo smbclient '//xxx/xxx/' xxxxxxxx -U domih  -c "put /mnt/raid/test/1GB.txt 1GB-hdd-2.5GbE.txt"

# Folder of 500 files
sudo smbclient '//xxx/xxx/' xxxxxxxx -U domih  -c "prompt OFF;deltree test_dir_1.6_GB_500_files_2.5GbE" ; \
sleep 30 ; \
time sudo smbclient '//h2a.25/h2a/' dev.2019 -U domih  -c "prompt OFF;recurse ON;mkdir test_dir_1.6_GB_500_files_2.5GbE; cd test_dir_1.6_GB_500_files_2.5GbE; lcd /mnt/raid/test/test_dir_1.6_GB_500_files;mput *"
iperf3 Testing
Let's do iperf3 tests to get a view of the landscape:

iperf3-results.png
iperf3-results.png (45.95 KiB) Viewed 1681 times

Samba Testing

Note about Samba testing: in order to eliminate variations due from caching I always run the a 1 GbE test once or twice before registering the next 6 results, no need for the 2.5 GbE tests which always follow the 1 GbE tests. I also eliminate outliers (e.g. variations due to background execution like CRON).

1GB.png
1GB.png (71.33 KiB) Viewed 1681 times
2GB.png
2GB.png (71.49 KiB) Viewed 1681 times
4GB.png
4GB.png (71.37 KiB) Viewed 1681 times
folder.png
folder.png (75 KiB) Viewed 1681 times

CONCLUSIONS

1. Note that the 1 GbE results are basically the same across the 4 NAS PC! So yes you can run a NAS with an H2 at 1 GbE for home or SOHO usage, it will do as good and consume less over the years.

2. The Odroid H2+ with 2.5 GbE being 2.464, 2.487, 2.485 times faster than 1 GbE for the 1, 2 and 4 GB file copy (pure sequential writing) is extremely close to the 2.5 times faster from iperf3. Only for the 500 files folder the ratio falls to 1.7 faster. 2.5 GbE is not as demanding at 10+ GbE for the internal components (e.g. disk IO, power of the CPU, etc). It is therefore probably that 2.5 GbE with a more powerful PC would provide similar performance. So yes you can run a NAS with an H2+ at 2.5 GbE for home or SOHO usage, it will do as good and consume less over the years.

3. For IPoIB, the internal "oomph" of the PC greatly affects the throughput. At iperf3 34.4 Gbits/sec the 3400G is approaching the expected practical maximal throughput of IPoIB over FDR10. At around 20.0 Gbits/sec the V1605B already reaches its limit (the CPU was made with 10 GbE). At 11 Gbits/sec the Odroid H2's Celeron J4105 does as best as it can. Consequently it is no wonder that IPoIB does not report ratios for Samba tasks as high as iperf3 ratios. This where you can see the difference between Enterprise/Data Center and Desktop/Consumer hardware. With the former, the internal components (e.g. beefy CPUs, offloading disk IO to RAID components) would be much more able to face the throughput of the network. It's just a question of $$$.
Last edited by domih on Sun Sep 20, 2020 3:00 pm, edited 2 times in total.

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

To (partially) illustrate point 3 of the conclusions in the previous post, I proceeded with a test with the TRX as Samba server with a share located on a 2 x NVMe RAID 0 (using mdadm). Here are the results:

trx.png
trx.png (85.56 KiB) Viewed 1665 times

If we compare the throughput ratios, we can see that the TRX samba server flies high above the other servers:

ratio-summary.png
ratio-summary.png (78.25 KiB) Viewed 1665 times

Note: note the quirk on copying the 500 files folder (which I have not looked into yet).

So if you really need an ultra-fast beefy NAS and IF money and energy consumption are not an issue, use:
- AMD EPYC (for the 128 PCIe 4 lanes, 8 channels memory)
- A beefy RAID card (a real one with real offloading of the RAID functions from the CPU to the card)
- 16 or 24 SAS 12Gbps (or 24 Gbps) SSD configured in RAID 10
- 100GbE NICs.

Then you'll have at home what they use in Enterprise or Data Centers. If it is still not enough, visit the site of the main players of this field (tip: find a Gartner report for the names, https://www.bing.com/images/search?q=en ... er&first=1).

Landing back on Earth :-) : For Home and SOHO, an H2+ based NAS over 2.5 GbE works pretty well.

If 2.5 GbE is not enough, you can scale up your H2 or H2+ with 10 GbE RJ-45 or SFP+ or ConnectX 3 Pro (with IPoIB and/or RDMA).

But do not forget that the numbers returned by iperf3, while real, are also theoretical once you involve the other components of the system (CPU, Memory, Disk IO).
These users thanked the author domih for the post:
odroid (Mon Sep 07, 2020 9:45 am)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

To complete the Copy (upload) Samba testing, here are the SFP+ tests.

iperf3

iperf3-with-sfp+.png
iperf3-with-sfp+.png (35.75 KiB) Viewed 1583 times

Samba

samba-sfp+.png
samba-sfp+.png (91.29 KiB) Viewed 1583 times

Optimizations (ratios) over 1 GbE


ratios-comparison.png
ratios-comparison.png (95.76 KiB) Viewed 1583 times

Note how the 10 SFP+ and IPoIB are basically identical within margins of error for copying the 1, 2 and 4 GB files.

The SFP+ acceleration only starts to collapse in comparison to that of IPoIB when copying the 500 files folder.

I suspect an 10 GbE RJ-45 NIC would show the same behavior.

CONCLUSION (mostly a repeat of what I posted in this or other threads)
1. If you want your H2+(s) and other computers to talk 10 GbE, you can go the RJ-45 or SFP+ way, the latter being significantly less expensive (as of this writing) with eBay used hardware in comparison to the new 10 GbE consumer hardware.
2. It only makes sense to use IPoIB if you already had the infrastructure and want to insert your H2+(s) into it or if you plan to use applications leveraging RDMA (e.g. NFS). InfiniBand eBay used hardware is more expensive than SFP+, but still less expensive than the new 10 GbE consumer hardware.
These users thanked the author domih for the post:
odroid (Tue Sep 08, 2020 1:16 pm)

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

For similar Samba testing results using an Odroid N2+ see viewtopic.php?p=305819#p305819.

And for the Odroid C4 see: viewtopic.php?p=306645#p306645

domih
Posts: 327
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 112 times
Been thanked: 121 times
Contact:

Re: White wood mini-tower of 3 x H2

Post by domih »

[ reserved ]

I'm done testing the N2+ and C4 with iperf3 and Samba upload and download over 1 and 2.5 GbE. For the H2/H2+, I still need to perform the Samba download tests over 1 and 2.5 GbE as well as 10G SFP+ and IPoIB. I'm currently replacing the 2 x H2 in the white wood mini-tower with 2 x H2+. I'll perform the remaining H2/H2+ Samba tests once I'm done with the overhaul of the tower.

[ reserved ]

Post Reply

Return to “Projects”

Who is online

Users browsing this forum: No registered users and 2 guests