Micro-server

Share here your ideas for new projects
exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Micro-server

Post by exilibris »

Hello.

My name is Sergey, I am an IT specialist in Russia. In our country, and all over the world, there is a great demand for micro-servers.

I have a question. Does your company have the resources to develop (I'm sure) a very popular product - a micro-server: a processor of the Rockchip 3399 level or higher, and at least 4 GB of RAM (for popular DBs and micro-services), 1 gigabit network port, 1 USB for peripherals (keyboard). It's all. In a format similar to MC-solo. The retail price is about $ 30-40.

Unfortunately, there are no such systems on the market. I studied the entire market and all manufacturers.

Best regards, Sergey.

User avatar
odroid
Site Admin
Posts: 37371
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1785 times
Been thanked: 1138 times
Contact:

Re: Micro-server

Post by odroid »

If you can sell it at <$50, we will buy a few thousands units right now. ;)

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

even if the economy does not allow implementing such a solution at a low price, is it possible to implement a micro-server of the MC-solo level with 4GB RAM and Allwinner H6 level cpu (without unnecessary wiring of peripheral ports and GPIOs)?

I mean literally only cpu + ram + RJ45 (Gigabit) in as compact form factor as possible

User avatar
odroid
Site Admin
Posts: 37371
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1785 times
Been thanked: 1138 times
Contact:

Re: Micro-server

Post by odroid »

Theoretically possible and you can save a couple of dollars if you remove some IO ports. But, the market size must be much smaller and it seems to be a main reason why you couldn't find a similar product.

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

Thanks for your reply. My main idea is to organize the maximum density of computations per volume unit, while ensuring the maximum degree of hardware diversification - the failure of one unit does not have a significant impact on the cluster as a whole.

I believe this opportunity is deeply underestimated by the market.

Unfortunately, systems of this kind are practically impossible to implement on existing solutions precisely because of their redundancy (hdmi, many usb, gpio, and other ports that are not necessary for computing units).

fvolk
Posts: 606
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 72 times
Contact:

Re: Micro-server

Post by fvolk »

You seem to want to build a cluster with many nodes/cores,
see also viewtopic.php?f=54&t=41467

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

Everything is correct. Unfortunately, there is no sufficiently compact solution on the market that is devoid of everything superfluous.

I'll explain.

The I / O ports and peripherals hinder the implementation of a compact design of the cluster system, since they do not allow effectively implementing the chip cooling system. We calculated computational density as the ratio of compute power to volume, taking into account diversification.

At the moment, I actively use a set from MC-solo to simulate the solution (as a laboratory setup).

Now I am actively looking for a vendor who could implement a compact solution, similar to solutions with the designation "core" or "zero" (as in FriendlyArm), but with a soldered RJ-45.

Most likely, vendors will not implement solutions of this format in the next few years with sufficient RAM for DB operation, since 256 and 512 MB seem to be an extremely insufficient solution. The working minimum for controlling the amount of data is 4 GB of RAM.

I am deeply convinced that the release of "zero" solutions in FriendlyArm terminology will allow us to implement microclusters at a completely new level.

I mean the most dense packaging of solutions using heat pipes in the volume of a 1U server.

fvolk
Posts: 606
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 72 times
Contact:

Re: Micro-server

Post by fvolk »

There are already multiple-ARM 1U servers available, e.g.
http://wiki.t-firefly.com/en/Cluster-Se ... arted.html
...why not one of those?

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

From SPEC:
>> CPU up to 11pcs RK3399 six-core 64-bit core
board, with 1.8GHz processor performance
of up to 66 cores.

---

According to my conservative estimates, about 300 "Zero" format SBCs can be effectively accommodated in a 1U server, which gives about 1200 cores vs 66.

This is the main reason.

fvolk
Posts: 606
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 72 times
Contact:

Re: Micro-server

Post by fvolk »

There is a newer one coming out: https://www.cnx-software.com/2021/01/28 ... 3328-soms/
72*6=432 cores in 2U

And you want 300 nodes in a small space and with a RJ45 plug, so you would have 300 network cables plus required network switches for that and the resulting node traffic on 1Gbit...? Professional solutions (see above) have a custom backplane for that...

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

Yes that's right. For a certain kind of tasks, such a configuration is optimal, relative to any solutions existing in the market.

According to my calculations, such a configuration is more efficient than AWS solutions based on "computational density per volume unit of space" by ~ 317 times.

My goal is to reduce the cost of ownership of infrastructure (in general) and in IT in particular by three orders of magnitude (1000+ times).

neildarlow
Posts: 1
Joined: Tue Feb 09, 2021 4:14 am
languages_spoken: english
ODROIDs: HC1
Has thanked: 0
Been thanked: 0
Contact:

Re: Micro-server

Post by neildarlow »

I have just built something similar with the ODROID HC1.

Hardware: HC1 with 240GB SSD, 32GB Boot microSD, TP-Link UE300 USB/Ethernet adaptor and CR-2032 RTC battery.

Software: archlinuxARM (kernel 4.14.180 - would prefer later version) host running systemd-nspawn containers configured with:

HOST - Shorewall (firewall/routing), Dnsmasq (DHCP server/DNS cache), Chrony (NTP server)
DBMS - MariaDB and PostgreSQL
HTTP - Lighttpd serving static websites
MAIL - Postfix (SMTP) and Dovecot (IMAP) with ClamAV, OpenDKIM and OpenDMARC
XMPP - Prosody with Coturn (STUN/TURN server) on host OS

The SSD is formatted with BTRFS so the containers may have snapshots taken before updates are made (allows for rollback).

The extra ethernet adaptor provided by the TP-Link UE300 (same chipset as onboard ethernet) permits a WAN/LOCAL LAN setup with all routing being performed through Shorewall (SNAT and DNAT).

I'm really pleased with the peformance and functionality of this setup. I hope that Hardkernel don't discontinue the HC1 any time soon because I can see me building more of these.

ATB,
Neil Darlow

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

It's very interesting solution.

I hope Hardkernel listens and appreciates the prospects for the development of the series HC1 with minor modifications:

1. minimization of SBC size as possible - something similar to ZeroPi form-factor (or smaller) by FriendlyArm
2. 4 Gb RAM or more
3. 64-bit CPU 4 cores or more (it can be AllWinner or Rockchip like cpu)
4. strongly optional their cooling radiator (i'll buy w/o aluminium radiator)
5. processor and memory on one side of the board (for easy installation of heat pipes and heatsink)

this is all that is needed for happiness and building huge supercluster systems!!!

powerful owl
Posts: 209
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 3 x H2, 1 x H2+
Has thanked: 65 times
Been thanked: 23 times
Contact:

Re: Micro-server

Post by powerful owl »

I think ODroid is saying there isn't the market to justify the development cost.

User avatar
odroid
Site Admin
Posts: 37371
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1785 times
Been thanked: 1138 times
Contact:

Re: Micro-server

Post by odroid »

neildarlow wrote:
Tue Feb 09, 2021 10:33 pm
I hope that Hardkernel don't discontinue the HC1 any time soon because I can see me building more of these.
The HC1 will be discontinued soon due to very low demands. Sorry about that.
But, The HC2 will be available one or two more years probably.

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

How much will it cost to develop the proposed solution in a Zero Pi form factor?

We can try to raise the required amount with the help of the community on kickstarter and similar sites.

What marginality ratio is acceptable for a company to understand how much it needs to raise? What is the multiplicity? 2x - 5x of the cost price?

P.S.
It seems to me that you underestimate the market for "in-house" cluster systems.

User avatar
odroid
Site Admin
Posts: 37371
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1785 times
Been thanked: 1138 times
Contact:

Re: Micro-server

Post by odroid »

Why don't you buy the firefly's products?
Their price looks quite cheaper than my expectation.

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

Their products do not fit the form factor.

The fact is that, from their point of view, a dense arrangement of computing units is possible only in their proprietary solutions, and this resembles the situation with solutions cluster based on the Raspberry Pi Zero.

Is it possible in principle to build a system on a proprietary basis - yes, of course it is.

Will it be convenient and will it allow to diversify the computing nodes in the network so that they are completely autonomous - definitely not!

This is the reason why a set of self-sufficient solutions is fundamentally required - that is, in the literal sense of the word, self-sufficient separate boards as computational units of the HC-1 / Zero Pi form factor.

This is the only condition under which it is possible to build unrestricted cluster systems with such a diversification of risks, in which the boards themselves become "consumables".

In solutions of the level of "ready-made systems", it is impossible to implement the required computational density. R1 level systems simply fundamentally do not allow this, since they were designed taking into account a certain (very, very low computation density).

66 cores in a 1U form factor is incredibly low density. The 64 + 4 core cluster is right now on my desk as a mini-lab, and it takes up space about two cups of coffee.

The market simply does not have a product (or idea of ​​a product) that can be created on the basis of such mini-bricks, like Lego bricks, offering the same type of solutions for media centers and DIY projects like weather stations everywhere - that is, in fact, all these solutions are no more than toys in the hands of geeks.

In reality, these solutions have great commercial sense and potential in processing transactions of e-commerce solutions !!!

User avatar
odroid
Site Admin
Posts: 37371
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1785 times
Been thanked: 1138 times
Contact:

Re: Micro-server

Post by odroid »

I meant 72*6=432 cores + 288GB RAM in 2U at $10000 is quite cheap.

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

I need something like that, but with 4-cores (or more) + 4 Gb RAM or more nodes, and that's all to be lower TCO cost of IT infrastructure (taking into account the software of course) about 1000+ times.

Image

powerful owl
Posts: 209
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 3 x H2, 1 x H2+
Has thanked: 65 times
Been thanked: 23 times
Contact:

Re: Micro-server

Post by powerful owl »

That's a lot of zeroes :lol:

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

powerful owl wrote:
Wed Feb 10, 2021 11:41 am
That's a lot of zeroes :lol:
In the picture, yes, there are a lot of Raspberry Pi Zero's (which is sad), and if the solution contained, say, many improved Nanopi Zero or HC-1 or MC-1 - that would be the perfect solution!

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

odroid wrote:
Wed Feb 10, 2021 11:26 am
I meant 72*6=432 cores + 288GB RAM in 2U at $10000 is quite cheap.
I found on their website the cost of the solution at $ 15,800 for R2, which is a lot with a relatively low computation density per volume (since the solution is 2U).

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

it seems strange to me that manufacturers do not release SoC-level solutions in one package with a cpu + ram sandwich, despite the fact that the market is saturated with solutions with a certain amount of memory (1-2 GB), this would be an excellent solution for building boards that would not be produced for a specific processor, but for a family of processors that would not be soldered to a board, but were a module, like memory chips in the good old days (or like modern CPUs for sockets), since the ports always remains the same.

Something similar (abstraction from the periphery), as far as I understand, was implemented by the Raspberry Pi with their Compute Module 4.

Image

User avatar
odroid
Site Admin
Posts: 37371
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1785 times
Been thanked: 1138 times
Contact:

Re: Micro-server

Post by odroid »

exilibris wrote:
Wed Feb 10, 2021 12:01 pm
odroid wrote:
Wed Feb 10, 2021 11:26 am
I meant 72*6=432 cores + 288GB RAM in 2U at $10000 is quite cheap.
I found on their website the cost of the solution at $ 15,800 for R2, which is a lot with a relatively low computation density per volume (since the solution is 2U).
It is still very cheap.
If you calculate the total power consumption and the heat dissipation, you can't increase the density.

fvolk
Posts: 606
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 72 times
Contact:

Re: Micro-server

Post by fvolk »

odroid wrote:
Wed Feb 10, 2021 10:19 am
The HC1 will be discontinued soon due to very low demands. Sorry about that.
I ran HC1 as micro servers for some time.

In practical use I stopped using them, because:
they were the only 32bit platforms - so always extra work for software installation/upgrades,
they never ran stable at full CPU load - had to limit all cores to 1.5GHz,
no CPU crypto acceleration - encrypted disk performance was limited,
did seem too much work to get an upstream kernel set up
...all together too much headache.
So I sold them, sorry about that.
But I liked the form factor :-)

I also ran a MC1 for early distributed computing experiments for some time,
the same as above applies but with the additional headache of supplying power, and to the fan, and never got diskless boot to work.
So I stopped also with the MC1 and build a C2/C4 cluster instead.
Less work to setup, more productive use.

IMHO

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

they never ran stable at full CPU load - had to limit all cores to 1.5GHz
I also had problems with stability under high load. Even running the "7z b" benchmark could crash the system.

By the way, I use 8 MC-solo for the test, and the problems were stable with 4 of them.

But I found a solution.

The matter turned out to be extremely high sensitivity to voltage drops when the processor current consumption increased, which occurred on the contacts of the DC 5.5 x 2.5 power connector, I began to use copper wires of a larger cross section with a good contact on the power plug. And the problems disappeared. I believe that I have a sufficiently representative sample of 8 systems to argue that it is a matter of changing the power wires. At the same time, the boards have places for soldering + and - power wires from 12v.

If in this form factor ODroid were to replace the CPU with a 64-bit one and add memory (4 Gb), and also make the cooling system optional, then this would be the best solution on the market for building "in-house" cluster systems today!

fvolk
Posts: 606
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 72 times
Contact:

Re: Micro-server

Post by fvolk »

I tried first with MC1 plus HC1 as storage node. Looks nice on photo, but not on photo are the 5 separate PSUs and cabling required - a mess :-(
The C4s are more cluster friendly: Variable input V from PSU, Y-cables cheap off the shelf, 4GB per node, 64bit CPU, crypto acceleration.
Only annoying thing with C4 cluster is that the power plug is at the side and so the stack uses quite some extra space.
Stacking Odroids in clusters and comparing performance is a weird hobby... ;-) viewtopic.php?p=299192#p299192

If the C4 would be put into a HC1 form factor the only thing missing would be an on-board RTC chip by default to make deployment as data collection device or micro server "somewhere" easier.
clustermc1.jpg
clustermc1.jpg (67.99 KiB) Viewed 1277 times
clusterc4.jpg
clusterc4.jpg (69.03 KiB) Viewed 1277 times

powerful owl
Posts: 209
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 3 x H2, 1 x H2+
Has thanked: 65 times
Been thanked: 23 times
Contact:

Re: Micro-server

Post by powerful owl »

fvolk wrote:
Thu Feb 11, 2021 8:34 am
Only annoying thing with C4 cluster is that the power plug is at the side and so the stack uses quite some extra space.
You could wire directly into the board though - ? -> https://wiki.odroid.com/odroid-c4/appli ... ower_input
(Don't have any myself, but your post is intriguing....)

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

This is evolution of my experiments with MC-solo cluster config look (it was fun)

1. simple stack (+ with fan)

Image
Image
Image
Image

2. Beautiful stack (with quiet fans) ^_^

Image
Image
Image

3. Extended stack with fan voltage control

Image
Image
Image
Image
Image

4. Horizontal improved stack

Image

5. Compact stack with blackwood frame

Image
Image
Image
Image
Image
Image
Image
Image

User avatar
rooted
Posts: 8512
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 740 times
Been thanked: 345 times
Contact:

Re: Micro-server

Post by rooted »

What are you running on those and how well do they perform?

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

I design mosaic clusters (that's what I call this kind of system). They are a type of load-adaptive hyperconverged system. Unfortunately, systems of this kind are ineffective to build according to classical schemes due to their consolidation and low computation density per unit volume.

I use several similar systems that are located in different locations, connected using a global state (this is where tasks go). Tasks are generated spontaneously according to the situation, I cannot predict the moments in time, so the structure of the task flow is unique in some way.

Tasks include image processing tasks, parsing commercial data (price lists), and pre-rendering static resource assemblies.

Depending on the load, the mosaic cluster, like a mosaic, reconfigures the working nodes in such a way as to optimally occupy the computing resources of the network. To do this, the nodes are networked, but not among themselves, but thanks to the mosaic network nodes that maintain a global state.

That is, I am building a certain global state, depending on the configuration of which the worker nodes are loaded in such a way as to ensure the minimum delay in the execution of tasks and optimal parallelism at the same time (that is, the efficiency automatically tends to the maximum). In the case when there are a minimum of working tasks, the nodes go into the mode of minimum consumption (dropping the frequency as much as possible).

This is the result of a ratio in solving a system of equations with an optimization criterion based on the cost of computational density per unit volume.

Therefore, classical solutions are not suitable. I changed the architecture of the system and deployment through pre-preparation (a kind of build stage) and pre-rendering, for the further delivery of the actually hot cache to static servers with a gigabit channel and further to the CDN.

I tried to explain the general concept in as simple terms as possible.

User avatar
mad_ady
Posts: 9405
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 599 times
Been thanked: 666 times
Contact:

Re: Micro-server

Post by mad_ady »

@exilibris let me be the first to say that I think you have a pretty nice rack... (ba-dum-tsss)
:)

User avatar
rooted
Posts: 8512
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 740 times
Been thanked: 345 times
Contact:

Re: Micro-server

Post by rooted »


exilibris wrote: I tried to explain the general concept in as simple terms as possible.
More than adequate response, thank you.

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

mad_ady wrote:
Fri Feb 12, 2021 3:41 pm
@exilibris let me be the first to say that I think you have a pretty nice rack... (ba-dum-tsss)
:)
Thank u! ;)

P.S.

I have also experimented with immersion cooling in an aquarium with oil (dielectric). Unfortunately, I have no photo left, but this cooling method makes sense only under two conditions:

1. Nodes are always almost completely loaded with work (~ 99% of the time).

2. There is a pump and an efficient heat exchange circuit, or a radiator with blowing, or just a circuit with cold water in contact with the oil circuit. Heat must be dissipated outside the system.

Then the system will be super-compact, that is, this is the condition for the most dense layout, of all that are possible, in my opinion, for the organization of super-dense and ultra-cheap calculations per unit of work performed.

I suppose most people do not understand the commercial potential of the venture, but in reality it allows you to influence the market (on the median prices, dropping them).

I don't really do this for fun or a hobby. I am trying from the IT side to make a liquidity accelerator through hardware and software.

fvolk
Posts: 606
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 72 times
Contact:

Re: Micro-server

Post by fvolk »

powerful owl wrote:
Wed Feb 10, 2021 10:04 am
I think ODroid is saying there isn't the market to justify the development cost.
I thought about that - and prototyped how to build a micro cluster from existing parts and minimal effort.
See photos, tried with 3xC2 and 3xC4.
c2c4front.jpg
c2c4front.jpg (57.48 KiB) Viewed 1185 times
c2c4top.jpg
c2c4top.jpg (73.66 KiB) Viewed 1185 times
c2c4side.jpg
c2c4side.jpg (49.97 KiB) Viewed 1185 times
ray.jpg
ray.jpg (84.26 KiB) Viewed 1185 times
As base a 8-port network switch, 15.5x9x2.5cm, robust metal case.
2 stacks of 3 boards, a stack is 9.5cm high.
If these were all C4s headless compute nodes, they would be able to run from one standard 15V/4A PSU, with a proper 6x Y power cable, with 90° angled plugs.
6x short+thin network cables (ok, these slim cables are kinda bad in quality)
2 network ports still free, one for a storage node (HC1 would be about the same size as switch) and another for the network uplink.

Hmm... so this "C4 clustering upgrade kit" would be a nice frame, a bag of spacers, a 6x Y power cable, 6x slim network cables
Standard C4s and PSU from Odroid,
network switch supplied by user.
Sounds... doable? :-)
No need to buy something fancier? (e.g. https://www.picocluster.com/)
Minimal volume and less cables than the MC1, no noisy fan needed if open frame design?

Just got creative while watching a boring stream... ;-)

powerful owl
Posts: 209
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 3 x H2, 1 x H2+
Has thanked: 65 times
Been thanked: 23 times
Contact:

Re: Micro-server

Post by powerful owl »

fvolk wrote:
Sat Feb 13, 2021 4:36 am
powerful owl wrote:
Wed Feb 10, 2021 10:04 am
I think ODroid is saying there isn't the market to justify the development cost.
I thought about that - and prototyped how to build a micro cluster from existing parts and minimal effort.
...
Minimal volume and less cables than the MC1, no noisy fan needed if open frame design?
Hi, that's cool and intriguing although doesn't meet exilibris requirements. I'm not in this business but I gather that once you remove the "unwanted" ports and replace the heatsink with a slug for custom cooling and higher density, it becomes a niche product with different economics. The Firefly linked earlier is USD220 per node, I assume that includes networking and power, compare to your "low density" stack at 50 per node + network and power. Actually I'm not sure why high density is so important, you would have to be making a very large cluster for it to matter I suspect and then you get into other issues like weight and how much power you can supply to a rack etc.

All that aside :) what software are you running on your cluster?

powerful owl
Posts: 209
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 3 x H2, 1 x H2+
Has thanked: 65 times
Been thanked: 23 times
Contact:

Re: Micro-server

Post by powerful owl »

I'm just wondering about the compute density again e.g. in terms of Geekbench CPU score you would need 8 x C4 to equal one 8th gen i5 NUC. The 8-node cluster would need to shrink quite a lot to get same density. (?) So... I mean, doesn't that suggest that using fewer, more powerful CPUs in the cluster would be more likely to meet the goals?

fvolk
Posts: 606
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 72 times
Contact:

Re: Micro-server

Post by fvolk »

powerful owl wrote: although doesn't meet exilibris requirements.
True, it was more like Odroid probably doesn't want to develop another separate MC* platform - so what needs to be available to build a cluster easily to capture additional markets?
powerful owl wrote: All that aside :) what software are you running on your cluster?
As the screenshot say thats Ray, V1.0 and first aarch64 testbuilds became available recently: https://github.com/ray-project/ray/issues/12128
so it is interesting to evaluate this framework :-)

A small Odroid cluster enables the experience of having real, physical nodes (multiple VMs and multiple physical nodes are not equal), while still enables evaluation of distributed software in homeoffice - small space, small cost, small power use. Make it easy to build a cluster without requiring a soldering iron! Price/Performance is another discussion. Building a larger Odroid cluster probably only makes sense in very specific use cases -> exilibris's
These users thanked the author fvolk for the post:
powerful owl (Sat Feb 13, 2021 6:04 pm)

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

The fact is that from benchmarks like Geekbench and Antutu it is impossible to say anything about the real workflow.

You should clearly understand that a processor is, by and large, a set of execution units, on top of which low-level microcode (micro instructions) is used, which the decoder interprets and then they are optimized on the pipeline.

The density of instructions in different tasks is different, so the efficiency of architectures cannot be fundamentally measured by such highly abstract benchmarks.

If we take scientific calculations with double precision, then the winner will be the architecture that has hardware blocks for executing instructions on data of a given precision.

Since a priori we do not know the instruction flow density and microarchitecture features, the only way to check performance is to run a worker thread on the CPUs that we have.

To do this, you just need to take the simplest Docker images from the docker hub and just run them. For example, some micro-blog Ghost.

A typical ecommerce workflow is telemetry processing (statistics) and processing of commercial information streams (JSON, XML formats), as well as image processing by libraries written in C and compiled with optimizations for one or another architecture.

Under such conditions, the results of absolutely all benchmarks cease to matter and the first place comes out how many requests a node can process per second (this is for geeks), and how much it will COST for you (this is for the viability of the business).

That is why top solutions with maximum computing power from the world's leading manufacturers are the death of your business!

Such high-performance solutions have a place to be ONLY THERE, where optimization is solely in the speed of calculations without regard to the cost, energy consumption, and, accordingly, heat release.

The cheapest solutions have a low computational density per space, since they are associated with placement in space, in the supply of communication (which does not depend, by and large, on what exactly you power). A date cable is just a data cable, it needs channels (topology).

You may say - but there are clouds! The same Graviton2 from Amazon. Yes, but it is optimized for the costs of a huge company and turns out to be in the hundreds !!! times more expensive than the infrastructure scattered over the network, conditionally in your garage, in the kitchen, in the office, in the server room, in the data center with high availability, etc., combined by software into a single whole, you provide both fault tolerance and response speed, due to huge parallelism within the given investment limits.

I'm not just saying that this is the future of all ecommerce. Not all manufacturers understand this potential. But this is just a consequence of ordinary mathematics.

Of course, you can just take and buy several racks of expensive equipment, but this will be equivalent to buying stocks overheated on the market, the dividends on which will pay off the purchase of the stock in 30+ years !!! It just doesn't make sense if you are not pursuing the goal of speculation for profit.

Therefore, software optimization for a specific hardware configuration is extremely important.

We do not even use docker, since this is an additional layer of abstraction, we put an extremely minimalistic Linux kernel and V8 with the necessary software is immediately rolled onto it.

Imagine you are painting a mosaic. Each color is optimized software for a specific task, and the structure of the mosaic is determined by the workflow itself. That is, it turns out such a kaleidoscope that seeks to occupy the entire available amount of computing resources, it constantly morphs its configuration to the workflow, in an effort to provide the optimal shape (like the optimal state of energy in space).

I like to create these things, they seem to me very harmonious with the very principles of nature.

But I need the appropriate hardware. Perhaps next year, we will order the production of the required batch of equipment directly, which will be strictly optimized for the need.

It is easy to scale, it is easy to change, there is nothing wrong if some of the nodes burn out during the execution of work, break, fail, it does not matter much and risks, since all tasks are diversified, depending on the degree of importance. If it is very important - simultaneous execution with a given redundancy (three nodes do the same thing comparing the results, quorum), if the tasks are not important, then diversification occurs with a shift in time, as if all your nodes are one big pipeline, the optimization of which occurs after it instant configuration of workload, automatically.

Do you understand what I mean? I always have trouble explaining what I mean :(

P.S. Also, if u use PHP u can read about PHP optimization for ARM on Amazon Graviton2 sample:
https://aws.amazon.com/ru/blogs/compute ... instances/

powerful owl
Posts: 209
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 3 x H2, 1 x H2+
Has thanked: 65 times
Been thanked: 23 times
Contact:

Re: Micro-server

Post by powerful owl »

Of course, what counts is application performance. But a benchmark is at least quantifiable data, in the absence of any other. Do you have data to demonstrate the higher density/power efficiency of your proposal for actual application...?

User avatar
mad_ady
Posts: 9405
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 599 times
Been thanked: 666 times
Contact:

Re: Micro-server

Post by mad_ady »

@exilibris isn't your approach shifting costs from hardware/power usage to software/administration costs?
I mean, the architecture sounds complex and you may need to have skilled sysadmins to keep everything up. And human cost is the one thing everyone wants to cut...

exilibris
Posts: 19
Joined: Fri Jan 29, 2021 9:08 pm
languages_spoken: english
ODROIDs: 8x MC-solo
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Micro-server

Post by exilibris »

powerful owl wrote:
Sun Feb 14, 2021 12:59 pm
Of course, what counts is application performance. But a benchmark is at least quantifiable data, in the absence of any other. Do you have data to demonstrate the higher density/power efficiency of your proposal for actual application...?
I will try to make a selection of processing tasks that will clearly show the difference.
mad_ady wrote:
Sun Feb 14, 2021 3:43 pm
@exilibris isn't your approach shifting costs from hardware/power usage to software/administration costs?
I mean, the architecture sounds complex and you may need to have skilled sysadmins to keep everything up. And human cost is the one thing everyone wants to cut...
If the only and key expenditure is my time, despite the fact that all other processes are reduced to banal devops, then I am happy with such a cost, because this is what I love to do in life, while I assess the profitability (that is, I do not solely as a hobby or entertainment).
These users thanked the author exilibris for the post:
powerful owl (Mon Feb 15, 2021 11:18 am)

fvolk
Posts: 606
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 72 times
Contact:

Re: Micro-server

Post by fvolk »

Mh, ok, for science (and fun! ;-), lets compare size/price/performance/power of small platforms.
The 3 players in a size comparison:
compare3.jpg
compare3.jpg (176.59 KiB) Viewed 960 times
Odroid C4, 4 cores, ~1.3W idle, ~70 EUR without microSD and PSU
Odroid H2, 4 cores, ~3.3W idle, ~130 EUR without memory, mass storage and PSU
ASRock DeskMini H470 with 10700T(=8C/16T), <10W idle, ~550 EUR without memory and mass storage

Test workload is typical business job,
Python script that crunches ~400MiB json files for input and outputs csv tables of result.

Evaluation result, number of parallel jobs per platform chosen as cores/threads are available:
C4: 4 jobs in parallel run for 83m10s, consuming ~3.1W
H2: 4 jobs in parallel run for 23m40s with cores at 2.4GHz, consuming ~14W
DM: 16 jobs in parallel run for 21m35s, with cores at 2.8GHz, consuming ~61W (at start short boost peak 3.5GHz with ~85W)

Now.... read out of this whatever you want :-)

I have no N2+ to compare to, I don't own one.
These users thanked the author fvolk for the post (total 2):
tobetter (Thu Feb 18, 2021 7:18 am) • odroid (Thu Feb 18, 2021 10:17 am)

powerful owl
Posts: 209
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 3 x H2, 1 x H2+
Has thanked: 65 times
Been thanked: 23 times
Contact:

Re: Micro-server

Post by powerful owl »

Cool. Unfortunately I no longer have the NUC with i5-8259 but I wanted to throw it in the following anyway. So just tossing up some more numbers:

Volume
Odroid C4: 85x56x12(?) = 57cc
Odroid H2: 110x110x47 = 569cc
NUC8i5BEK: 117x112x36 = 471cc
H470: 155x155x80 = 1922cc

Power efficiency (lower is better) [updated]
Odroid C4: 83.2x3.1/4 = 64.5
Odroid H2: 23.7*14/4 = 83
NUC8i5BEK: ?
H470: 21.6*61/16 = 82.4

Compute density (lower is better) [updated]
Odroid C4: 83.2x57/4 = 1186
Odroid H2: 23.7*569/4 = 3371
NUC8i5BEK: ?
H470: 21.6*1922/16 = 2594

Would you be willing to to run the test on the H470 with 8 parallel jobs? Just that it's only about the same runtime as the H2 so I wonder whether something else is going on :) [Thank you, updated numbers above.]
Last edited by powerful owl on Thu Feb 18, 2021 6:35 pm, edited 1 time in total.

fvolk
Posts: 606
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 72 times
Contact:

Re: Micro-server

Post by fvolk »

powerful owl wrote: Volume H470: 155x155x80 = 1922cc
Note the DM case has on one side 2 slots for 2.5" drives, so there is some empty space in the case.
powerful owl wrote: Unfortunately I no longer have the NUC with i5-8259
I think NUCs are too small, they struggle to get rid of the heat (see overheating reports on Amazon reviews) and need a strong(=loud) cooling fan. The DM allows a larger fan and with a 35W CPU it stays silent even under load - however a 65W CPU (also AMD models...) pushes a DM also to its heat limit. In theory taking a 65W Intel CPU - they are cheaper - and limiting it in UEFI probably works too, but I never tried. Others, like Gigabyte Brix or Zotac Zbox have similar design trade-offs, if you want a micro server you have to decide your priority.
To make a custom dense compute cluster probably the best bet is to take the Mini-STX mainboard from the H470 and build a custom dense mounting rack and cooling solution. For more than 8 cores, go up to Mini-ITX.
powerful owl wrote: Would you be willing to to run the test on the H470 with 8 parallel jobs? Just that it's only about the same runtime as the H2 so I wonder whether something else is going on :)
I already did that but didn't post the numbers. Half the load is gone, so it is roughly twice as fast, makes sense:
DM: 8 jobs in parallel run for 11m35s, with cores at 3.0GHz, consuming ~58W (at start short boost peak 3.5GHz with ~75W)

I didn't try AVX* enabled code as the H2 doesn't support it, but the DM would probably be faster, run in frequency slower and draw more power.
These users thanked the author fvolk for the post:
powerful owl (Thu Feb 18, 2021 6:35 pm)

User avatar
rooted
Posts: 8512
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 740 times
Been thanked: 345 times
Contact:

Re: Micro-server

Post by rooted »


fvolk wrote:
powerful owl wrote: Volume H470: 155x155x80 = 1922cc
I think NUCs are too small, they struggle to get rid of the heat (see overheating reports on Amazon reviews) and need a strong(=loud) cooling fan.
No doubt, I had the precursor to the NUC the Dell Zino and it died a slow death from heat, it was AMD based so it probably ran a little hotter than an Intel NUC however.

powerful owl
Posts: 209
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 3 x H2, 1 x H2+
Has thanked: 65 times
Been thanked: 23 times
Contact:

Re: Micro-server

Post by powerful owl »

fvolk wrote:
Thu Feb 18, 2021 4:39 pm
powerful owl wrote: Would you be willing to to run the test on the H470 with 8 parallel jobs? Just that it's only about the same runtime as the H2 so I wonder whether something else is going on :)
I already did that but didn't post the numbers. Half the load is gone, so it is roughly twice as fast, makes sense:
DM: 8 jobs in parallel run for 11m35s, with cores at 3.0GHz, consuming ~58W (at start short boost peak 3.5GHz with ~75W)
I misunderstood your numbers, I assumed that the total work in each case was the same, but done in parallel (e.g. like make -j). However I realize now that each "job" is the same amount of work. I've updated my calcs above accordingly.

powerful owl
Posts: 209
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 3 x H2, 1 x H2+
Has thanked: 65 times
Been thanked: 23 times
Contact:

Re: Micro-server

Post by powerful owl »

I think the NUC is fair to include in this comparison ->
fvolk wrote:
Thu Feb 18, 2021 5:57 am
Mh, ok, for science (and fun! ;-), lets compare size/price/performance/power of small platforms.
I realize the volume calculations are not apples to apples. And that the NUC's thermals aren't intended for continuous high % use (yet paradoxically they don't make very good desktops because of the noise...). But these are small platforms you can buy off the shelf and which we can run tests on if we want to have some science fun ;) (Do we?)
Last edited by powerful owl on Thu Feb 18, 2021 7:35 pm, edited 1 time in total.

powerful owl
Posts: 209
Joined: Thu Mar 28, 2019 8:57 pm
languages_spoken: english
ODROIDs: 3 x H2, 1 x H2+
Has thanked: 65 times
Been thanked: 23 times
Contact:

Re: Micro-server

Post by powerful owl »

Anyway, leaving the NUC out for now, it seems the power efficiency of the C4 is not much higher than the others, a bit over 20%:

Power efficiency (lower is better) [updated]
Odroid C4: 83.2x3.1/4 = 64.5
Odroid H2: 23.7*14/4 = 83
H470: 21.6*61/16 = 82.4

(Did I work this out correctly?) With higher communication/coordination overhead, it will likely have no advantage at all.

Compute density (lower is better) [updated]
Odroid C4: 83.2x57/4 = 1186
Odroid H2: 23.7*569/4 = 3371
H470: 21.6*1922/16 = 2594

It seems you could pack more compute power into a tighter space, but there's no magic, you still have to get rid of the heat...

Post Reply

Return to “The Ideas”

Who is online

Users browsing this forum: No registered users and 0 guests