ODROID Magazine October Issue Available Now! (Home Server)

Moderators: meveric, mdrjr, odroid, robroy

ODROID Magazine October Issue Available Now! (Home Server)

Unread postby robroy » Fri Oct 13, 2017 11:39 am

Hardkernel is proud to announce that the ODROID Magazine for October 2017 is now available!

This month, we concentrate on home server and high performance computing solutions using the new ODROID-MC1 and ODROID-HC1. OpenMediaVault is a way to securely share files across the Internet so that you can access them from your smartphone or remotely from anywhere in the world. Docker Swarm mode allows multiple computers to be used as a distributed server, increasing the redundancy of critical applications. Because of the modularity of the new ODROIDs, parallel computing is much more available using MPJ Express as a foundation for solving complex issues with distributed computing models. KVM is a way to virtualize your computer in order to turn it into a hypervisor, allowing multiple operating systems to share a single hardware host.

Gaming is always a popular application for ODROIDs, and we feature two DIY articles that show you how to create a portable gaming machine to support a variety of console emulators. Brian Kim, a Hardkernel engineer, also offers a peek into his lifestyle with our popular Meet An ODROIDian column. For those who enjoyed the previous PDF version of ODROID Magazine before we transitioned to an HTML-based webzine, rest assured that we are continuing to improve the accessibility and legibility of our magazine, and plan to offer an additional offline PDF version of the current issues soon.


Image
Rob Roy, Editor-in-Chief of ODROID Magazine (submit an article | make a donation)
User avatar
robroy
 
Posts: 3347
Joined: Mon Feb 25, 2013 2:29 am
Location: San Francisco, California
languages_spoken: English
ODROIDs: Q2, X, X2, U2, U3, XU-E, XU3, XU4, C1, C1+, C2

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby memeka » Fri Oct 13, 2017 11:47 am

User avatar
memeka
 
Posts: 3730
Joined: Mon May 20, 2013 10:22 am
languages_spoken: english
ODROIDs: XU rev2 + eMMC + UART
U3 + eMMC + IO Shield + UART

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby robroy » Sat Oct 14, 2017 4:29 am


The figures in these articles have been restored.

memeka wrote:figures poor quality in https://magazine.odroid.com/article/hom ... omization/ (300x175px for code screenshot?!)

The actual code is included right above each of the figures.

memeka wrote:figures poor quality in https://magazine.odroid.com/article/odr ... ted-guide/

The images have been replaced with higher resolution versions. Right-click on the image and select "View Image" to view them full-screen.
Rob Roy, Editor-in-Chief of ODROID Magazine (submit an article | make a donation)
User avatar
robroy
 
Posts: 3347
Joined: Mon Feb 25, 2013 2:29 am
Location: San Francisco, California
languages_spoken: English
ODROIDs: Q2, X, X2, U2, U3, XU-E, XU3, XU4, C1, C1+, C2

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby odroid » Sat Oct 14, 2017 9:41 am

Yes! all the issues have been sorted. :)
User avatar
odroid
Site Admin
 
Posts: 24987
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby elatllat » Sun Oct 15, 2017 3:10 pm

I'd say the XU4 can have 'large' storrage, while the HC1 is limited to single drive setups. (Looks bad to call small things large)
elatllat
 
Posts: 614
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby memeka » Sun Oct 15, 2017 5:33 pm

elatllat wrote:I'd say the XU4 can have 'large' storrage, while the HC1 is limited to single drive setups. (Looks bad to call small things large)


One of my suggestions was to use both usb3 hosts for sata - only data, so user have to add their own power (maybe make a separate power adapter for drives, maybe include on the board connectors for 2x2.5” drives) - and then remove emmc and use Ethernet over sdio.
This way you have 2 high speed sata ports (each can support sata multiplexer), and still have gb Ethernet (or close to).
The price would be higher, but I think worth it?
User avatar
memeka
 
Posts: 3730
Joined: Mon May 20, 2013 10:22 am
languages_spoken: english
ODROIDs: XU rev2 + eMMC + UART
U3 + eMMC + IO Shield + UART

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby elatllat » Sun Oct 15, 2017 6:42 pm

memeka wrote:...2 high speed sata ports...

Is not much better than 1;
77 (or whatever the max USB device number is) is a lot more scalable than 1 or 2.
But this is the Magazine thread so my point is the Magazine should read
"Simply Storing Multimedia Files"
not
"Storing A Large Amount Of Multimedia Files"
as it does not cover the large data requirements of multiple disks, and related fun software (mdadm/lvm2/zfs/GFS2/GlusterFS)
elatllat
 
Posts: 614
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby memeka » Sun Oct 15, 2017 6:58 pm

You can have 8 multiplexed sata per port, and with one sata per usb3 host, you get decent good speed for 4-8 drives total (2-4 drives per usb3 host).
But yeah this is not what the thread is about :)
User avatar
memeka
 
Posts: 3730
Joined: Mon May 20, 2013 10:22 am
languages_spoken: english
ODROIDs: XU rev2 + eMMC + UART
U3 + eMMC + IO Shield + UART

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Sun Oct 29, 2017 10:53 pm

Hello Im new to this forum.

I got xu4 and happy.

but need to go serious if possible

Instead of investing in $$$KKKKkkkk for server component (eg: webserver + MySQL server + failover + redundancy)

Im implanting an online market place serving dynamic content. I havnt launched yet, so Ive got no hits, but id like to know how my petite ODROID
will scale to handling 100-500 concurrent users x multiple requests (eg: 10 requests per webpage X 100 users=1000 hits/second , x 500users=5000 hits/second)

how can I achieve this using a stack of xu4's (1 stack (X) x XU4 to power webserver (using Lighttpd / PHP 7 FastCGI mod) , 1 stack (X) x XU4 to power MySQL ?

Webserver will be local (EMMC) across all ODROID stack of nodes, same to for MYSQL server (runs off EMMC) , but would prefer data to be accessed from NAS or USB 3.0 drive.

Is this a silly idea ???? or will it work , seriously .....

BTW..

I opt to use ethernet NAS (maybe small stack offered by ODROID where my data files and MySQL will reside and where my webserver(stack) can access to serve
content to end user

OR..
set of 2 x usb3 drives (load balanced / fail over set) somehow connected to a usb hub from where my stack of xu4 can connect to this hub

Which will serve content faster, from EMMC direct from ODROID or using USB Hub of connected drives or Ethernet NAS via 1gb network ???

Honestly , im not very good at this, its a dream I hope to become reality... (Go Go ODROID !)

ive read articles in your forum/mag for home clustering or Home Cloud or Dockers etc, but Im not going down that path. I wish to avoid Docker or other complex
frameworks.

Im over complexities..... due to arising (future) headaches when faults,bugs or someother thing pops up ,{

Any help welcome

cheers

Jim.
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby mad_ady » Sun Oct 29, 2017 11:45 pm

I can't tell you how it will behave, but there are tools to test it out - like ab from apache: https://www.petefreitag.com/item/689.cfm
Point it to pages that make use of the database to see how they load as well.
User avatar
mad_ady
 
Posts: 2741
Joined: Wed Jul 15, 2015 5:00 pm
Location: Bucharest, Romania
languages_spoken: english
ODROIDs: XU4, C1+, C2

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Sun Oct 29, 2017 11:59 pm

Hi M.A,

quick reply thanks, heard about AB but not using apache, using lighttpd cause its light weight.

but im more focused on how I can setup this config where each stack segmented for purpose (eg: stack of webservers , stack of MySQL servers)
how do I go about and getting these stacks to share the load (cpu, data transfer, etc.) between each node of a particular stack , or
does lighttpd handle this (for webserving) and Ubuntu Xenial (for O/S related) ?
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Mon Oct 30, 2017 12:15 am

mad_ady wrote:I can't tell you how it will behave, but there are tools to test it out - like ab from apache: https://www.petefreitag.com/item/689.cfm
Point it to pages that make use of the database to see how they load as well
.


Hi M.A,

quick reply thanks, heard about AB but not using apache, using lighttpd cause its light weight.

but im more focused on how I can setup this config where each stack segmented for purpose (eg: stack of webservers , stack of MySQL servers)
how do I go about and getting these stacks to share the load (cpu, data transfer, etc.) between each node of a particular stack , or
does lighttpd handle this (for webserving) and Ubuntu Xenial (for O/S related) ?
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby mad_ady » Mon Oct 30, 2017 12:18 am

ab is a client - you can test any web server with it. Regarding architecture - it depends. Normally you would have a set of webserver hosts that get served requests from a load balancer. The load balancer should be aware of tcp sessions so that a client communicates with the same server to preserve state.
The servers could talk to a single database but it doesn't scale, or could talk to one of X databases (or through another load ballancer). The problem is how do you synchronize X databases? It depends on your use case
User avatar
mad_ady
 
Posts: 2741
Joined: Wed Jul 15, 2015 5:00 pm
Location: Bucharest, Romania
languages_spoken: english
ODROIDs: XU4, C1+, C2

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby elatllat » Mon Oct 30, 2017 6:00 am

How to scale should really be it's own thread, but it's hardware independent so just read this.
Last edited by elatllat on Tue Oct 31, 2017 5:02 am, edited 1 time in total.
elatllat
 
Posts: 614
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Mon Oct 30, 2017 8:03 am

mad_ady wrote:ab is a client - you can test any web server with it. Regarding architecture - it depends. Normally you would have a set of webserver hosts that get served requests from a load balancer. The load balancer should be aware of tcp sessions so that a client communicates with the same server to preserve state.
The servers could talk to a single database but it doesn't scale, or could talk to one of X databases (or through another load ballancer). The problem is how do you synchronize X databases? It depends on your use case


That's the idea , but don't know how to implement this.

I think with webserver, load balancing is built in,

I also read articles on the mc1 (my cluster) but that is for computational developer purposes. uses dtcc ? to distribute the processing , but not user if its used for webserver or database serving purposes.

I heard about rsync (files) and mysqldump (MySQL) etc to do copies , syncing ..

I got files to read/write to I guess they'll sit with webserver(local) on EMMC or sit as NAS hosting MYSQL server (Network).

how do I accomplish this ?

hoping theres someone in ODROID whos got experience in setting this type of setup im looking to do so I can launch my site.

need to launch ASAP.
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Mon Oct 30, 2017 8:46 am

[quote="elatllat"]How to scale should really be it's own thread, but it's hardware independent so just read this;

https://en.m.wikipedia.org/wiki/Load_ba ... (computing)[/
quote]

Empty Page !!
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby odroid » Mon Oct 30, 2017 9:14 am

User avatar
odroid
Site Admin
 
Posts: 24987
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Mon Oct 30, 2017 2:48 pm

Works now !!!

before page came up empty...
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Tue Oct 31, 2017 1:26 pm

elatllat wrote:How to scale should really be it's own thread, but it's hardware independent so just read this.


Hi Elatlat,

Yeh ive looked at it.

But im not a networking guy.

I know how to install s/w (I come from windows background) now im treading on Linux turf (using Ubuntu Xenial).

Is anyone know how I can setup a loadbalanced cluster so I can drive my website that connects with MYSQL server
currently both these services run on a single unit. I have fastCGI running on Lighttpd webserver.

From artlices ive read I may use fastCGI's load balancing config OR use Mod Proxy ??
Ive never used ModProxy I dont know what it does or how it will benefit .
SO im a stuck ... This part has got me Stumped !!!! What route shoud I take ???

I don't know if both MYSQL/Lighttpd should be on same unit or should be dedicated per 1 service (1 for Ligttpd, 1 for MySQL)

I also wish to run NAS (eg: HC1 unit to power an SSD enterprise drive), I may buy 2 of these units so in someway perform mirroring or
realtime/incremental backups on files OR use RAID to perform this (Ive never setup RAID before)..

Yet again im not sure where I should place my MYSQL files, should they be on a HC1 unit or on the dedicated MYSQL Server Unit ?

Maybe I can setup NAS to store all information there (eg: System boot, webserver files , MySQL db, etc..) so when a unit (webserver or MySQL server) boots (in a cluster environment) it can be done from NAS device/s ? is this possible ?
If so will there be any advantage of speed via Ethernet vs local EMMC storage ?

Which setup will perform better, cause im conscious of the 2gb memory limit and limit on 1gb Ethernet.

I don't know which course to take.

I havnt purchased extra units (XU4) yet, until I know what to do, theres no point in forking $ on XU4.

Hasn't anyone built a webserver/MySQL load balance/failover/redundant system using Xu's or other SBC ?

Surely theres gotta be some here in the forums who can guide me into setting up something like this ????

Jim.
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby mad_ady » Tue Oct 31, 2017 4:38 pm

jimodroid wrote:
From artlices ive read I may use fastCGI's load balancing config OR use Mod Proxy ??
Ive never used ModProxy I dont know what it does or how it will benefit .
SO im a stuck ... This part has got me Stumped !!!! What route shoud I take ???

It seems that most people recommend using HAProxy as a load ballancer http://www.haproxy.org/.
I'm assuming a high traffic site, so I'd recommend this setup (though I have not setup loadballancers or high-traffic servers):
1. Set up DNS records pointing to different IPs in round-robin fashion (e.g. www.example.com -> resolves to A 100.200.100.1 and 100.200.100.2, which are your load ballancers). Having separate IP addresses can help you in case of a long DDoS attack you can disable the attacked IP and ask your ISP to blackhole it and announce the other IP through DNS so that customers can still connect.
2. Set up load ballancers to be "sticky" (http://cbonte.github.io/haproxy-dconv/1 ... html#3.3.6) so that the same user lands on the same server. The load ballancer will be the public facing resource and needs to be able to absorb and filter all the garbage that is online (e.g. DDoS attacks)
3. Have a "farm" of X Web servers that are serviced by the load ballancer. These can run Apache/NGIX/Lighttpd
4. Regarding MYSQL - it's a bit out of my league. If your site has mostly "read-only" content you could have X Mysql servers which are queried by the web servers in a 1:1 fashion, but the changes to the database will be local. You would need a mechanism to synchronize changes between the databases, which is something that mysql is not known for (Oracle can do this, but it's more expensive). If the databases need to be identical, you could use one single database + replication for backup, but you would hit a memory barrier with the XU4/HC1. You could use a regular server with loads of RAM for this.

I don't know if both MYSQL/Lighttpd should be on same unit or should be dedicated per 1 service (1 for Ligttpd, 1 for MySQL)

Due to high RAM requirements of MySQL, they should be separate for high load environments.
I also wish to run NAS (eg: HC1 unit to power an SSD enterprise drive), I may buy 2 of these units so in someway perform mirroring or
realtime/incremental backups on files OR use RAID to perform this (Ive never setup RAID before)..

Note that enterprise SSD drives may use 12V, which the HC1 doesn't support. For RAID support look at the Cloudshell 2. But remember that RAID is not backup. If you delete a file on a RAID array it gets deleted from both disks! So you need backup in addition to RAID. RAID only protects you from a disk failure.

Yet again im not sure where I should place my MYSQL files, should they be on a HC1 unit or on the dedicated MYSQL Server Unit ?

Maybe I can setup NAS to store all information there (eg: System boot, webserver files , MySQL db, etc..) so when a unit (webserver or MySQL server) boots (in a cluster environment) it can be done from NAS device/s ? is this possible ?
If so will there be any advantage of speed via Ethernet vs local EMMC storage ?

Ideally a NAS will use all available RAM for caching, so it's a good idea to have a dedicated NAS without too many services running. Though on SSDs you won't see too many delays due to cache misses.
You could have a NAS that is used to boot the other boards with NFS boot (http://odroid.us/mediawiki/index.php?ti ... ile_System), but I/O performance will be limited by the network. Having multiple systems boot from a single NAS will reduce performance. Having a big (32-64GB) EMMC with local content should be faster than getting it over NFS from a SSD because the emmc bandwidth is larger than the network card bandwidth.

Hasn't anyone built a webserver/MySQL load balance/failover/redundant system using Xu's or other SBC ?

Surely theres gotta be some here in the forums who can guide me into setting up something like this ????

Jim.

I haven't seen any guides that show such a system with load ballancer and multiple MYSQL hosts, but it looks like it could benefit a lot of Odroid users (including Hardkernel - they could move their servers to XU4s).
User avatar
mad_ady
 
Posts: 2741
Joined: Wed Jul 15, 2015 5:00 pm
Location: Bucharest, Romania
languages_spoken: english
ODROIDs: XU4, C1+, C2

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby elatllat » Tue Oct 31, 2017 10:06 pm

yah the hardest part would be the database;
One could use a mariadb cluster which is nice until the indexes for your data don't fit in RAM anymore, after which distributed transactions are helpful but the 2 technologies are not compatible (AFAIK) so it's like you can have RAID 1 or 0 but not 10... but before jumping off the deep end, what are the exact requirements here?

For example if you were planning on hosting a website backed by wordpress you can just add an IP for each node to the domain name, have each node running lightted serving pre-compressed and merged cached files that are only ever updated from mariadb when content is added. If you want some user settings, a shopping cart, etc implement it client side in SQLite (html5 has some fun distributed technology).

Then if you really want a RAID 10 analogous data store you can look in to flat file frameworks on top of a distributed file system.
elatllat
 
Posts: 614
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Wed Nov 01, 2017 10:43 am

elatllat wrote:yah the hardest part would be the database;
One could use a mariadb cluster which is nice until the indexes for your data don't fit in RAM anymore, after which distributed transactions are helpful but the 2 technologies are not compatible (AFAIK) so it's like you can have RAID 1 or 0 but not 10... but before jumping off the deep end, what are the exact requirements here?

For example if you were planning on hosting a website backed by wordpress you can just add an IP for each node to the domain name, have each node running lightted serving pre-compressed and merged cached files that are only ever updated from mariadb when content is added. If you want some user settings, a shopping cart, etc implement it client side in SQLite (html5 has some fun distributed technology).

Then if you really want a RAID 10 analogous data store you can look in to flat file frameworks on top of a distributed file system.



Hi Elatllat,
Yes I was looking into that (MariaDB Galeria), theres also some s/w on BSD license u can use to administer setup for replication using NAS called WebAdmin (http://www.webmin.com) its gotta lot of plugins to work with MySQL and other resources , MariaDB can be utilised for the above using this, still haven't read IN DEPTH...

That's another area to investigate (RAID10 / distributed file system) which ill consider, but whats the performance advantage going this way rather than using MySQL or MAriaDB / cluster ?


My site is custom built on PHP platform, im a web developer , im not using any framework...
Ive never used MariaDB (setup by one of the original founders of MYSQL to keep it community orientated) though I understand its using same or similar to MYSQL engine.
Ill look into this....

HTML5/SqlLite is an option as well, though been lazy regarding implementing this alternative.

At present not launched, and will take time to gather traffic.

I just don't know if a cluster (load balancing) working of the XU's will be able to deliver a performance gain rather than using a decent (low end) server such as HP Gen 9 or 10 ($900) , matter is using its own OS (ClearOS) which I know nothing about. ( ANY SUGGESTIONS ?)

If a cluster of XU's can work properly via the distributed computing model (CURRENT SETUP IS MYSQL , LIGHTTPD webserver on EMMC+SDCARD) via the Load Balancer (ie: FastCGI using Lighttpd webserver with configs) or as suggested by
MadAddy (to go HAProxy) , ive never setup something like this before , and if my guess is correct that really it depends on the amount of network traffic (internet packet requests sent by individuals) to query my website, then I think 2gb ram on webserver @ 2ghz should be sufficient to handle (X) amount of concurrent requests.
But I don't know the limit of concurrent users based on the the hardware limits of XU4 using a 1gb Ethernet.

BTW

Im not talkin big data here, I don't think ill pass the 1 million record (accounts) on any 1 MySQL db table.
Im not going to store images in this either.
I don't anticipate each account record to exceed 5kb of textual information
so 1mill records x 5kb/records =~ 5gb ddb.

If anyone can provide some data regarding how much an Xu4 can handle concurrent users/hits per page on website that queries a db size of approx. 5gb, would be of great benefit to analize what system/nework config ill need to get this started.

( I WISH SOMEONE FROM HARDKERNEL CAN GIVE ME A SOLUTION, SPENT 2 WEEKS TRYING TO GATHER INFORMATION, GOING IN CIRCLES.... )

Even to suggest a basic configuration (for now) whether a mirroring or replication model be used for a setup consisting of

1.
a) (X) x XU4 webservers + (X) x XU4 MySQL servers OR
b) (X) x XU4 (webserver+MySql server bundled together)

2.
a) (X) x XU4 AS NAS (Filessystem for storage other data) OR
b) (X) x XU4 Filesystem + other data on EMMC (XU4 - EMMC has limit of 64gb , I may pass this storage requirement mark , eventually but not foreseeable soon)

I did forget to mention ill be running cron job every (5) mins to perform cleanup duties

All this on XU4 ?

is this possible and u think it may pass the test of feasibility for end user experience ???

thanks
Jim
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby odroid » Wed Nov 01, 2017 11:06 am

We've tried to run this forum with multiple HC1 boards and SSDs. But we had a problem to implement the clustered database.
PHPBB is using mysql which can't run on a cluster system due to our limited knowledge. We couldn't find any proper solution. Sorry about that.
viewtopic.php?f=55&t=28361
User avatar
odroid
Site Admin
 
Posts: 24987
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Wed Nov 01, 2017 2:53 pm

odroid wrote:We've tried to run this forum with multiple HC1 boards and SSDs. But we had a problem to implement the clustered database.
PHPBB is using mysql which can't run on a cluster system due to our limited knowledge. We couldn't find any proper solution. Sorry about that.
viewtopic.php?f=55&t=28361


Hi Odroid,

thanks for the followup. had a read from link suppliers. gives me a better idea, I did come across your article on using Docker Swarm Part 1,
but was hoping to avoid taking that route because I have no knowledge of this framework.

the 1 x webserver, 1 x database (independent units) is probably the easiest way to go. but need replication or mirroring, to achieve this havnt done before.

the cluster (mc1 ?) or 3-4 x XU4 using EMMC for I/O would be I guess the best performing route since EMMC buffer/datastream is > Ethernet 1gb.
Again, if I decided to keep MySQL then I have limitation on EMMC size of 64gb + low mem of 2gb ram.

but if I avoid db use altogether and use file system instead then backingup, replicating or mirroring shouldn't be an issue.
All I need to know is what storage solution should I use if I decide to use the File System route ?
The file system may work on the EMMC or may use an external SSD drive
What setup would you recommend for file system route ? (HC1 or cloudshell , other ?)
** BTW a note on file system : I use Xenial im not sure if im using fat32,ntfs,ext4 or other.
Either file format, what limits are on total number of directories , and total number files written to a directory available in each file system ?
any idea what the limit is ? if it was 100's thousands great but only 1000 subfolders per folder or 10,000 files per folder that may be worriesome.
Another issue would be performance on accessing through a hugle list of files or just getting the right file from the heap of files in a folder.
over concurrent request/s this may be poor performance... UNLESS there is a filesystem out there that has been built to deal with these scenario/s,
if u know of any please tell me...

Regarding a H/W and S/W configuration guide (Need help please) :

How would I do this if I were using the cluster (MC1 or stack of XU4's) to connect to either :

1.USB3 SSD drive (HC1, cloudshell, other device ?)
2.Ethernet on NAS ? (HC1, cloudshell , other device ?)

in option #1 , how would each node in the cluster access this USB3 drive/s, where would these drive/s sit ? would I need a set of these drive/s for each node, or can each node be setup to access these single set/s of USB drive/s ?

option #2 same as option#1 but instead accessing Ethernet

Ive exhausted my theoretical knowledge of all of the above.

Any practical real solution for either config scenario mentioned above is appreciated.
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby odroid » Wed Nov 01, 2017 3:13 pm

Here are some resources of the cluster computer configuration. But we don't have any experience of storage/database clustering.
https://wiki.odroid.com/odroid-xu4/appl ... build_farm
https://magazine.odroid.com/article/odr ... ted-guide/
https://magazine.odroid.com/article/odr ... g-started/
User avatar
odroid
Site Admin
 
Posts: 24987
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby elatllat » Wed Nov 01, 2017 8:21 pm

Start with ext4 and you won't hit any limits. For a light web site one issue you will hit is a hard 60k limit per node on connections, wether you run out of ram, cpu, or io first is largely dependent on software optimization... someone can give you lighttpd / mariadb benchmarks as a best case scenario but you would have to test your code to know for sure.
elatllat
 
Posts: 614
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Thu Nov 02, 2017 3:58 pm

elatllat wrote:Start with ext4 and you won't hit any limits. For a light web site one issue you will hit is a hard 60k limit per node on connections, wether you run out of ram, cpu, or io first is largely dependent on software optimization... someone can give you lighttpd / mariadb benchmarks as a best case scenario but you would have to test your code to know for sure.


Thanks Elatllat, for the tip.

is this 60k thing related to concurrent users or concurrent requests (hits) on a webpage or some thing else ?

ive decided im not going to combine webserver/MySQL server together but leave separate. the replication or mirroring thing is another issue
ill have to address. undecided. may even abandon the db thing altogether and go flat file.... (that would make life a whole lot easier...)

Im not assuming the XU4 is a permanent solution to this whole idea of my mine , but im guessing for now its a start to get the whole thing up and running
until (better to do beforehand) it reaches its max threshold of I/O, concurrency, memory use etc, whichever comes first. Im not a network guy...
It would have been great if there were real (live) stats showing how a webserver/dbserver using XU4 stands up to the reality of as a webhost or ecommerce thing.
Ive read comments on websites (eg: ameridroid -> projects) People say theve managed to do it and get some performance but theres no info to prove it

Anyway, ive received a lot of feedback, and its been helpful.

thanks to all...
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby elatllat » Thu Nov 02, 2017 9:49 pm

TCP was designed to use an unsigned 16-bit integer for the port number so each IP can have a maximum of 65535 simultaneous connections.
You could attempt practically limitless connections to a computer with many IPs but unless your application is quite unusual it's more helpful to split the load over more than one computer.

commonly developer time is more costly than more computers so optimisation is less common than one might like, but it's a noble/interesting pursuit.
elatllat
 
Posts: 614
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4

Re: ODROID Magazine October Issue Available Now! (Home Serve

Unread postby jimodroid » Fri Nov 03, 2017 3:21 pm

elatllat wrote:Start with ext4 and you won't hit any limits. For a light web site one issue you will hit is a hard 60k limit per node on connections, wether you run out of ram, cpu, or io first is largely dependent on software optimization... someone can give you lighttpd / mariadb benchmarks as a best case scenario but you would have to test your code to know for sure.




Again Elatllat,

You've been a great help, thanks for the link resources regarding tools/benchmarks.
Ill ijnvestigate this further...
jimodroid
 
Posts: 11
Joined: Fri Oct 27, 2017 10:58 pm
languages_spoken: english
ODROIDs: xu4


Return to ODROID Magazine

Who is online

Users browsing this forum: No registered users and 0 guests