"ODROID Bench"

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

"ODROID Bench"

Unread post by tobetter » Wed Sep 19, 2018 11:22 am

Hello all,

We have set up a remote ODROID experience zone for someone who wants to see the performance of ODROID SBCs. The ODROID SBCs in our testbed connected to the Gbit ethernet and open to the public. The experience zone offer benchmarks for performance, cloud server and more by SSH-ing.

For more detail, please visit this link.
https://medium.com/@tobetter/odroid-bench-c5c1a10d6bec

And please keep tracking this thread for request and questions, we are welcome your suggestions and requests.
But do not access them only for fun or hack, please... :mrgreen:
Attachments
odroid bench.jpeg
odroid bench.jpeg (136.39 KiB) Viewed 6776 times

User avatar
mad_ady
Posts: 5668
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 15 times
Been thanked: 16 times
Contact:

Re: "ODROID Bench"

Unread post by mad_ady » Thu Sep 20, 2018 11:49 pm

It's a great idea to have some systems for evaluation purposes.
Sorry, I was curious and I logged in for fun to have a look around and I have some suggestions security-wise:

1. ufw can leak IP addresses of people connecting in the logs. E.g. on host with port 2220 you can see some IPs in dmesg. Maybe ufw should not be logging to dmesg.
2. Are ssh connections triggering a docker instance to start up so that each session is isolated? I could connect to the same session from the same source:

Code: Select all

odroid@docker-xu4-20:~$ w
 14:17:01 up 2 days, 23:46,  2 users,  load average: 0.67, 0.29, 0.18
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
odroid   pts/1    80.97.238.77     14:11   53.00s  0.18s  0.11s sshd: odroid [p
odroid   pts/2    80.97.238.77     14:16    4.00s  0.03s  0.01s w
Edit: it seems not. Files that I created in a docker instance are still there after disconnecting and reconnecting. Somebody could fill up the disk with garbage bricking the system.

It would be educational/great to have users connect to their own docker instance of the system and to explain the setup in a magazine article.
3. You should be monitoring all systems with something like munin/cacti/mynetdata and trigger alerts when cpu/network gets too high to catch attackers/spammers.
4. You should turn off port forwarding and proxying over ssh. I was able to do:

Code: Select all

ssh -D8080 -p 2220 odroid@maze.odroid.com
And proxy my browser's traffic through your systems. It could be abused as a proxy or made a relay for SPAM. I do hope that the network IP is on a different internet connection than your corporate connection because you may become a botnet/ddos target that could affect your site/corporate access.
You can use PermitTunnel no in your sshd_config.
5. You could add a small command that exposes the current number of logged in users (each docker instance reads a file exposed by the host and the script runs periodically on the host). That would allow somebody to see how used a host is at a particular moment.
6. You should be logging user commands. You should say you are logging user commands in the motd you get when you log into a shell. This should at least make your legal department happy (you have an acceptable use policy), otherwise attackers could legally say they had no idea it was forbidden to start attacks from your system.

To log user commands on a "jumphost" type server I'm administering for internal users I use a system-wide /etc/bashrc with the following setup:

Code: Select all

if [ "$USER" == "root" ]; then
        echo "do nothing - no logging" > /dev/null
else

#set up logging
DATE=`date +%Y.%m.%d-%H.%M.%S`
SCRIPTLOG=/mnt/sdb1/bash-logs/$USER-$DATE
#prevent starting multiple instances
if [ ! -f "$SCRIPTLOG.typescript" ]; then
   script -q --timing=$SCRIPTLOG.timing $SCRIPTLOG.typescript
   exit
fi

The user's bash shell (you need to prevent the user from starting something else with ssh -t user@host /bin/sh) starts "script" which logs everything (input + output) to a separate disk and spawns a bash shell inside script. When the user quits that bash shell, script exits and closes the shell as well. The logs are stored in a directory owned by root with other w permission:

Code: Select all

[root@server ~]# ls -ld /mnt/sdb1/bash-logs/
drwx----wt 2 root root 548864 Sep 20 17:30 /mnt/sdb1/bash-logs/


This means that the unprivileged user can open a log only for writing. The directory holding the logs also has chattr +a so that files within are not deletable/truncatable.

Code: Select all

[root@server ~]# lsattr /mnt/sdb1/
-----a----I--e-- /mnt/sdb1/logs
-----a----I--e-- /mnt/sdb1/bash-logs

But since you give users root, they might be able to revert that permissions and erase their tracks.

To replay a log you can do something like:

Code: Select all

scriptreplay -t /mnt/sdb1/bash-logs/odroid-2014.11.12-08.32.07.timing /mnt/sdb1/bash-logs/odroid-2014.11.12-08.32.07.typescript 10
I would personally not give open access from the internet unless I were setting up a honeypot. You could have users ask for access via email/ticketing system and supply you with their IP address. But this would mean a greater operational overhead, but would keep the scum of the internet out (hopefully).

User avatar
rooted
Posts: 6449
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 4 times
Been thanked: 4 times
Contact:

Re: "ODROID Bench"

Unread post by rooted » Fri Sep 21, 2018 12:32 am

I think your complete message would have been better as a PM @mad_ady, certainly nothing wrong with the information but could help any script kiddie who may read it.

It's extremely helpful but too informative about the current exploitable condition of the servers. You know like when an whitehat finds exploits but emails the vendors before releasing :)

It is ideal to let someone like you (network engineer) help secure the system since the end users have root access, giving each new ssh instance it's own time limited container would negate a lot of the problems.

User avatar
mad_ady
Posts: 5668
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 15 times
Been thanked: 16 times
Contact:

Re: "ODROID Bench"

Unread post by mad_ady » Fri Sep 21, 2018 2:21 am

You're right, in retrospect I should have PMed it, but I also wanted to have an open discussion about how you secure something where you give away the root account.

Also - a suggestion. It may be wise to have the host restart periodically (e.g. every 12h) and to reinitialize the docker image so that people trying to get persistence would have a harder time.

One more suggestion - instead of odroid/odroid use a random 8 character password that changes every 12h. Present the current password in a web page as a captcha-like image (there are php scripts that can generate them). That way an automatic script would have trouble parsing them. Also disable ssh key authentication so an attacker can't bypass password changes.

User avatar
rooted
Posts: 6449
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 4 times
Been thanked: 4 times
Contact:

Re: "ODROID Bench"

Unread post by rooted » Fri Sep 21, 2018 5:34 am

That's why I said "complete message", since we all can learn from your posts.

Have you done a write up on securing something like this in the magazine?

User avatar
mad_ady
Posts: 5668
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 15 times
Been thanked: 16 times
Contact:

Re: "ODROID Bench"

Unread post by mad_ady » Fri Sep 21, 2018 2:58 pm

I don't think I'm qualified to give a complete example because I'm sure my advice is incomplete... I wouldn't want anyone to get bitten in the behind after following my advice thinking it was complete.
The problem with this game is that a security professional needs to secure all the holes, while an attacker needs to find only one that is open.

Regarding "how do you make ssh spawn a new docker instance" - here's what I would try:
You'd need the odroid user to log into the host and on login spawn the docker instance around them. You most likely can have per-user configuration in sshd_config that runs a command to jail that user into the docker instance. If not you could use the same /etc/bashrc trick above to start a docker instance on login. But it needs to be tested and most importantly ways around it (e.g. running non-interactively or with a different shell) need to be tested.

So, it's a lot of work (hopefully for nothing), but we can all learn from the experiment.

User avatar
rooted
Posts: 6449
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 4 times
Been thanked: 4 times
Contact:

Re: "ODROID Bench"

Unread post by rooted » Fri Sep 21, 2018 3:23 pm

Without per MAC | IP access control (which still can be beat) certainly it will be less secure. All Internet facing devices are subject to exploit as we should all know. Even LAN devices can be had through social engineering.

Most important is these devices are not connected to internal LAN at all.

It would be cool to print the load statistics on a available webpage somewhere. Then users could know which device is less loaded before choosing.

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Fri Sep 21, 2018 3:57 pm

@mad_ady, super thank you for security concern and your advice...I would like to learn more and apply all your idea to here. It's a shame that I am not that qualified.

When the ODROID Bench idea comes, we only concerned about how the device can be recovered whenever the device is messed up by using a command like rm -rf /, that's the reason why we use Docker container in ODROID which gives more flexibility to manage the devices at the same time it leads us to have different experience to set up the services and configurations which is fun but a headache as well. The good thing is the network for ODROID Bench is completely isolated from our internal network since we are also afraid of being hacked by a malicious user.

1. I haven't success to spawn a docker per a user logged in, technically I've failed and I didn't invest much time for this since it would be a little bit hard to run multiple containers in a small device with limited memory. It's worth to try, but I push back to do later. Also, we've reached that there would be an extra overhead to manage the login account per request and schedule the usage, this would be less worth to do...so we have only one account odroid anybody can use. :twisted:

2. That's a good point, to prevent spamming. I haven't reached this idea before I read your post. Thank you again, I just applied to XU4-20 and will apply the same to others. So PermitTunnel no will be applied to all devices.

3. Logging of ufw is just disabled, I couldn't find the solution to prevent showing them to dmesg. The easiest way is just disabled it. :D

4. I've installed netdata to monitor the system resources, but they are not showing to the public network yet since there no reverse proxy server yet. The browser in a client that access Netdata getting invalid response with https://maze.odroid.com:9920, 9920 is mapped to 19999 of XU4-20, netdata support http not https, correct me if I am wrong, http://maze.odroid.com:9920 is not properly working because of https://maze.odroid.com for NextCloude. So currently it's only accessable from internal network, considering to put a reverse proxy later which is in the plan to put more devices and running services.

5. This is the most difficult part that I wanted to figure out before launching ODROID Bench. As you described to store bash shell logging into a system, at least I wanted to isolate the bash shell history per connection. I've tried to store the logging to a remote system, not in a local storage since local storage also can be corrupted and space is also limited. Also as you pointed out, the logging in a local storage can be removed or changed by one for fun. Maybe I am overthinking, maybe storing in the local storage would be just fine. ;)

Even though I know the security solutions, obviously very poor, it's not always working in a docker container due to different resource management and some limitations. Also, difficulties to manage the containers in a different hardware although I only have 5 devices today...trying to set up a script or batch command set to manage them like installing, rebooting or more. If I become good at this, I would be able to start my own business to host the bunch of ODROID hardware.

Thank you for your advice again, and let me get back to fix ODROIDs on the bench.

Please keep giving your idea and advice, sir. :D

User avatar
mad_ady
Posts: 5668
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 15 times
Been thanked: 16 times
Contact:

Re: "ODROID Bench"

Unread post by mad_ady » Fri Sep 21, 2018 4:15 pm

2. Don't forget to test it with both ssh -D8080 and ssh -L8080:172.217.16.100:443 (www.google.com).
4. I am able to access netdata on http://maze.odroid.com:9920
5. Regarding logging you can try the following as well - recompile bash and have it export the command line to a remote syslog server via udp just before it's executed. Then you could store the commands offsite. The risk is if the user installs a different shell or reinstalls bash, or hides its commands in a script that they call.
Right, there's no easy solution.

There are professional solutions that do this kind of recording (like CyberArk PSM) that force you to log into a special system and can do video/keystrokes recording, but it's expensive and probably overkill for your use case.

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Fri Sep 21, 2018 4:34 pm

mad_ady wrote:2. Don't forget to test it with both ssh -D8080 and ssh -L8080:172.217.16.100:443 (http://www.google.com).
What is it supposed to be if I run ssh -p 2222 -L:8080:172.217.16.100:443 odroid@maze.odroid.com from my desktop?
4. I am able to access netdata on http://maze.odroid.com:9920
I am not... :cry: So, just in case...I opened all devices to run Netdata in different port numbers, 9920 / 9921 / 9930 / 9931.
[/quote]
5. Regarding logging you can try the following as well - recompile bash and have it export the command line to a remote syslog server via udp just before it's executed. Then you could store the commands offsite. The risk is if the user installs a different shell or reinstalls bash, or hides its commands in a script that they call.
Right, there's no easy solution.

There are professional solutions that do this kind of recording (like CyberArk PSM) that force you to log into a special system and can do video/keystrokes recording, but it's expensive and probably overkill for your use case.
Nothing easy...I just hope people who want to use ODROID are like an angel and polite, humble...and figure out the funny thing all with their own device.

User avatar
mad_ady
Posts: 5668
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 15 times
Been thanked: 16 times
Contact:

Re: "ODROID Bench"

Unread post by mad_ady » Fri Sep 21, 2018 4:42 pm

You shouldn't see this after connecting:

Code: Select all

adrianp@frost:~$ netstat -tpan | grep 8080
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      14361/ssh           
tcp6       0      0 :::8080                 :::*                    LISTEN      14361/ssh          
Regarding good behavior - remember this old russian saying: "Pray to God, but keep rowing to shore"
Expect the worse from your guests :)

elatllat
Posts: 1224
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1
Has thanked: 0
Been thanked: 2 times
Contact:

Re: "ODROID Bench"

Unread post by elatllat » Sat Sep 22, 2018 4:28 pm

mad_ady wrote:... there's no easy solution...
That's my security conclusion, but maybe a network monitoring/limiting solution on a routing node would help;
measure time(1day)/bandwidth(1GB)/outgoing ips(1k) per incoming IP and block after any of those thresholds are reached.

and as mad_ady said data persistence should be addressed; maybe a new docker image per day.

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Mon Oct 29, 2018 6:00 pm

Please welcome new members, ODROID-H2 on the bench.
Image

For details about them, please visit the link below and let us have your inputs regarding ODROID Bench here.

Have fun with ODROIDs.

tkaiser
Posts: 671
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by tkaiser » Mon Oct 29, 2018 7:54 pm

I would modify motd to display a welcome message reminding guests that they can't expect native performance due to some Docker overhead (surprisingly low BTW!) and other guests probably being active at the same time. People love 'fire and forget' benchmarks, execute an sysbench or something like this and publish results not having in mind that someone else might run something similar in parallel and all numbers are just fractions of 'real performance'.

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Mon Oct 29, 2018 8:06 pm

tkaiser wrote:I would modify motd to display a welcome message reminding guests that they can't expect native performance due to some Docker overhead (surprisingly low BTW!) and other guests probably being active at the same time. People love 'fire and forget' benchmarks, execute an sysbench or something like this and publish results not having in mind that someone else might run something similar in parallel and all numbers are just fractions of 'real performance'.
Good idea, so I would make it tomorrow KST since some may access them tonight. Thanks.

elatllat
Posts: 1224
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1
Has thanked: 0
Been thanked: 2 times
Contact:

Re: "ODROID Bench"

Unread post by elatllat » Mon Oct 29, 2018 11:23 pm

The test server example/listing is out of date, should be

Code: Select all

ssh -oUserKnownHostsFile=known_odroid_hosts -p 22$X odroid@maze.odroid.com
where X is one of

Code: Select all

nmap -p 2222 192.168.0.0/24 | grep 192 | perl -pe 's/^.*\.//g'
1
20
21
31
35
40
42
44
Last edited by elatllat on Tue Oct 30, 2018 1:44 am, edited 1 time in total.

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Mon Oct 29, 2018 11:32 pm

elatllat wrote:The test server example/listing is out of date, should be

Code: Select all

ssh -oUserKnownHostsFile=known_odroid_hosts -p 22$X odroid@maze.odroid.com
where X is one of

Code: Select all

nmap -p 2222 192.168.0.0/24 | grep 192 | perl -pe 's/^.*\.//g'
1
20
21
31
35
40
42
44
Where is the list out dated you found?

back2future
Posts: 199
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by back2future » Sun Nov 04, 2018 8:50 pm

"They can be accessed through “ssh” with a port number dedicated to each machine, your access is limited in the Docker container on top of native Ubuntu “18.04.1” and the Linux kernel “4.15.0-38-generic”."
viewtopic.php?p=236119#p236119

Is there a possibility for updating basic Ubuntu “18.04.1” with (kvm or) virtualbox kernel modules for 4.15 kvm-kernel and made available inside docker containers?
Thx

[ edit:
lsmod | grep kvm
kvm_intel 212992 0
kvm 598016 1 kvm_intel
irqbypass 16384 1 kvm ]
Last edited by back2future on Mon Nov 05, 2018 2:53 am, edited 2 times in total.

wtarreau
Posts: 25
Joined: Thu Jan 21, 2016 1:22 am
languages_spoken: english, french
ODROIDs: C2, GO, MC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: "ODROID Bench"

Unread post by wtarreau » Sun Nov 04, 2018 11:46 pm

Hi,

it's really great to have opened this service. I could run my build farm benchmark on the H2 and add the results to the table : http://wiki.ant-computing.com/Choosing_ ... build_farm.

It turns out that the H2 is an excellent performer. Per core, it's 1.5 times faster than a 2 GHz cortex-A72, and 1.5 times slower than my 3.3 GHz core-i5! I suspect it's the highest performance we'll find for a long time in a fanless design. I think this board could find its way at unexpected places. For example, Korg uses mini-itx PC boards equipped with fans inside some of its high-end synthesizers. Such fans are a source of noise and failure and are really not welcome in such devices. Such a reliable high-performance board could make a difference in this type of devices.

tkaiser
Posts: 671
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by tkaiser » Mon Nov 05, 2018 12:19 am

wtarreau wrote:I suspect it's the highest performance we'll find for a long time in a fanless design.
Please keep in mind that Hardkernel reports temperatures at around 75°C when running stress-ng --cpu 4 --cpu-method matrixprod for 25 minutes over there: viewtopic.php?f=29&t=32536

Image

When I tested with sbc-bench (directly on the hardware not inside a container) highest reported temperature was 62°C with a 5 min cpuminer workload making use of SIMD extensions: http://ix.io/1qb0

Testing with stress-ng --cpu 4 --cpu-method matrixprod remotely the reported temperature jumped from 41°C in idle to 60°C within 5 minutes and still is at 69°C after 25 minutes (71°C after an hour). This is a somewhat different temperature graph suggesting Hardkernel either improved thermal efficiency of the heatsink or there is at least some airflow at Hardkernel's location...

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Mon Nov 05, 2018 10:28 pm

back2future wrote:"They can be accessed through “ssh” with a port number dedicated to each machine, your access is limited in the Docker container on top of native Ubuntu “18.04.1” and the Linux kernel “4.15.0-38-generic”."
viewtopic.php?p=236119#p236119

Is there a possibility for updating basic Ubuntu “18.04.1” with (kvm or) virtualbox kernel modules for 4.15 kvm-kernel and made available inside docker containers?
Thx

[ edit:
lsmod | grep kvm
kvm_intel 212992 0
kvm 598016 1 kvm_intel
irqbypass 16384 1 kvm ]
Sorry, for some reason, a container doesn't run KVM inside docker now. It would take more time if I need to do so.

back2future
Posts: 199
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by back2future » Fri Nov 09, 2018 11:22 am

Someone did "reboot" inside docker container. How does a container react to that command?
Are there hints for common dos and don'ts for usage with docker containers from experienced users?
Thx

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Fri Nov 09, 2018 11:42 am

back2future wrote:Someone did "reboot" inside docker container. How does a container react to that command?
Are there hints for common dos and don'ts for usage with docker containers from experienced users?
Thx
Probably not, two of them were rebooted twice...last week due to power fault in this region and early this week for experimental with them.
Sorry for the confusion.

back2future
Posts: 199
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by back2future » Fri Nov 09, 2018 11:54 am

Thanks for Your explanations.
What i did not understand: If there is "reboot" command from inside a container, will that container restart like a common os and is there all change to root filesystem reset?
(Often reboot is recommended after apt-get update, apt-get upgrade, but could be unwise within docker containers then?)

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Fri Nov 09, 2018 12:00 pm

** Note **
One ODROID-H2 (w/ DDR4 32GB) is restarted with new docker option --shm-size="6g" as per the request.

Currently running ODROID-H2 devices:
  • Unit_____SSH Port______DDR4
  • #1_______2240_________Samsung (4G + 4G)
  • #2_______2242_________Essencore (4G + 4G)
  • #3_______2244_________Samsung (16G + 16G)

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Fri Nov 09, 2018 12:04 pm

back2future wrote:Thanks for Your explanations.
What i did not understand: If there is "reboot" command from inside a container, will that container restart like a common os and is there all change to root filesystem reset?
(Often reboot is recommended after apt-get update, apt-get upgrade, but could be unwise within docker containers then?)
Oh, I see...basically such system command does not work in the container even though you obtain the root permission since UID of the 'root' is not real 'root' of the host. Also, I didn't build the docker image to have permanant storage or volume for the root file system which means restarting a container will be completely rolled back like it was started at the first time. If you think a certain package should be installed in the docker container, please let me know....then I would rebuild the container and restart it. But, basically since one like you could have a doubt of stability of ODROID-H2, I do not recommend to reboot the container...or I have to be diligent. :)

back2future
Posts: 199
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by back2future » Fri Nov 09, 2018 12:20 pm

tobetter wrote:
back2future wrote:Thanks for Your explanations.
What i did not understand: If there is "reboot" command from inside a container, will that container restart like a common os and is there all change to root filesystem reset?
(Often reboot is recommended after apt-get update, apt-get upgrade, but could be unwise within docker containers then?)
Oh, I see...basically such system command does not work in the container even though you obtain the root permission since UID of the 'root' is not real 'root' of the host. Also, I didn't build the docker image to have permanant storage or volume for the root file system which means restarting a container will be completely rolled back like it was started at the first time. If you think a certain package should be installed in the docker container, please let me know....then I would rebuild the container and restart it. But, basically since one like you could have a doubt of stability of ODROID-H2, I do not recommend to reboot the container...or I have to be diligent. :)
Ok, i see.
One thing that made me think about rebooting, was cron service not (really) doing periodical tasks. It occured to me, that if one communication port is blocked (by whatever reason), cron could restart a ssh server every (maybe) 30 minutes for stability of access?
For testing I tried to push sensors output (package lm-sensors) to a file in shared memory (/dev/shm/sensors_log), but cron did that task not once, nor every minute?

[ Found "Getting started guide" for setting up docker containers in Odroid Magazine ( https://magazine.odroid.com/article/odr ... ted-guide/ ) but not that much information about handling a docker container from user side and useful possibilities therefore. Worth an article for Odroid Magazine? ]

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Fri Nov 09, 2018 12:39 pm

back2future wrote:
tobetter wrote:
back2future wrote:Thanks for Your explanations.
What i did not understand: If there is "reboot" command from inside a container, will that container restart like a common os and is there all change to root filesystem reset?
(Often reboot is recommended after apt-get update, apt-get upgrade, but could be unwise within docker containers then?)
Oh, I see...basically such system command does not work in the container even though you obtain the root permission since UID of the 'root' is not real 'root' of the host. Also, I didn't build the docker image to have permanant storage or volume for the root file system which means restarting a container will be completely rolled back like it was started at the first time. If you think a certain package should be installed in the docker container, please let me know....then I would rebuild the container and restart it. But, basically since one like you could have a doubt of stability of ODROID-H2, I do not recommend to reboot the container...or I have to be diligent. :)
Ok, i see.
One thing that made me think about rebooting, was cron service not (really) doing periodical tasks. It occured to me, that if one communication port is blocked (by whatever reason), cron could restart a ssh server every (maybe) 30 minutes for stability of access?
For testing I tried to push sensors output (package lm-sensors) to a file in shared memory (/dev/shm/sensors_log), but cron did that task not once, nor every minute?

[ Found "Getting started guide" for setting up docker containers in Odroid Magazine ( https://magazine.odroid.com/article/odr ... ted-guide/ ) but not that much information about handling a docker container from user side and useful possibilities therefore. Worth an article for Odroid Magazine? ]
The way what I set up the bench is different with the one you read from the magazine. That is likely about how to manage the distributed system to manage the system load and the machines on the bench are running as an individual system. The reason why I use the docker for the machines is that if one does corrupt a system whatever the reason is, I would be tied to the bench to bring it up, format, install, set up...blahblah. So the users who visit the bench should do anything whatever they want to do but not allowed to compromise the system. Even though the system in the container is corrupted, I just need to reboot the hardware...then all set. :)
The technique what I use to set up the bench is simple, basic knowledge to write 'Dockerfile' and couple of arguments of 'docker run'.
These users thanked the author tobetter for the post:
back2future (Wed Apr 10, 2019 8:11 am)

elatllat
Posts: 1224
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1
Has thanked: 0
Been thanked: 2 times
Contact:

Re: "ODROID Bench"

Unread post by elatllat » Sat Nov 24, 2018 11:18 am

40 and 35 are down and apt is broken on 20 and 21, 31 / is full. 44 worked.

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Mon Nov 26, 2018 1:10 pm

elatllat wrote:40 and 35 are down and apt is broken on 20 and 21, 31 / is full. 44 worked.
Thank you for the update, I was able to manage them today and found some errors on ARM based devices.

Available ports for ODROID-H2 devices are:
  • 2240
    2242
    2244
Another device 35 is fine, it only accepts the connection through a web browser and "NextCloud" service is running.

Other devices, ODROID-XU4 and ODROID-C2 are still down. Actually, they are failing at apt even the host system is the same. Looking into the issue. Let me keep updating regarding if any.

EDIT:
ODROID-C2 with 31 is back with new space, now 59% of the root file system is available.

elatllat
Posts: 1224
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1
Has thanked: 0
Been thanked: 2 times
Contact:

Re: "ODROID Bench"

Unread post by elatllat » Tue Dec 04, 2018 1:24 am

tobetter wrote:...still down...
Can you put them all back on the same subnet/port so it's easier to find the ones that are not broken?

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Tue Dec 04, 2018 1:43 am

elatllat wrote:
tobetter wrote:...still down...
Can you put them all back on the same subnet/port so it's easier to find the ones that are not broken?
All working ODROID devices are in the same subnet and have the same port number as configured. Two devices are not there, port number 2220 and 2230 unfortunately.
Did you see any device which is down now?

elatllat
Posts: 1224
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1
Has thanked: 0
Been thanked: 2 times
Contact:

Re: "ODROID Bench"

Unread post by elatllat » Tue Dec 04, 2018 2:36 am

They all have the same IP (172.17.0.2) so they are not on the same subnet.

I don't know what should be up but ssh is at least partly working on 21, 31, 40, 42, and 44.

Code: Select all

for X in $(seq 1 99) ; do ssh-keyscan -p 22$X maze.odroid.com >> known_odroid_hosts; done; perl -pe 's/ .*//g;s/.*:22//g' known_odroid_hosts |sort -u

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Tue Dec 04, 2018 2:51 am

elatllat wrote:They all have the same IP (172.17.0.2) so they are not on the same subnet.

I don't know what should be up but ssh is at least partly working on 21, 31, 40, 42, and 44.

Code: Select all

for X in $(seq 1 99) ; do ssh-keyscan -p 22$X maze.odroid.com >> known_odroid_hosts; done; perl -pe 's/ .*//g;s/.*:22//g' known_odroid_hosts |sort -u
The IP 172.17.0.2 is for virtual NIC being used by the docker container.
If you like to access a device after getting into a device in maze.odroid.com, you can commonly use the port number 2222 with IP address 192.168.0.xx where xx is the device number like 21, 31, 40, 42 and 44.

Assuming that you in 192.168.0.21 with this command.

Code: Select all

ssh -p 2221 odroid@maze.odroid.com
And now if you like to jump to another device H2, their device numbers are 40, 42 and 44, you can use 2222 for the port number.

Code: Select all

ssh -p 2222 odroid@192.168.0.40
You might be curious why 2222 instead of 22 which is the default ssh port number. Since 22 is being used by ssh server in the host and 2222 is for ssh server in the docker. And 22 is blocked to be accessed since I don't like the containers are modified by the users.

elatllat
Posts: 1224
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1
Has thanked: 0
Been thanked: 2 times
Contact:

Re: "ODROID Bench"

Unread post by elatllat » Tue Dec 04, 2018 3:05 am

I see they are all sharing the same parent subnet. Guess the issues between now and the first time I ran nmap were not related. Thanks for clarifying.

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Tue Dec 04, 2018 3:15 am

elatllat wrote:I see they are all sharing the same parent subnet. Guess the issues between now and the first time I ran nmap were not related. Thanks for clarifying.
No problem, it might be confusing...that's why it's maze.odroid.com.

back2future
Posts: 199
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by back2future » Sat Feb 02, 2019 3:28 pm

Would be interesting, if we could have output from

Code: Select all

 modprobe eeprom && decode-dimms 
[ perl script for DDR4: https://kernel.googlesource.com/pub/scm ... code-dimms ]

for H2's ram so-dimms on port 2240,2243,2244?

lungfish
Posts: 16
Joined: Tue Feb 19, 2019 8:13 pm
languages_spoken: english
ODROIDs: C2, XU4, N2 (soon).
Has thanked: 0
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by lungfish » Thu Feb 21, 2019 7:16 pm

Not sure how feasible this is, but would it be possible to allow people, on request, to run dev tools on some of the systems? I've been playing with cryptodev (see this thread) and would like to try it on other platforms, but that requires building both cryptodev and loading the driver to create /dev/crypto, and building and running the diagnostic software that uses it. This required a pretty full build environment, including kernel sources, to be installed on the system.

I realise this may not be practical, just wondering...

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Thu Feb 21, 2019 7:25 pm

ODROID-N2 boards are ready now

Now, 4 ODROID-N2 boards are sitting on the bench waiting for users.

In order to access ODROID-N2 command line, please do ssh with this command. 4 port numbers, 2226 / 2227 / 2228 / 2229, are dedicated to ODROID-N2. The access account and password are all same as other odroid / odroid.

Code: Select all

ssh -p < 2226 | 2227 | 2228 | 2229 > odroid@maze.odroid.com
If you just like to monitor how is the system load or temperature, please visit web pages of them.
* http://maze.odroid.com:9926
* http://maze.odroid.com:9927
* http://maze.odroid.com:9928
* http://maze.odroid.com:9929

Enjoy...
Last edited by tobetter on Thu Feb 21, 2019 7:42 pm, edited 1 time in total.

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Thu Feb 21, 2019 7:28 pm

lungfish wrote:
Thu Feb 21, 2019 7:16 pm
Not sure how feasible this is, but would it be possible to allow people, on request, to run dev tools on some of the systems? I've been playing with cryptodev (see this thread) and would like to try it on other platforms, but that requires building both cryptodev and loading the driver to create /dev/crypto, and building and running the diagnostic software that uses it. This required a pretty full build environment, including kernel sources, to be installed on the system.

I realise this may not be practical, just wondering...
Basically, you can do on the bench as long as the system is not that busy. You could be able to install the tools what you need for your work, you can try and keep posting if you need some support. As long as I am not quite busy, I would be able to customize the container.

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Thu Feb 21, 2019 7:29 pm

back2future wrote:
Sat Feb 02, 2019 3:28 pm
Would be interesting, if we could have output from

Code: Select all

 modprobe eeprom && decode-dimms 
[ perl script for DDR4: https://kernel.googlesource.com/pub/scm ... code-dimms ]

for H2's ram so-dimms on port 2240,2243,2244?
Sorry for long wait, now you would be able to run the command decode-dimms on H2. Please try... :)

lungfish
Posts: 16
Joined: Tue Feb 19, 2019 8:13 pm
languages_spoken: english
ODROIDs: C2, XU4, N2 (soon).
Has thanked: 0
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by lungfish » Thu Feb 21, 2019 7:47 pm

tobetter wrote:
Thu Feb 21, 2019 7:28 pm
Basically, you can do on the bench as long as the system is not that busy. You could be able to install the tools what you need for your work, you can try and keep posting if you need some support. As long as I am not quite busy, I would be able to customize the container.
Just to check, this would require installing an awful lot of stuff, gcc and the full dev environment and kernel sources. Just wanted to make sure this won't upset anyone...

User avatar
tobetter
Posts: 3039
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 9 times
Been thanked: 22 times
Contact:

Re: "ODROID Bench"

Unread post by tobetter » Thu Feb 21, 2019 7:49 pm

lungfish wrote:
Thu Feb 21, 2019 7:47 pm
tobetter wrote:
Thu Feb 21, 2019 7:28 pm
Basically, you can do on the bench as long as the system is not that busy. You could be able to install the tools what you need for your work, you can try and keep posting if you need some support. As long as I am not quite busy, I would be able to customize the container.
Just to check, this would require installing an awful lot of stuff, gcc and the full dev environment and kernel sources. Just wanted to make sure this won't upset anyone...
Rather than space, I think you cannot test kernel since the devices on ODROID Benches are only allowed to access in Docker container wise.

lungfish
Posts: 16
Joined: Tue Feb 19, 2019 8:13 pm
languages_spoken: english
ODROIDs: C2, XU4, N2 (soon).
Has thanked: 0
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by lungfish » Thu Feb 21, 2019 7:57 pm

tobetter wrote:
Thu Feb 21, 2019 7:49 pm
Rather than space, I think you cannot test kernel since the devices on ODROID Benches are only allowed to access in Docker container wise.
Hmm, OK. Let's see what insmod lets me do from the container. Worst case I could build it in the container and... can it be exported/loaded from outside? Just working on it now, will report back, it still needs additional tools installed to start the build process.

lungfish
Posts: 16
Joined: Tue Feb 19, 2019 8:13 pm
languages_spoken: english
ODROIDs: C2, XU4, N2 (soon).
Has thanked: 0
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by lungfish » Thu Feb 21, 2019 8:22 pm

OK, ran into a problem, uname -r reports kernel 3.16.57-25 (on the C2 systems), but an apt-cache search of available headers indicates only 4.9.0.* headers are available. This means it's more or less a dead-end, it's possible there's some repository out there that still has the old headers but finding those things is typically a pain.

elatllat
Posts: 1224
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1
Has thanked: 0
Been thanked: 2 times
Contact:

Re: "ODROID Bench"

Unread post by elatllat » Thu Feb 21, 2019 9:51 pm

Code: Select all

git clone --depth 1 --branch odroidc2-v3.16.y https://github.com/hardkernel/linux
make odroidc2_defconfig
make headers_install
Last edited by elatllat on Thu Feb 21, 2019 10:12 pm, edited 1 time in total.

lungfish
Posts: 16
Joined: Tue Feb 19, 2019 8:13 pm
languages_spoken: english
ODROIDs: C2, XU4, N2 (soon).
Has thanked: 0
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by lungfish » Thu Feb 21, 2019 10:08 pm

$ git clone --depth 1 --branch odroidc2-v3.16 https://github.com/hardkernel/linux
Cloning into 'linux'...
warning: Could not find remote branch odroidc2-v3.16 to clone.
fatal: Remote branch odroidc2-v3.16 not found in upstream origin
There's only 3.14. If all else fails I'll try that, but mixing headers from different versions is usually a recipe for disaster.

elatllat
Posts: 1224
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1
Has thanked: 0
Been thanked: 2 times
Contact:

Re: "ODROID Bench"

Unread post by elatllat » Thu Feb 21, 2019 10:13 pm

I missed the .y (it's odroidc2-v3.16.y )

lungfish
Posts: 16
Joined: Tue Feb 19, 2019 8:13 pm
languages_spoken: english
ODROIDs: C2, XU4, N2 (soon).
Has thanked: 0
Been thanked: 0
Contact:

Re: "ODROID Bench"

Unread post by lungfish » Thu Feb 21, 2019 10:28 pm

That worked, thanks! Sorry to be a pain, but what's the equivalent for setting up /lib/modules? It's normally set up by the install linux-headers-xxx, but grabbing the headers from the git repository bypasses this. Specifically, /usr/src/linux is a broken link so I can't create a link to /lib/modules/3.16.57-25/build from it. sudo ln -s /home/odroid/linux/ /lib/modules/3.16.57-25/build gets me a bit further, but just leads to breakage later on due to other things not being set up properly.
Last edited by lungfish on Thu Feb 21, 2019 10:42 pm, edited 1 time in total.

elatllat
Posts: 1224
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1
Has thanked: 0
Been thanked: 2 times
Contact:

Re: "ODROID Bench"

Unread post by elatllat » Thu Feb 21, 2019 10:37 pm

Code: Select all

make modules
but you should already have that as you can't run the kernel without it, and the version of running kernel should match the git unless you happened to grab it just before an upgrade.

Post Reply

Return to “News”

Who is online

Users browsing this forum: No registered users and 1 guest