Slow network connection to Nextcloud

Post Reply
joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Slow network connection to Nextcloud

Unread post by joerg » Fri Feb 15, 2019 9:53 pm

Hello all,
I have a issue with a very slow answer for a caldav request to Nextcloud only from one C1 running with Ubuntu 18.04 minimal with Home Assistant.
The Nextcloud is running on my N1 with OMV. When I make this request from my main Linux machine I get answer within 0.28 seconds, when I make this request from the C1 I get answer within 25 seconds.
I have another C1 running with Ubuntu 18.04 minimal where only a small prog is running that collects all my RF24 sensors and send it via mqtt messages to the slow C1 with Home Assistant. When I make this request from this C1 I also get the answer fast within 0.38 seconds. The odd is that I had the same issue weeks ago and after a lot of reboots and updates it was running fast again, but I never found the reason why.
Now after having a lot of trouble with updating Home Assistant, I find this issue again. Maybe there is one network crack out there, who can give some hints how to debug this further?

From the C1 with this issue:

Code: Select all

time ./calendar_events.sh user passw
{"event": "none"}

real	0m25,684s
user	0m0,070s
sys	0m0,020s
From the other C1:

Code: Select all

time ./calendar_events.sh user passw
{"event": "none"}

real	0m0,388s
user	0m0,030s
sys	0m0,030s
Ping is fast:

Code: Select all

ping 192.168.1.15
PING 192.168.1.15 (192.168.1.15) 56(84) bytes of data.
64 bytes from 192.168.1.15: icmp_seq=1 ttl=64 time=0.666 ms
64 bytes from 192.168.1.15: icmp_seq=2 ttl=64 time=0.866 ms
64 bytes from 192.168.1.15: icmp_seq=3 ttl=64 time=0.673 ms
^C
Netstat, a lot of connections:

Code: Select all

netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 localhost:8883          localhost:56396         ESTABLISHED
tcp        0      0 odroid:1883             AlarmCentral.frit:57887 TIME_WAIT  
tcp        0      0 odroid:56578            JW-NAS.fritz.box:90     TIME_WAIT  
tcp        0      0 localhost:57064         localhost:1883          ESTABLISHED
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:55198 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:43641 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:57231 TIME_WAIT  
tcp        0      0 odroid:ssh              JW-Mint.fritz.box:51250 ESTABLISHED
tcp        0      0 localhost:42506         localhost:42657         ESTABLISHED
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:54293 ESTABLISHED
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:40840 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:32832 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:47298 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:60785 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:49092 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:34019 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:45284 TIME_WAIT  
tcp        0      0 localhost:56396         localhost:8883          ESTABLISHED
tcp        0      0 odroid:ssh              JW-Mint.fritz.box:51382 ESTABLISHED
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:52894 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:54642 TIME_WAIT  
tcp        0      0 localhost:8883          localhost:37001         ESTABLISHED
tcp        0      0 odroid:1883             AlarmCentral.frit:42064 TIME_WAIT  
tcp        0      0 localhost:37001         localhost:8883          ESTABLISHED
tcp        0      0 odroid:1883             AlarmCentral.frit:36048 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:51235 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:58973 TIME_WAIT  
tcp        0      0 localhost:8883          localhost:56394         ESTABLISHED
tcp        0      0 odroid:1883             AlarmCentral.frit:59053 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:51878 TIME_WAIT  
tcp        0      0 odroid:8123             JW-Mint.fritz.box:46718 ESTABLISHED
tcp        0      0 odroid:1883             AlarmCentral.frit:55203 TIME_WAIT  
tcp        0      0 localhost:56395         localhost:8883          ESTABLISHED
tcp        0      0 odroid:1883             AlarmCentral.frit:49114 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:37714 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:56786 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:57505 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:41539 TIME_WAIT  
tcp        0      0 odroid:40894            TX-NR545.fritz.bo:60128 ESTABLISHED
tcp        0      0 odroid:1883             AlarmCentral.frit:47017 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:50148 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:59444 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:51259 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:33239 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:53162 TIME_WAIT  
tcp        0      0 localhost:42657         localhost:42506         ESTABLISHED
tcp        0      0 odroid:nfs              JW-Mint.fritz.box:1009  ESTABLISHED
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:37812 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:56810 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:47590 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:53371 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:37580 TIME_WAIT  
tcp        0    116 odroid:ssh              JW-Mint.fritz.box:51522 ESTABLISHED
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:58651 TIME_WAIT  
tcp        0      0 localhost:56394         localhost:8883          ESTABLISHED
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:33168 TIME_WAIT  
tcp        0      0 odroid:1883             AlarmCentral.frit:60798 TIME_WAIT  
tcp        0      0 localhost:8883          localhost:56395         ESTABLISHED
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:48528 TIME_WAIT  
tcp        0      0 odroid:58146            VELUX-KLF-399B.fr:51200 ESTABLISHED
tcp        0      0 odroid:8883             odroid.fritz.box:46418  ESTABLISHED
tcp        0      0 odroid:1883             AlarmCentral.frit:37490 TIME_WAIT  
tcp        0      0 localhost:1883          localhost:57064         ESTABLISHED
tcp        0      0 odroid:1883             AlarmCentral.frit:44010 TIME_WAIT  
tcp        0      0 odroid:1883             Lenovo-TAB10.frit:55531 TIME_WAIT  
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  8      [ ]         DGRAM                    5390     /run/systemd/journal/socket
unix  2      [ ]         DGRAM                    308250   /run/user/1000/systemd/notify
unix  19     [ ]         DGRAM                    5428     /run/systemd/journal/dev-log
unix  2      [ ]         DGRAM                    5440     /run/systemd/journal/syslog
unix  3      [ ]         DGRAM                    5339     /run/systemd/notify
unix  2      [ ]         DGRAM                    5341     /run/systemd/cgroups-agent
unix  2      [ ]         DGRAM                    301985   
unix  2      [ ]         DGRAM                    9230     
unix  3      [ ]         STREAM     CONNECTED     8752     /var/run/dbus/system_bus_socket
unix  3      [ ]         DGRAM                    5343     
unix  3      [ ]         STREAM     CONNECTED     9276     
unix  3      [ ]         STREAM     CONNECTED     359748   
unix  3      [ ]         STREAM     CONNECTED     8751     
unix  3      [ ]         STREAM     CONNECTED     9058     /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    296686   
unix  3      [ ]         STREAM     CONNECTED     335459   
unix  2      [ ]         DGRAM                    3584     
unix  3      [ ]         STREAM     CONNECTED     306758   
unix  3      [ ]         STREAM     CONNECTED     359551   /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     310248   
unix  3      [ ]         STREAM     CONNECTED     8732     
unix  3      [ ]         STREAM     CONNECTED     9372     
unix  3      [ ]         STREAM     CONNECTED     7769     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     7861     
unix  3      [ ]         STREAM     CONNECTED     326347   
unix  2      [ ]         DGRAM                    8716     
unix  3      [ ]         STREAM     CONNECTED     310247   
unix  3      [ ]         DGRAM                    9235     
unix  3      [ ]         STREAM     CONNECTED     7574     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     359114   /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     8660     
unix  3      [ ]         DGRAM                    9237     
unix  2      [ ]         DGRAM                    301997   
unix  3      [ ]         STREAM     CONNECTED     7623     
unix  3      [ ]         DGRAM                    8580     
unix  3      [ ]         STREAM     CONNECTED     5863     
unix  3      [ ]         STREAM     CONNECTED     326346   
unix  3      [ ]         DGRAM                    308253   
unix  2      [ ]         DGRAM                    8583     
unix  2      [ ]         DGRAM                    359749   
unix  2      [ ]         DGRAM                    314730   
unix  2      [ ]         DGRAM                    314675   
unix  3      [ ]         STREAM     CONNECTED     8710     
unix  3      [ ]         DGRAM                    4749     
unix  3      [ ]         STREAM     CONNECTED     2872     /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    9025     
unix  3      [ ]         STREAM     CONNECTED     359098   /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    326338   
unix  3      [ ]         STREAM     CONNECTED     10347    
unix  3      [ ]         STREAM     CONNECTED     7644     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     7642     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     6520     
unix  3      [ ]         STREAM     CONNECTED     6449     
unix  3      [ ]         STREAM     CONNECTED     6724     
unix  2      [ ]         DGRAM                    6725     
unix  3      [ ]         STREAM     CONNECTED     359680   /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     359643   /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    6729     
unix  3      [ ]         STREAM     CONNECTED     358962   
unix  2      [ ]         DGRAM                    331101   
unix  3      [ ]         STREAM     CONNECTED     8721     
unix  3      [ ]         DGRAM                    5344     
unix  3      [ ]         STREAM     CONNECTED     9277     
unix  3      [ ]         STREAM     CONNECTED     335460   /var/run/mysqld/mysqld.sock
unix  2      [ ]         DGRAM                    306176   
unix  3      [ ]         STREAM     CONNECTED     10299    
unix  3      [ ]         STREAM     CONNECTED     335454   
unix  3      [ ]         STREAM     CONNECTED     358212   
unix  3      [ ]         DGRAM                    9238     
unix  3      [ ]         STREAM     CONNECTED     358051   
unix  2      [ ]         DGRAM                    8730     
unix  2      [ ]         DGRAM                    9281     
unix  2      [ ]         DGRAM                    359789   
unix  3      [ ]         STREAM     CONNECTED     335453   
unix  2      [ ]         DGRAM                    7617     
unix  3      [ ]         STREAM     CONNECTED     6741     /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     9374     /var/run/dbus/system_bus_socket
unix  2      [ ]         STREAM     CONNECTED     9203     
unix  3      [ ]         STREAM     CONNECTED     9481     /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     7573     
unix  3      [ ]         DGRAM                    9236     
unix  2      [ ]         DGRAM                    358097   
unix  3      [ ]         STREAM     CONNECTED     7522     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     7615     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     7521     
unix  3      [ ]         STREAM     CONNECTED     9477     
unix  3      [ ]         STREAM     CONNECTED     9278     /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    5770     
unix  3      [ ]         STREAM     CONNECTED     301978   
unix  3      [ ]         STREAM     CONNECTED     7709     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     9279     /var/run/dbus/system_bus_socket
unix  3      [ ]         DGRAM                    8581     
unix  2      [ ]         DGRAM                    14543    
unix  3      [ ]         STREAM     CONNECTED     7624     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     301979   /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     11345    /var/run/dbus/system_bus_socket
unix  3      [ ]         DGRAM                    4748     
unix  3      [ ]         STREAM     CONNECTED     306759   
unix  3      [ ]         DGRAM                    308252   
unix  3      [ ]         STREAM     CONNECTED     8610     
unix  2      [ ]         DGRAM                    359682   
unix  3      [ ]         STREAM     CONNECTED     6622     
unix  3      [ ]         STREAM     CONNECTED     6571     
unix  3      [ ]         STREAM     CONNECTED     9280     /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     7457     /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    2874     
unix  3      [ ]         STREAM     CONNECTED     7750     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     359046  
Top:

Code: Select all

top

top - 13:43:10 up 1 day, 19:15,  1 user,  load average: 0,60, 0,57, 0,49
Tasks: 121 total,   2 running, 119 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2,9 us,  3,1 sy,  0,0 ni, 93,8 id,  0,3 wa,  0,0 hi,  0,0 si,  0,0 st
KiB Mem :   823424 total,   110912 free,   362748 used,   349764 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   444997 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
17494 homeass+  20   0  260972  53616   5112 S  10,2  6,5   8:03.87 hass        
  259 root      20   0       0      0      0 S   4,6  0,0  37:07.78 w1_bus_mas+ 
17525 root      20   0   35100   3292   2684 S   3,3  0,4   1:51.23 wiring-mqtt 
   62 root      rt   0       0      0      0 S   2,3  0,0  55:53.11 cfinteract+ 
  209 root      19  -1  450412 156976 156000 S   1,6 19,1  33:10.51 systemd-jo+ 
17664 joerg     20   0    6644   1396    964 R   1,6  0,2   0:00.10 top         
   76 root     -51   0       0      0      0 S   0,7  0,0  15:28.97 irq/110-sd+ 
  351 syslog    20   0   24912   1544    360 S   0,7  0,2  11:50.34 rsyslogd    
  500 mysql     20   0  662216 187036   2620 S   0,7 22,7  24:16.70 mysqld      
13556 mosquit+  20   0    5936   2236   1524 S   0,7  0,3   3:33.91 mosquitto   
17316 root      20   0       0      0      0 S   0,7  0,0   1:37.16 kworker/0:2 
17532 root      20   0  140164   3432   2744 S   0,7  0,4   0:23.99 cover-mqtt  
17539 root      20   0   34068   3304   2704 S   0,7  0,4   0:24.98 pwm8-mqtt   
17547 root      20   0   35732   2320   1940 S   0,7  0,3   0:48.53 wdog-mqtt   
   67 root      20   0       0      0      0 S   0,3  0,0  13:24.24 kthread_di  
17613 root      20   0       0      0      0 S   0,3  0,0   0:01.30 kworker/2:1 
    1 root      20   0   29236   3504   1536 S   0,0  0,4   0:24.06 systemd
For completeness, the calendar_events.sh:

Code: Select all

#!/bin/bash

username=$1
password=$2
dav_server="http://192.168.1.15:90"
dav_user_calendar_path="/remote.php/dav/calendars/joerg/abfuhrtermine/"

# Define start/end date
date_start=$(date  +"%Y%m%dT000000" -d tomorrow)
date_end=$(date  +"%Y%m%dT235959" -d tomorrow)

# Tomorrow's events
events=$(curl --silent \
                    --request REPORT \
                    --header "Depth: 1" \
                    --header "Content-Type: text/xml" \
                    --data '<c:calendar-query xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav"> 
                              <d:prop><d:getetag /><c:calendar-data /></d:prop>
                              <c:filter>
                                <c:comp-filter name="VCALENDAR">
                                  <c:comp-filter name="VEVENT">
                                     <c:time-range  start="'${date_start}'" end="'${date_end}'"/>
                                  </c:comp-filter>
                                </c:comp-filter>
                              </c:filter>
                            </c:calendar-query>' \
                    --basic \
                    --user ${username}:${password} \
                    ${dav_server}${dav_user_calendar_path}| \
		    xmlstarlet sel -t -v 'd:multistatus/d:response/d:propstat/d:prop/cal:calendar-data' -n | grep ^SUMMARY )

events_count=$(echo "$events" | grep -c ^SUMMARY)

# Print events in Json format
    if [ "$events_count" -ge "1" ]; then
      printf '{"event": "'
      printf "$events" |  awk '{split($0, a, ":"); print a[2]}' | while read event; do
      printf '%s' "$event "
      done
      printf '"}\n'
    else
      printf '{"event": "%s"}\n' "none"
    fi
BTW, this script requests events for tomorrow and puts them out in JSON format.

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Mon Feb 18, 2019 4:52 pm

Nobody with some idea?

User avatar
rooted
Posts: 6878
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 267 times
Been thanked: 58 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by rooted » Mon Feb 18, 2019 4:57 pm

If you swap SD/eMMC between the two C1 does it still exhibit this slow response issue? Does the other C1 now have the problem?

If so, swap network cables and try again. If issue persists swap power supplies.

Doing this will let you know if it's hardware.

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Mon Feb 18, 2019 5:11 pm

Thank you @rooted, I will check this in the afternoon.

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Tue Feb 19, 2019 2:12 am

Hm, very odd. When I checked again, without to change anything, I get a fast response now:

Code: Select all

time ./calendar_events.sh user passw
{"event": "Graue Tonne"}

real	0m0,447s
user	0m0,040s
sys	0m0,050s

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Wed Feb 20, 2019 1:34 am

Now it is again slow in response. Before I restarted several times home assistant, then suddenly it became slow.
I replaced the C1 by a spare C1+, but it doesn't help, it stays slow. So I am sure it is not hardware related.

User avatar
rooted
Posts: 6878
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 267 times
Been thanked: 58 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by rooted » Wed Feb 20, 2019 3:49 am

Is power savings enabled on the LAN?

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Wed Feb 20, 2019 4:10 am

If this is what you meant? Wake-On-Lan? Seems to be disabled:

Code: Select all

$ sudo ethtool eth0
Settings for eth0:
	Supported ports: [ TP MII ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Half 1000baseT/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Half 1000baseT/Full 
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Port: MII
	PHYAD: 1
	Transceiver: external
	Auto-negotiation: on
	Supports Wake-on: d
	Wake-on: d
	Link detected: yes

User avatar
rooted
Posts: 6878
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 267 times
Been thanked: 58 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by rooted » Wed Feb 20, 2019 4:23 am

I was thinking the LAN has a power savings mode that kicked it down to 10/100 but I guess it doesn't.

You could try setting the LAN speed to 100 megabit, other than that I'm out of ideas on the C1 side.

There is always a chance it is some weirdness in the router.

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Wed Feb 20, 2019 5:42 pm

It begins to drive me crazy. As per log of Home Assistant in the night at 1:50 am there was the last error message due to response time. Now it's fast again.
Yesterday, I changed the command in Home assistant with a '/usr/bin/timeout 3' at the beginning. This way it will not block the Home Assistant to much, but I will get empty response when the timeout happens.

Code: Select all

Wed Feb 20 2019 01:50:49 GMT+0100 (Central European Standard Time)
Empty reply found when expecting JSON data
Command failed: /usr/bin/timeout 3 /home/homeassistant/.homeassistant/scripts/calendar_events.sh joerg passw

User avatar
mad_ady
Posts: 6860
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 236 times
Been thanked: 177 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by mad_ady » Thu Feb 21, 2019 3:23 am

To see what happens I suggest the following:
* Run your script with bash -x for debugging and see where it seems to hang (to isolate the problem)
* Run a tcpdump -n -i eth0 -s 0 -w /tmp/webdav.pcap host ip.of.webdav.server. Analyze the pcap in wireshark to see where the delays were. You should see dropped packets/retransmisions if any
* Reboot your c1 before the test and after run netstat -s - it will tell you tcp/ip stats (corrupt packets/retransmissions).
* check that the local dns is set correctly on all hosts. You can get such big delays when ssh or mysql try to do a reverse dns query to figure out the name of the host which connected
* check server apache/php error logs if the delay is server side.

Bonus question: what are you using your calendar in home assistant for? Maybe I could use it as well...

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Thu Feb 21, 2019 6:10 pm

Hello @mad_ady, thank you for this. For now it works, but when next time this issue occurs, I will follow this.
To your bonus question:
Generaly I have created this calendar in Nextcloud to disconnect all my mobile devices from Google. You can call me paranoid. But it's my target not to be synchronized with the Google cloud, but with my own cloud. On my mobile devices I have installed davDroid to synchronize to Nextcloud.
With this working I had the idea to implement a notification in Home Assistant. Here in Germany specially in my city the waste company want that we place the trashcan near the street at the day they come, but they come very early and only every 14 days and sometimes because of celebration days, the day change. So I made this script to request the events for tomorrow from the waste calendar. The waste calendar I got from the waste company and have it imported to Nextcloud.
So at 19:30pm Home Assistant is sending via a Telegram Bot that tomorrow will be the waste collection and we can put the trashcan to the street right at the evening. So often we have forgotten it and had to wait another two weeks... :x

Code: Select all

calendar:
  - platform: caldav
    username: joerg
    password: !secret caldav_password
    url: http://192.168.1.15:90/remote.php/dav/
    custom_calendars:
      - name: Muell
        calendar: Abfuhrtermine
        search: 'Tonne'

telegram_bot:
  - platform: broadcast
    api_key: !secret telegram_api
    allowed_chat_ids:
      - 2349xxxxx
      - 4058xxxxx

notify:
  - name: Joerg
    platform: telegram
    chat_id: 2349xxxxx

sensor:
  - platform: command_line
    name: get events
    scan_interval: 60
    json_attributes:
      - event
    command: /usr/bin/timeout 3 /home/homeassistant/.homeassistant/scripts/calendar_events.sh {{ states.sensor.caldav_user.state }} {{ states.sensor.caldav_password.state }}
    value_template: '{{ value_json.event }}'
    
automation:
  - alias: 'send message'
    trigger:
      platform: time
      at: '19:30:00'
    condition:
      - condition: template
        value_template: '{{ states.sensor.get_events.state | string != "none" }}'
    action:
      - service: telegram_bot.send_message
        data_template:
          title: 'Erinnerung Müllabfuhr'
          target: '2349xxxxx'
          message: '{{ states.sensor.get_events.state | string }}'
      - service: telegram_bot.send_message
        data_template:
          title: 'Erinnerung Müllabfuhr'
          target: '4058xxxxxx'
          message: '{{ states.sensor.get_events.state | string }}'
Bildschirmfoto vom 2019-02-21 09-54-09.png
Bildschirmfoto vom 2019-02-21 09-54-09.png (25.37 KiB) Viewed 1586 times

User avatar
mad_ady
Posts: 6860
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 236 times
Been thanked: 177 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by mad_ady » Thu Feb 21, 2019 6:29 pm

Nice setup. But you're missing something... You need to add a personal "robot" to move the trash can for you. Only then will home automation pay for itself :D
Thanks for sharing.

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Thu Feb 21, 2019 8:27 pm

mad_ady wrote:
Thu Feb 21, 2019 6:29 pm
But you're missing something... You need to add a personal "robot" to move the trash can for you.
Ahh, maybe I can construct one by myself with the N2 received 5 minutes ago... :)

User avatar
rooted
Posts: 6878
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 267 times
Been thanked: 58 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by rooted » Thu Feb 21, 2019 8:54 pm

I'm glad you got an N2 :)

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Mon Nov 18, 2019 10:08 pm

Now I am again fighting with this delayed response.
I have updated my Homeassistant C1 that was running hardkernels ubuntu now with the one I prepared with armbian and mainline kernel.
Everything works fine except the webdav request.
Here is the netstat -s output before and after the request,
Before:

Code: Select all

Ip:
    Forwarding: 2
    7402 total packets received
    0 forwarded
    0 incoming packets discarded
    7402 incoming packets delivered
    5308 requests sent out
Icmp:
    0 ICMP messages received
    0 input ICMP message failed
    ICMP input histogram:
    0 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
Tcp:
    124 active connection openings
    384 passive connection openings
    1 failed connection attempts
    62 connection resets received
    8 connections established
    5035 segments received
    4405 segments sent out
    9 segments retransmitted
    0 bad segments received
    80 resets sent
Udp:
    2276 packets received
    0 packets to unknown port received
    0 packet receive errors
    1152 packets sent
    0 receive buffer errors
    0 send buffer errors
    IgnoredMulti: 207
UdpLite:
TcpExt:
    1 resets received for embryonic SYN_RECV sockets
    267 TCP sockets finished time wait in fast timer
    65 delayed acks sent
    Quick ack mode was activated 1 times
    875 packet headers predicted
    1583 acknowledgments not containing data payload received
    632 predicted acknowledgments
    2 congestion windows recovered without slow start after partial ack
    TCPTimeouts: 2
    TCPLossProbes: 7
    TCPDSACKOldSent: 1
    TCPDSACKRecv: 8
    52 connections reset due to unexpected data
    12 connections reset due to early user close
    TCPDSACKIgnoredNoUndo: 4
    TCPDeferAcceptDrop: 62
    TCPRcvCoalesce: 179
    TCPChallengeACK: 8
    TCPSpuriousRtxHostQueues: 6
    TCPAutoCorking: 114
    TCPOrigDataSent: 2264
IpExt:
    InMcastPkts: 1904
    OutMcastPkts: 443
    InBcastPkts: 407
    OutBcastPkts: 53
    InOctets: 2195579
    OutOctets: 1642119
    InMcastOctets: 497123
    OutMcastOctets: 140672
    InBcastOctets: 36565
    OutBcastOctets: 7871
    InNoECTPkts: 7713
    InECT1Pkts: 12
    InECT0Pkts: 5
    InCEPkts: 4
After:

Code: Select all

Ip:
    Forwarding: 2
    7543 total packets received
    0 forwarded
    0 incoming packets discarded
    7543 incoming packets delivered
    5409 requests sent out
Icmp:
    0 ICMP messages received
    0 input ICMP message failed
    ICMP input histogram:
    0 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
Tcp:
    126 active connection openings
    395 passive connection openings
    1 failed connection attempts
    63 connection resets received
    8 connections established
    5152 segments received
    4504 segments sent out
    9 segments retransmitted
    0 bad segments received
    81 resets sent
Udp:
    2295 packets received
    0 packets to unknown port received
    0 packet receive errors
    1164 packets sent
    0 receive buffer errors
    0 send buffer errors
    IgnoredMulti: 212
UdpLite:
TcpExt:
    1 resets received for embryonic SYN_RECV sockets
    276 TCP sockets finished time wait in fast timer
    66 delayed acks sent
    Quick ack mode was activated 1 times
    881 packet headers predicted
    1627 acknowledgments not containing data payload received
    647 predicted acknowledgments
    2 congestion windows recovered without slow start after partial ack
    TCPTimeouts: 2
    TCPLossProbes: 7
    TCPDSACKOldSent: 1
    TCPDSACKRecv: 8
    53 connections reset due to unexpected data
    12 connections reset due to early user close
    TCPDSACKIgnoredNoUndo: 4
    TCPDeferAcceptDrop: 63
    TCPRcvCoalesce: 183
    TCPChallengeACK: 8
    TCPSpuriousRtxHostQueues: 6
    TCPAutoCorking: 114
    TCPOrigDataSent: 2319
IpExt:
    InMcastPkts: 1923
    OutMcastPkts: 455
    InBcastPkts: 412
    OutBcastPkts: 53
    InOctets: 2215517
    OutOctets: 1684938
    InMcastOctets: 502583
    OutMcastOctets: 144862
    InBcastOctets: 36777
    OutBcastOctets: 7871
    InNoECTPkts: 7854
    InECT1Pkts: 12
    InECT0Pkts: 5
    InCEPkts: 4
An as reference, the netstat -s as it works fast from another client,
Before:

Code: Select all

Ip:
    Forwarding: 2
    7619 total packets received
    0 forwarded
    0 incoming packets discarded
    7619 incoming packets delivered
    5483 requests sent out
Icmp:
    0 ICMP messages received
    0 input ICMP message failed
    ICMP input histogram:
    0 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
Tcp:
    128 active connection openings
    398 passive connection openings
    1 failed connection attempts
    64 connection resets received
    8 connections established
    5220 segments received
    4556 segments sent out
    9 segments retransmitted
    0 bad segments received
    84 resets sent
Udp:
    2303 packets received
    0 packets to unknown port received
    0 packet receive errors
    1186 packets sent
    0 receive buffer errors
    0 send buffer errors
    IgnoredMulti: 212
UdpLite:
TcpExt:
    1 resets received for embryonic SYN_RECV sockets
    278 TCP sockets finished time wait in fast timer
    67 delayed acks sent
    Quick ack mode was activated 1 times
    892 packet headers predicted
    1638 acknowledgments not containing data payload received
    664 predicted acknowledgments
    2 congestion windows recovered without slow start after partial ack
    TCPTimeouts: 2
    TCPLossProbes: 7
    TCPDSACKOldSent: 1
    TCPDSACKRecv: 8
    54 connections reset due to unexpected data
    13 connections reset due to early user close
    TCPDSACKIgnoredNoUndo: 4
    TCPDeferAcceptDrop: 64
    TCPRcvCoalesce: 185
    TCPChallengeACK: 9
    TCPSpuriousRtxHostQueues: 6
    TCPAutoCorking: 114
    TCPOrigDataSent: 2349
IpExt:
    InMcastPkts: 1931
    OutMcastPkts: 456
    InBcastPkts: 412
    OutBcastPkts: 53
    InOctets: 2225996
    OutOctets: 1700642
    InMcastOctets: 503616
    OutMcastOctets: 145021
    InBcastOctets: 36777
    OutBcastOctets: 7871
    InNoECTPkts: 7930
    InECT1Pkts: 12
    InECT0Pkts: 5
    InCEPkts: 4
After:

Code: Select all

Ip:
    Forwarding: 2
    7648 total packets received
    0 forwarded
    0 incoming packets discarded
    7648 incoming packets delivered
    5492 requests sent out
Icmp:
    0 ICMP messages received
    0 input ICMP message failed
    ICMP input histogram:
    0 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
Tcp:
    128 active connection openings
    399 passive connection openings
    1 failed connection attempts
    64 connection resets received
    8 connections established
    5235 segments received
    4565 segments sent out
    9 segments retransmitted
    0 bad segments received
    84 resets sent
Udp:
    2316 packets received
    0 packets to unknown port received
    0 packet receive errors
    1186 packets sent
    0 receive buffer errors
    0 send buffer errors
    IgnoredMulti: 213
UdpLite:
TcpExt:
    1 resets received for embryonic SYN_RECV sockets
    278 TCP sockets finished time wait in fast timer
    67 delayed acks sent
    Quick ack mode was activated 1 times
    892 packet headers predicted
    1641 acknowledgments not containing data payload received
    669 predicted acknowledgments
    2 congestion windows recovered without slow start after partial ack
    TCPTimeouts: 2
    TCPLossProbes: 7
    TCPDSACKOldSent: 1
    TCPDSACKRecv: 8
    54 connections reset due to unexpected data
    13 connections reset due to early user close
    TCPDSACKIgnoredNoUndo: 4
    TCPDeferAcceptDrop: 64
    TCPRcvCoalesce: 185
    TCPChallengeACK: 9
    TCPSpuriousRtxHostQueues: 6
    TCPAutoCorking: 114
    TCPOrigDataSent: 2356
IpExt:
    InMcastPkts: 1944
    OutMcastPkts: 456
    InBcastPkts: 413
    OutBcastPkts: 53
    InOctets: 2232344
    OutOctets: 1702788
    InMcastOctets: 508117
    OutMcastOctets: 145021
    InBcastOctets: 36845
    OutBcastOctets: 7871
    InNoECTPkts: 7959
    InECT1Pkts: 12
    InECT0Pkts: 5
    InCEPkts: 4
I have looked into /proc/sys/fs/file-nr, server: 2624 0 396513 C1: 1280 0 200000 (file_max increased already to 200000).
But I am sure that this is not the bottle neck.

And here is also the pcap shown in wireshark,
delayed:
Bildschirmfoto vom 2019-11-18 14-01-13.png
Bildschirmfoto vom 2019-11-18 14-01-13.png (216.26 KiB) Viewed 385 times
fast:
Bildschirmfoto vom 2019-11-18 14-01-57.png
Bildschirmfoto vom 2019-11-18 14-01-57.png (80 KiB) Viewed 385 times
Until row 5 it is the same. In case of fast answer, the answer is in row 6 after 290ms. In case of delayed answer, the answer is in row 21 after 25 seconds.

User avatar
mad_ady
Posts: 6860
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 236 times
Been thanked: 177 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by mad_ady » Tue Nov 19, 2019 12:29 am

Off the top of my head I'm thinking of a dns issue.
Is the nextcloud database hosted in the same system as the frontend? Are they running as containers?

Mysql (and ssh) do a reverse dns lookup of the client ip on new connections (for authentication/logging). Of dns is unreachable you get this 20-30s timeout.

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Sat Nov 23, 2019 3:45 am

Hello, @mad_ady, I tried a lot but I haven't found a solution. I installed on another C1 Armbian with mainline kernel and I got a response time of about 3.7s. This is faster, but 10 time slower than from other machines.
Now I gave up on it and I made a program that runs on my OMV/Nexcloud server (N1) and sends the calendar events over mqtt to the Homeassistant.
This case here stays unsolved.
Thank you for you support.

User avatar
mad_ady
Posts: 6860
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 236 times
Been thanked: 177 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by mad_ady » Sat Nov 23, 2019 5:35 am

I love a good mistery, but I prefer to see them solved.
Hmm, if you can identify a running process that handles the request (or if you can start it single-threaded in foreground) you could attach strace to its pid and reproduce the issue. That should show you what is causing the wait.

joerg
Posts: 912
Joined: Tue Apr 01, 2014 2:14 am
languages_spoken: german, english, español
ODROIDs: C1, C1+, C2
Location: Germany
Has thanked: 7 times
Been thanked: 21 times
Contact:

Re: Slow network connection to Nextcloud

Unread post by joerg » Sat Nov 23, 2019 4:07 pm

mad_ady wrote:
Sat Nov 23, 2019 5:35 am
I love a good mistery, but I prefer to see them solved.
I'm too.
mad_ady wrote:
Sat Nov 23, 2019 5:35 am
Hmm, if you can identify a running process that handles the request (or if you can start it single-threaded in foreground) you could attach strace to its pid and reproduce the issue. That should show you what is causing the wait.
No, I spent to many time in that. I am satisfied with my new solution.

I have several small progs that are using same construction. Made with C++, starting daemonized by systemd, using paho.mqtt.cpp and openssl. Now for this prog I searched for a small library that makes the Http requests and this one I found: https://github.com/mattdh666/rpi-http-request. Not complete, but it cross compiles on my machine also for the N1 and is enough for my needs.
With my prog I had the same delay on the C1, so I think the delay is not caused by curl. On the N1 itself the Http request runs nearly without delay and the transfer over mqtt to the C1 is also without delay. Crazy that only this setup with my script has this delay. On the Ubuntu with stock kernel sometimes, on the Armbian with mainline kernel always.

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 2 guests