rpc-srv/tcp: nfsd: sent ... when sending ... bytes - shutting down socket

[mad_ady]

I'm experiencing a weird lock-up on my N1, running kernel 5.8...
It happens only occasionally, and only when streaming data over NFS (2-3Mbps tx from the N2). Usually when streaming data over long periods of time (watching a movie). The dmesg message I get (logged over netconsole, which still works) is:

Code: Select all

Feb 21 20:23:42 aldebaran.lan  [1089023.107839] rpc-srv/tcp: nfsd: sent 95568 when sending 524416 bytes - shutting down socket
Feb 21 20:23:42 aldebaran.lan  [1089023.113636] rpc-srv/tcp: nfsd: sent 101360 when sending 524416 bytes - shutting down socket

Apparently networking locks up, (ssh is not reachable anymore), though some protocols might still work (icmp/udp). I haven't done a full analysis of what works in this state and was forced to reboot.

The funny thing is - this lock-up happened to me twice in the past month, though never before on the same N1, running the same kernel for about 6 months...
I've searched the forums, and others had similar issues that were solved at the nic driver level:
https://www.spinics.net/lists/netdev/msg466375.html

Not sure if the nic on the N1 suffers from the same problem, but I wanted to ask the few N1 users around if you've ever seen this with your systems?

[meveric]

I'm not using NFS, normally only samba or something, currently I have a 500GB SSD connected to the N1, so normally I don't have the need to pull large data over the network, and once again, not using NFS.
Have you tried other methods than NFS to see if it's a general network issue or just NFS?

[mad_ady]

No, I'm 90% using nfs... But I will think of some stress tests over samba as well...

[meveric]

you can stream video over samba as well, should be the same scenario.
Or just loop a copy command from samba to /dev/null.

[mad_ady]

I think I need to throttle the throughput. I have the suspicion the problem happens at low, but sustained speeds...

[mad_ady]

Well, here I am again. After a year of stability (with the same 5.8.0-rc5 kernel), I managed to lock it up 5 times in 2 days. I can lock it up by having 2 Kodi instances playing from the NFS share at the same time (Kodi on Android - one on an N2, one on an Android TV). The streams being played are low-bandwidth - one was 480p, one was 720p, total rate < 5Mbps.

I'm writing down what I'm willing to try to fix it, so I don't forget:
1. Replace kernel with meveric's kernel from his universal Debian build. Might lose fan support (and will also need to redo my gpio-ir-tx configuration).
2. Upgrade to Ubuntu 22.04 - though it seems like a kernel bug, not a userspace bug. Nevertheless, an upgrade needs to be done at some point...
3. Transition from NFS to SMB in Kodi. This is more problematic, because I need to change the shares in the shared Kodi config, and figure out a way to preserve my "seen list", even if the path changes. Or is Kodi smart enough to see that a movie in the database has changed paths and keep the seen metadata? I'm using a shared Mysql database for both players.

I'll report back...

[renegat]

For all running into the same issue, searching for a solution and not finding one:

Check your network!
In my case the error was caused by a weak / interfered wlan signal. (Cat lying on router)
Placing the cat in a better location - issue was gone.

It is just one of the usual blank, unuseful and cryptic linux error messages that lead nowhere instead of just telling e.g.:
'Hey, you're having some network issue. This is why we can transfer no data!'

[mad_ady]

Though I'm not ruling out a network issue, for me it's not wifi-related since clients and server are in the same wired LAN. And my NFS runs over TCP, which should rule out feline-induced outages. Unless the message means the lower protocol gave up for some reason, which would make sense.