bash: fork: retry: Resource temporarily unavailable

Post Reply
User avatar
mad_ady
Posts: 6868
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 238 times
Been thanked: 178 times
Contact:

bash: fork: retry: Resource temporarily unavailable

Unread post by mad_ady » Wed Nov 27, 2019 8:55 pm

From time to time I get this annoying issue - I can't start new processes under my own user because I seem to hit some limit and I can't fork new things. The way I usually solve it is by closing long-running byobu/tmux sessions.

Here's the problem - I'm not sure what limit I'm hitting, and I'd like to learn how to find out. I don't think it's the number of running processes/threads, because that is low:

Code: Select all

$ ps hH -u adrianp | wc -l 
14
$ ps hH -u adrianp
  798 ?        Ssl   28:44 /usr/bin/transmission-daemon -f -g /etc/transmission-daemon --log-error
  798 ?        Ssl    0:00 /usr/bin/transmission-daemon -f -g /etc/transmission-daemon --log-error
  798 ?        Ssl  417:32 /usr/bin/transmission-daemon -f -g /etc/transmission-daemon --log-error
  798 ?        Ssl   21:58 /usr/bin/transmission-daemon -f -g /etc/transmission-daemon --log-error
 1262 ?        Ss    31:41 /lib/systemd/systemd --user
 1270 ?        S      0:00 (sd-pam)
 1594 ?        Ss   7681:38 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc new-session -n - /usr/bin/byobu-shell
 1609 pts/2    Ss     0:04 /bin/bash
 8309 pts/0    Ss+    0:00 /bin/bash
 9809 pts/4    Ss+    0:00 /bin/bash
15231 pts/1    Ss+    0:05 /bin/bash
17263 ?        S      1:23 sshd: adrianp@pts/7
17266 pts/7    Ss+    0:00 -bash
17281 pts/7    S+     0:00 tmux -2 new-session -t 1 -s _1-17281
$ sleep 5 && sleep 5 && sleep 5 && true
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
Now, this affects things like ssh-ing with my own user, which is annoying. Strangely enough, it sometimes affects the root user as well, but when running commands as root from within a tmux session, not when running directly from logging in as root:

Code: Select all

# ps hH -u adrianp
-su: fork: retry: Resource temporarily unavailable
-su: fork: retry: Resource temporarily unavailable
-su: fork: retry: Resource temporarily unavailable
-su: fork: retry: Resource temporarily unavailable
-su: fork: Resource temporarily unavailable
I am running a few things on this N1, a bunch of docker containers, transmission, NFS/SMB, but it shouldn't be out of the ordinary and it's hardly loaded.
I suspect it may be tied into the byobu/tmux shell I usually connect to. When connecting without it, commands work.
Let's see some limits:

Code: Select all

$ ulimit
unlimited
$ cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - a user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#        - NOTE: group and wildcard limits are not applied to root.
#          To apply a limit to the root user, <domain> must be
#          the literal username root.
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#        - chroot - change root to directory (Debian-specific)
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#root            hard    core            100000
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#ftp             -       chroot          /ftp
#@student        -       maxlogins       4

# End of file
bash: fork: retry: Resource temporarily unavailable
$ free -m
              total        used        free      shared  buff/cache   available
Mem:           3878        1468        1204           8        1204        2369
Swap:           969         327         642
bash: fork: retry: Resource temporarily unavailable
$ uptime
 13:51:31 up 47 days,  5:29,  6 users,  load average: 0.89, 0.81, 0.75
bash: fork: retry: Resource temporarily unavailable
$ cat /etc/systemd/system.conf | grep -v '#'                                                                                                    
                                        
[Manager]
DefaultTimeoutStartSec=20s                        
DefaultTimeoutStopSec=10s            
bash: fork: retry: Resource temporarily unavailable
$ cat /etc/systemd/logind.conf | grep -v '#'

[Login]
UserTasksMax=90%
bash: fork: retry: Resource temporarily unavailable
So - how do I learn what I'm running out of? Why can't the system fork new processes?

User avatar
rooted
Posts: 6899
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 270 times
Been thanked: 64 times
Contact:

Re: bash: fork: retry: Resource temporarily unavailable

Unread post by rooted » Fri Nov 29, 2019 5:21 pm

From a bit of reading you could try adding this to /etc/sysctl.conf

Code: Select all

 
kern.maxprocperuid=1000
kern.maxproc=2000
These users thanked the author rooted for the post:
mad_ady (Fri Nov 29, 2019 6:06 pm)

User avatar
mad_ady
Posts: 6868
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 238 times
Been thanked: 178 times
Contact:

Re: bash: fork: retry: Resource temporarily unavailable

Unread post by mad_ady » Fri Nov 29, 2019 6:06 pm

Thanks for the idea!
However, it doesn't seem to exist in my kernel (4.4 on Odroid N1):

Code: Select all

$sudo sysctl -a 2>/dev/null | grep maxproc
$sudo sysctl kern.maxproc=2500
sysctl: cannot stat /proc/sys/kern/maxproc: No such file or directory
$ sudo sysctl kernel.maxproc=2500
sysctl: cannot stat /proc/sys/kernel/maxproc: No such file or directory


User avatar
rooted
Posts: 6899
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 270 times
Been thanked: 64 times
Contact:

Re: bash: fork: retry: Resource temporarily unavailable

Unread post by rooted » Fri Nov 29, 2019 6:11 pm

Yeah it's not on my XU4 either, maybe it's something missing from the kernel configuration.

User avatar
rooted
Posts: 6899
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 270 times
Been thanked: 64 times
Contact:

Re: bash: fork: retry: Resource temporarily unavailable

Unread post by rooted » Fri Nov 29, 2019 6:13 pm

Maybe it's now in:

/etc/security/limits.conf

You can check a process limits at:

Code: Select all

cat /proc/PID/limits 
You may try:

Code: Select all

 sudo sysctl kernel.pid_max=4194304 
But I feel like something else is going on, surely you aren't hitting the current limit?

*edit*
Apparently ARM has a different limit, I tried the above and it wouldn't work. The maximum is 32768.

What is the 'kernel.threads-max' set at?
These users thanked the author rooted for the post:
mad_ady (Fri Nov 29, 2019 6:24 pm)

User avatar
mad_ady
Posts: 6868
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 238 times
Been thanked: 178 times
Contact:

Re: bash: fork: retry: Resource temporarily unavailable

Unread post by mad_ady » Fri Nov 29, 2019 6:49 pm

Thanks for the pointers.
Here are the current limits of my byobu process (that I suspect is causing the starvation):

Code: Select all

$ cat /proc/19809/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             15481                15481                processes 
Max open files            1024                 1048576              files     
Max locked memory         16777216             16777216             bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       15481                15481                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us
According to this: https://unix.stackexchange.com/question ... hard-limit, for unprivileged accounts, the soft limit is actually "hard" and I suspect I run out of open files, since they are quite low (1024). Next time this happens (for now I have restarted byobu), I'll check the number of open files under its process:

Code: Select all

$ sudo lsof | grep 19809 | wc -l
26
I remember we had this issue in production with mysql running out of open files on systemd systems, that was solved with LimitNOFILE=65535 in the systemd service. Either that, or it's running out of stack memory?

The threads-max is set at kernel.threads-max = 30962. I doubt I can have that many threads without significantly increasing system load.

I guess I'll wait for the problem to reproduce and check on the number of open files per process.
Thanks again for the push in the hopefully right direction.

User avatar
rooted
Posts: 6899
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 270 times
Been thanked: 64 times
Contact:

Re: bash: fork: retry: Resource temporarily unavailable

Unread post by rooted » Fri Nov 29, 2019 6:53 pm

No problem, hopefully you won't run into the issue again but at least you have a few things to check if you do.

Post Reply

Return to “General Topics”

Who is online

Users browsing this forum: No registered users and 0 guests