Compiler failed to spawn thread, "Resource temporarily unavailable" on hello-world; Ubuntu VM.

[Datarecovery09]

Seemingly random bug; cannot reproduce reliably. Tends to vanish for some time when I reboot the system, but always comes back after some time. CPU and RAM are barely used when it occurs.

System

Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

Code

fn main() {
    println!("Hello, world!");
}

Meta

Bug occurs in nightly and stable versions.

rustc --version --verbose:

rustc 1.43.0-nightly (58b834344 2020-02-05)
binary: rustc
commit-hash: 58b834344fc7b9185e7a50db1ff24e5eb07dae5e
commit-date: 2020-02-05
host: x86_64-unknown-linux-gnu
release: 1.43.0-nightly
LLVM version: 9.0

Error output

`rustc --crate-name hello --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=3030fab4ddb057dd -C extra-filename=-3030fab4ddb057dd --out-dir /home/jweber/rust/hello/target/debug/deps -C incremental=/home/jweber/rust/hello/target/debug/incremental -L dependency=/home/jweber/rust/hello/target/debug/deps`
thread '<unnamed>' panicked at 'failed to spawn thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:619:5

Backtrace

   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:77
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1052
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1428
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:204
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:224
  10: rustc_driver::report_ice
  11: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:474
  12: rust_begin_unwind
             at src/libstd/panicking.rs:378
  13: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  14: core::option::expect_none_failed
             at src/libcore/option.rs:1199
  15: std::thread::spawn
  16: rustc_codegen_ssa::back::write::spawn_work

[jonas-schievink]

Root cause seems to be #46345

[Datarecovery09]

Temporary Workaround:
Set nr of threads used by cargo to 1, either via setting an environment variable
CARGO_BUILD_JOBS=1
or by running/building with
cargo build -j 1
This will slow down compilation time, but it seems to prevent the error from occuring.

---

EDIT

Nevermind, Bug still can happen - although less frequently - with nr of build threads = 1.

[alecmocatta]

@Datarecovery09 Your setup is likely slightly atypical as most Rust users don't see this error. Mind reporting back the results of these commands next time you see this error to shed some light on what might be causing this?

ps -AL --no-headers | wc -l
cat /proc/meminfo
ulimit -Sa
ulimit -Ha
cat /proc/sys/kernel/threads-max
cat /proc/sys/vm/max_map_count
cat /proc/sys/kernel/pid_max
systemctl --version
systemctl status user-$UID.slice
cat /proc/sys/vm/overcommit_*

Are there any ways you're aware of that your setup might be atypical? My best guess is low total RAM, which would lead to low threads-max which is reduced 2/3 further by systemd?

If that is indeed the case, you could try the steps outlined here as well as echo 100000 | sudo tee /proc/sys/kernel/threads-max and see if that makes it occur less frequently?

[Datarecovery09]

@alecmocatta Thanks for the fast reply!
I'm not aware of any special atypicalities of the system; yet I'm not an expert on linux/ubuntu distributions, virtual machines or shared servers.
I can't completely follow the steps in the Stackoverflow-thread you've linked; I can't reboot the server right now. echo 100000 | sudo tee /proc/sys/kernel/threads-max results in bash: fork: retry: The resource is temporarily unavailable.

Here is the information on the system:

ps -AL --no-headers | wc -l
143

cat /proc/meminfo

MemTotal:        8388608 kB
MemFree:         7565036 kB
MemAvailable:    7680104 kB
Cached:           145168 kB
Buffers:               0 kB
Active:           463060 kB
Inactive:         289424 kB
Active(anon):     407456 kB
Inactive(anon):   250072 kB
Active(file):      55604 kB
Inactive(file):    39352 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               124 kB
Writeback:             0 kB
AnonPages:        657528 kB
Shmem:             50212 kB
Slab:              47076 kB
SReclaimable:      20112 kB
SUnreclaim:        26964 kB

ulimit -Sa

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1546428
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62987
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

ulimit -Ha

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1546428
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62987
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

cat /proc/sys/kernel/threads-max
3092856

cat /proc/sys/vm/max_map_count
65530

cat /proc/sys/kernel/pid_max
32768

systemctl --version

systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

systemctl status user-$UID.slice

● user-1000.slice
   Loaded: loaded
   Active: inactive (dead)

cat /proc/sys/vm/overcommit_*

0
1
50

I hope this is helpful.

[jonas-schievink]

echo 100000 | sudo tee /proc/sys/kernel/threads-max results in bash: fork: retry: The resource is temporarily unavailable.

This definitely means that something is not working right with your system, since even bash fails to spawn another process.

[alecmocatta]

Echoing @jonas-schievink, something's definitely not right with your machine if bash's fork is failing. The outputs you pasted look fine, possibly with the exception of the systemctl status user-$UID.slice one. Perhaps check cgroups aren't stopping you forking new processes?

[Datarecovery09]

Hi,
I've done some more testing on multiple systems, and I could only reproduce the error on two OpenVZ based environments (Ubuntu 18.04 & 16.04) and under heavy load. It "helped" when swapfile was disabled. I've written to my host and asked them about some configuration details (I know they're using OpenVZ/Virtuozzo for their virtualization and disable swapfiles) and am now awaiting their response.

[vioan]

@Datarecovery09 did you solve this issue? I also got the same problem when I run some containers. I have a few of them running but now I can not start anymore new containers. And I think we are discussing about the same "host", perhaps Strato. :)

[jonas-schievink]

(Closing since this is not a Rust issue)