shadowsocks-rust failed to start after last update via snap

[fthorus]

Hi,

after last update via snap I'm unable to start shadowsocks-rust and getting errors, which I don't know how to solve. Please help.

---

$/usr/bin/snap run shadowsocks-rust.ssserver-daemon -c /root/shadowsocks.json
2024-02-16T13:07:46.519584293+03:00 INFO shadowsocks server 1.18.0 build 2024-02-07T18:36:11.165817751+00:00
thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/multi_thread/worker.rs:447:13:
OS can't spawn worker thread: Operation not permitted (os error 1)
stack backtrace:
0: 0x55beab41695c -
1: 0x55beab178e20 -
2: 0x55beab3e0182 -
3: 0x55beab4194fe -
4: 0x55beab418c70 -
5: 0x55beab419e32 -
6: 0x55beab41985c -
7: 0x55beab4197b6 -
8: 0x55beab4197a1 -
9: 0x55beab0512e4 -
10: 0x55beab3626ce -
11: 0x55beab0fbbaf -
12: 0x55beab0c5d53 -
13: 0x55beab0c64c9 -
14: 0x7f126c930d90 -
15: 0x7f126c930e40 - __libc_start_main
16: 0x55beab072da5 -
17: 0x0 -
Aborted

$ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.

$ snap list
Name Version Rev Tracking Publisher Notes
core 16-2.61.1 16574 latest/stable canonical✓ core
core18 20231027 2812 latest/stable canonical✓ base
core20 20231123 2105 latest/stable canonical✓ base
core22 20240111 1122 latest/stable canonical✓ base
shadowsocks-rust v1.18.0.r1.g65eb758a 661 latest/stable ducksoft held
snapd 2.61.1 20671 latest/stable canonical✓ snapd

uname -ai
Linux 3.10.0-1160.105.1.el7.x86_64 #1 SMP Thu Dec 7 15:39:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

---

Config:
cat shadowsocks.json
{
"server": "0.0.0.0",
"mode":"tcp_and_udp",
"server_port":444,
"password":"*",
"timeout":300,
"method":"2022-blake3-aes-256-gcm"
}

[zonyitoo]

The root cause was:

2024-02-07T18:36:11.165817751+00:00
thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/multi_thread/worker.rs:447:13:
OS can't spawn worker thread: Operation not permitted (os error 1)

which is reported right here: https://github.com/tokio-rs/tokio/blob/master/tokio/src/runtime/blocking/pool.rs#L418-L437 .

The spawn_thread will call std::thread directly to Rust's standard library, which will eventually making calls to libc and pthread APIs.

So I highly suspect that there was a syscall that your system didn't support, because:

1. v1.18.0 was built in snap with core22, which is a quite new version: https://snapcraft.io/core22 .
2. Your Kernel is 3.10.0, which is a rather old version comparing to core22 (v5.15 and v5.17), Release Note.

My built with cross running on Debian 11 with Linux 6.1.0 works quite well. So there must be something incompatible with your Linux Distribution environment. Here are some possible ways to help debug:

1. Run ssserver binary with strace and see which exactly syscall returns EPERM
2. Consider upgrade your Distribution to the latest version
3. Provide your Linux Distribution detail, I may have to try to install one in VM and reproduce your case.

[fthorus]

@zonyitoo , thank you for quick response and thoughtful explanation.

I found several EPERM in strace, attached the output to the reply.
strace usrbinsnap run shadowsocks-r.txt

You're absolutely right, my current system is almost obsolete Centos 7. And before upgrading to Centos 9 I was trying to find out if it is possible to use previous version of SSServer-rust. But it seems that snap repo switched hardly to v.1.18.0 providing no previous versions to install:
channels:
latest/stable: v1.18.0.r1.g65eb758a 2024-02-16 (661) 10MB -
latest/candidate: v1.18.0.r1.g65eb758a 2024-02-16 (661) 10MB -
latest/beta: v1.18.0.r1.g65eb758a 2024-02-16 (661) 10MB -
latest/edge: v1.18.0.r13.g69498c91 2024-02-16 (681) 11MB -
installed: v1.18.0.r1.g65eb758a (661) 10MB held

Is there any possibility to get previous snap of shadowsocks-rust which I could revert to and use? Maybe you could provide me with that snap?

In case the answer is 'no', then I will have to upgrade Distribution to the latest version and reconfigure everything once again (OpenSSL, OpenVPN, XRay, Nginx and other related services) . Welp, it was my strategic error to stay on Centos 7...

[zonyitoo]

Just looked through your strace log, why EPOLL_CTL_ADD returned EPERM? Couldn't understand.

Is there any possibility to get previous snap of shadowsocks-rust which I could revert to and use? Maybe you could provide me with that snap?

Well, before v1.18.0, it only supports i386 architecture, it may not be able to run well on your system, too.

I don't know if I downgrade the core version to, maybe core20 or core18 could make it working on your system.

[zonyitoo]

Since Kernel 3.10 was used in Ubuntu 13.04, which was released in 2013, far older than core18 https://snapcraft.io/core18 , I believe the most reliable ways are:

1. Use docker releases: https://github.com/orgs/shadowsocks/packages?repo_name=shadowsocks-rust
2. Clone source code and build a ssserver for your environment
3. Use MUSL prebuilt binaries: https://github.com/shadowsocks/shadowsocks-rust/releases/tag/v1.18.0