iperf3: error - control socket has closed unexpectedly - when testing with xray

[shakibamoshiri]

what is the issue?

I would like to test and measure VPNs speed with iperf3, and have tested many

• ocserv
• SoftEther
• tinc
• hysteria 2
• etc

but with xray or singbox (as client) protocols vless-grpc/tcp the tests fail with this error

iperf3: error - control socket has closed unexpectedly

I already have seen badO1a5A90/Performance tests and the method mentioned there

A,iperf client+Xray inbound(dokodemo-door),outbound(freedom with TLS)
<--B,inbound(dokodemo-door),outbound(various protocol combinations)
<--C,inbound(various protocol combinations) ,outbound(freedom)
<--D,Xray inbound(dokodemo-door with TLS),outbound(freedom)+iperf server

but was not clear and did not provide enough details of how to run tests

why I ask the question here ?

Because that repo is not active and there is one asked with no answer Can you send me the test configuration?

how did I conduct the test (worked with hysteria 2)

First
Because xray or hy2 does not have built-in tun interface (at both side) I have created two tun interfaces both sides with iproute2 e.g

### client side
ip tuntap add dev tun77 mode tun
ip addr add 192.168.77.2/30 brd + dev tun77

### server side
ip tuntap add dev tun77 mode tun
ip addr add 192.168.77.1/30 brd + dev tun77

Second
connecting the client (xray or singbox) to the server

Third
Testing with curl if socks5 inbound works

curl --socks5-hostname 127.0.0.1:2080 api.ipify.org; echo

Fourth
Running tun2socks at client side

./tun2socks -device tun77 -proxy socks5://127.0.0.1:2080

Fifth
Running iperf3 at client side

iperf3 --client 192.168.77.1 --port 1234   --reverse

result

with hysteria 2 , it runs as expected but xray / singbox (as client) gives the mentioned error

additional contexts

• both hy2 and xray socks5 with no pass used
• the tunnel from client to the server is a "split tunnel" not a "full tunnel"
• xray versions are the same Xray 1.8.7 (Xray, Penetrates Everything.) 3f0bc13 (go1.21.5 linux/amd64)
• xray config client [ inbound(socks5) => outbound (vless/tcp) ] ---> server [ inbound(vless-tcp) => outbound(direct) ]

what could be the cause ?

I doubt the way I configured the xary be the cause
Please let me know if anyone knows the right solution to test
Regards

configs

client config for xray

{
 "log": {
    "loglevel": "debug",
    "access": "./xray-debug.log"
  },
  "dns": {
    "hosts": {
      "domain:googleapis.cn": "googleapis.com"
    },
    "servers": [
      "1.1.1.1"
    ]
  },
  "inbounds": [
    {
      "listen": "127.0.0.1",
      "port": 2080,
      "protocol": "socks",
      "settings": {
        "auth": "noauth",
        "udp": true,
        "userLevel": 8
      },
      "sniffing": {
        "destOverride": [
          "http",
          "tls"
        ],
        "enabled": true
      },
      "tag": "socks"
    },
    {
      "listen": "127.0.0.1",
      "port": 10809,
      "protocol": "http",
      "settings": {
        "userLevel": 8
      },
      "tag": "http"
    }
  ],
  "log": {
    "loglevel": "warning"
  },
  "outbounds": [
    {
      "mux": {
        "concurrency": -1,
        "enabled": false,
        "xudpConcurrency": 8,
        "xudpProxyUDP443": ""
      },
      "protocol": "vless",
      "settings": {
        "vnext": [
          {
            "address": "1.2.3.4",
            "port": 1234,
            "users": [
              {
                "encryption": "none",
                "flow": "",
                "id": "993a90e4-3778-44c6-b6d8-ea983d7a0fc5",
                "level": 8,
                "security": "auto"
              }
            ]
          }
        ]
      },
      "streamSettings": {
        "network": "tcp",
        "security": "none",
        "tcpSettings": {
          "header": {
            "type": "none"
          }
        }
      },
      "tag": "proxy"
    },
    {
      "protocol": "freedom",
      "settings": {},
      "tag": "direct"
    },
    {
      "protocol": "blackhole",
      "settings": {
        "response": {
          "type": "http"
        }
      },
      "tag": "block"
    }
  ],
  "routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [
      {
        "ip": [
          "1.1.1.1"
        ],
        "outboundTag": "proxy",
        "port": "53",
        "type": "field"
      }
    ]
  }
}

logs debug

cat xray-debug.log 

### curl test okay 
2024/05/15 11:57:32 tcp:127.0.0.1:47614 accepted tcp:api.ipify.org:80 [socks >> proxy]

### ipef3 test error
2024/05/15 11:57:46 tcp:127.0.0.1:35546 accepted tcp:192.168.77.1:10123 [socks >> proxy]

### iperf3 error
2024/05/15 11:58:21 tcp:127.0.0.1:49678 accepted tcp:192.168.77.1:10123 [socks >> proxy]

### curl okay 
2024/05/15 11:58:34 tcp:127.0.0.1:55514 accepted tcp:api.ipify.org:80 [socks >> proxy]

### ipef3 error
2024/05/15 11:58:37 tcp:127.0.0.1:55526 accepted tcp:192.168.77.1:10123 [socks >> proxy]

### curl okay 
2024/05/15 11:59:26 tcp:127.0.0.1:34476 accepted tcp:api.ipify.org:80 [socks >> proxy]

[Fangliding]

Is the sock5 available?

curl -x socks5://127.0.0.1:2080 ip.sb

And is the tun interface available?

curl ip.sb --interface tun77

AAnd where is the config and log

[shakibamoshiri]

Is the sock5 available?

mentioned third curl socks test works

curl --socks5-hostname 127.0.0.1:2080 api.ipify.org; echo
1.2.3.4 # server ip address 

And is the tun interface available?

but curl query over a virtual interface does not work, it works over physical interface (as long as I know and tested)

curl --interface tun77   api.ipify.org; echo

### hangs , has no output 

AAnd where is the config and log

I will update and attach configs

[Fangliding]

I mean after tun2socks started, is the socks5 still work

[shakibamoshiri]

I mean after tun2socks started, is the socks5 still work

yes tun2socks is running and curl test is fine
question updated with log and configurations

[Fangliding]

Complete log

[shakibamoshiri]

Complete log

I added

 "log": {
    "loglevel": "debug",
    "access": "./xray-debug.log"
  },

and this is the complete log ,

2024/05/15 11:57:32 tcp:127.0.0.1:47614 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 11:57:46 tcp:127.0.0.1:35546 accepted tcp:192.168.77.1:10123 [socks >> proxy]
2024/05/15 11:58:21 tcp:127.0.0.1:49678 accepted tcp:192.168.77.1:10123 [socks >> proxy]
2024/05/15 11:58:34 tcp:127.0.0.1:55514 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 11:58:37 tcp:127.0.0.1:55526 accepted tcp:192.168.77.1:10123 [socks >> proxy]
2024/05/15 11:59:26 tcp:127.0.0.1:34476 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 12:13:32 tcp:127.0.0.1:51960 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 12:13:34 tcp:127.0.0.1:51970 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 12:13:56 tcp:127.0.0.1:55644 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 12:13:57 tcp:127.0.0.1:55656 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 12:26:03 tcp:127.0.0.1:59888 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 12:26:11 tcp:127.0.0.1:40548 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 12:26:13 tcp:127.0.0.1:40562 accepted tcp:192.168.77.1:10123 [socks >> proxy]
2024/05/15 12:26:44 tcp:127.0.0.1:40834 accepted tcp:192.168.77.1:10123 [socks >> proxy]

is there more verbose level than debug ?

[Fangliding]

"error" ?
https://xtls.github.io/config/log.html#logobject

[shakibamoshiri]

"error" ? https://xtls.github.io/config/log.html#logobject

starting fresh

 "log": {
    "loglevel": "debug",
    "error": "./xray-error.log",
    "access": "./xray-access.log",
    "debug": true
  },

running

curl --socks5-hostname localhost:2080 api.ipify.org; echo
curl --socks5-hostname localhost:2080 api.ipify.org; echo

### then
iperf3 --client 192.168.77.1 --port 10123
iperf3: error - control socket has closed unexpectedly

logs

 cat *.log
2024/05/15 18:21:39 tcp:127.0.0.1:37194 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 18:21:46 tcp:127.0.0.1:57994 accepted tcp:api.ipify.org:80 [socks >> proxy]
2024/05/15 18:22:10 tcp:127.0.0.1:45336 accepted tcp:192.168.77.1:10123 [socks >> proxy]
2024/05/15 18:21:32 [Warning] core: Xray 1.8.11 started

screenshot

[Fangliding]

?
There shoud be such info [Info] transport/internet/tcp: listening TCP on 127.0.0.1:xxxx why yours only have a start message...
Maybe delete "error" and “access" then run xray in console and get a screen shoot?

[shakibamoshiri]

? There shoud be such info [Info] transport/internet/tcp: listening TCP on 127.0.0.1:xxxx why yours only have a start message... Maybe delete "error" and “access" then run xray in console and get a screen shoot?

I did but the log is the same

here is xray full screenshot

here is hy2 full screenshot (iperf3 works)

further checking with tcpdump shows that using hy2 packets reach the host : 192.168.77.1 but with xray never

tcpdump on server for hy2

tcpdump -q -n -t  -i any host 192.168.77.1
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
lo    In  IP 192.168.77.1.53758 > 192.168.77.1.5201: tcp 0
lo    In  IP 192.168.77.1.5201 > 192.168.77.1.53758: tcp 0
lo    In  IP 192.168.77.1.53758 > 192.168.77.1.5201: tcp 0
lo    In  IP 192.168.77.1.53758 > 192.168.77.1.5201: tcp 37
lo    In  IP 192.168.77.1.5201 > 192.168.77.1.53758: tcp 0
lo    In  IP 192.168.77.1.5201 > 192.168.77.1.53758: tcp 1
lo    In  IP 192.168.77.1.53758 > 192.168.77.1.5201: tcp 0
...
...

tcpdump xray on the same server

tcpdump -q -n -t  -i any host 192.168.77.1
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

# nothing is captured

the problem seems on how Xray handles NAT !? or OUTPUTing packets !? at server side

[Fangliding]

Attempt to capture packets on the client side to confirm if any socks5 requests from tun2socks have been sent to xray inbound

[shakibamoshiri]

2024/05/15 18:22:10 tcp:127.0.0.1:45336 accepted tcp:192.168.77.1:10123 [socks >> proxy]

did you check log ?

2024/05/15 18:22:10 tcp:127.0.0.1:45336 accepted tcp:192.168.77.1:10123 [socks >> proxy]

the xray client accepts the connection, it is received at server side by xray server , but xray server does not forward/deliver packets to 192.168.77.1. it seems the issue is from server side that does not forward/deliver to 192.168.77.1

it is strange to me! no one knows how to test the speed ? how developers and maintainers test xray performance ?
At least there should be a doc on how to test the speed

[Fangliding]

There are many ways for testing performance, and using tun2socks is a stupid one.(to be honest, I don't very care about it. If necessary, I can flip it out. It's basically just some meaningless charts)
I'm just curious why tun2socks doesn't work(since curl can use it)
You seem to have a vless server, what's it's log
I noticed that you seem to have specified a port 10123 for the iperf of vless, but hy2 has not, why?

[shakibamoshiri]

There are many ways for testing performance, and using tun2socks is a stupid one.(to be honest, I don't very care about it. If necessary, I can flip it out. It's basically just some meaningless charts) I'm just curious why tun2socks doesn't work(since curl can use it) You seem to have a vless server, what's it's log I noticed that you seem to have specified a port 10123 for the iperf of vless, but hy2 has not, why?

how do you test a server to server without tun2socks ? When the VPN server does not support tun/tap devices this is the simplest way of testing and this is why xray clients on phones handle the connections.

the iperf3 port does not matter. (could be the default one or a user-defined one)
the problem to my understanding is the server side configuration , it is not client side nor is tun2socks

Assuming the stupidity of this method , please share some clever ways of testing server to server with iperf over Xray protocols.

By the way I am not looking for speedtest test or speed.cloudflare.com test , these are not accurate nor are informative

[Fangliding]

Well, you can add this in your config

    {
      "listen": "127.0.0.1",
      "port": 5210,
      "protocol": "dokodemo-door",
      "settings": {
        "address": "192.168.77.1",
        "port": 5210,
        "network": "tcp，udp"
      },
      "tag": "doko"
      }

This will let xray listen on your client‘s port 5210 and forward all traffic to 192.168.77.1:5210(according to the info your provided, it should be an iperf3 server)
Speaking of which, I noticed that many default xray configuration files block connections sent to private IP range (like 192.168. x.x), maybe the problem lies here

[shakibamoshiri]

finally could make it work.

The main issue was the setup part using xui(s), disabling xui and running it directly + enabling debug showed that private IPs are blocked . Removing this part let the tun2socks functions as expected

Also tested your suggestion with dokodemo worked well

probably the documentation should be updated (or I should take a part)

[yuhan6665]

Thanks @shakibamoshiri