vmess+ws+tls works on windows/mac, but NOT in Android #139

ea0fdb · 2022-10-16T16:52:49Z

We have run two servers a while ago. All of the client was working fine until a week ago. Now computer clients are working fine, but android clients don't work in any of the servers. We have tested v2rayNG with both versions 1.7.20 and 1.7.21 (uTLS fingerprint).
It doesn't work on Raspberry Pi either. Maybe related to arm processors.

Server/Protocol: vmess+ws+tls with v2fly-core and nginx
Working Clients: Windows / Linux / Mac, with v2fly-core. iOS is working too with Shadowrocket.
Not Working Clients: Almost any android client, including v2rayNG. Raspberry Pi Linux with v2fly-core.
Country: Iran
Related Issue & server logs: 2dust/v2rayNG#1685

Please share any idea you have about this situation.
Maybe this was happened in China before.

Thanks.

ea0fdb · 2022-10-16T17:10:44Z

@wkrp
Based on this issue, It seems that this is happening in China too. So, please add the China label too.

wkrp · 2022-10-16T17:18:22Z

It doesn't work on Raspberry Pi either. Maybe related to arm processors.

It is possible it has something to do with the processor type. OONI reported something similar once. (However, I don't know how this kind of hardware dependence might interact with uTLS.)

In mid-April 2021, we discovered the likely cause of blocking. V2Ray developers previously documented this issue in v2fly/v2ray-core#557.

The root cause is that the TLS Client Hello generated by Golang on Android devices changes depending on Golang’s understanding of the hardware capabilities. Golang’s TLS library uses a hardware implementation of AES when possible. Otherwise, it falls back to AES code written in pure Go. According to the official docs, such code is not constant over time. Probably, for this reason, when Golang thinks that there is no hardware support for AES, it reorders the preferred cipher list in the Client Hello. This reordering aims to significantly reduce the probability that the server will select AES ciphers by moving such ciphers towards the bottom of the list.

There was also a report in another thread that snowflake-client worked on macOS and Windows desktop, but not in Orbot on Android or iOS.

It will help if you can get a packet capture of the the TLS Client Hello under both conditions, in order to check whether there are any differences. On desktop, you can use tcpdump or Wireshark. The tcpdump command will be something like this (replace $V2RAY_HOST and $V2RAY_PORT):

tcpdump -U -w v2ray-desktop.pcap -f "host $V2RAY_HOST and port $V2RAY_PORT"

On Android, I have had luck using PCAPdroid. You can select a specific app to capture traffic from and save to a pcap file.

You can then upload the pcap files to https://tlsfingerprint.io/pcap and see if they have the same fingerprint.

To see what's inside each Client Hello record, you can open the pcap files in Wireshark, select the "Client Hello" packet, right-click "Secure Sockets Layer" in the tree view, select "Expand Subtrees", then right-click again and "Copy → All Visible Selected Tree Items". Then paste it into a file. Or you can use the tshark command like so:

tshark -V -O ssl -Y ssl.handshake.ciphersuites -r v2ray-desktop.pcap

Don't post the raw pcap files here, because they may contain IP addresses you do not want to reveal.

ea0fdb · 2022-10-16T21:07:05Z

@wkrp Thanks for your reply.

I have run the experiments. I used PCAPdroid on Android and tcpdump on other platforms. The results are summarized in the table below.

	Server	Device	Client	Is Working?	TLS Fingerprint
1	S1:vmess+ws+tls	mac	v2fly (5.1.0)	yes	844**********1ad
2	S1:vmess+ws+tls	android	xray (1.6.0)	no	750e3f0f585283bd
3	S1:vmess+ws+tls	raspberry pi	v2fly (5.1.0)	no	750e3f0f585283bd
----	-----------------	--------------	---------------	-------------	-----------------
4	S2:vless+ws+tls	raspberry pi	v2fly (5.1.0)	no	750e3f0f585283bd
5	S2:vless+ws+tls	raspberry pi	xray (1.6.0)	no	750e3f0f585283bd
6	S2:vless+ws+tls	raspberry pi	xray (1.6.0) + uTLS (any browser)	no	750e3f0f585283bd

Both servers S1 and S2 are working for linux/mac/windows. None of them are working for Raspberry Pi or Android device (with V2rayNG).
The servers have different IPs, Hostnames, and Protocols.

It's interesting that:

The use of uTLS didn't change the fingerprint for these device and protocols. It's worth mentioning that the uTLS configuration will change the fingerprint for some other protocols, even in the Raspberry Pi.
In these experiments the fingerprint of ARM devices are 750e3f0f585283bd for both servers.

I suspect that the 750e3f0f585283bd fingerprint causes the blocking.

OnceUponATimeInAmerica · 2022-10-17T03:37:28Z

You could try a VLESS+TCP+TLS config (e.g. using ProxySU) and then set the uTLS fingerprint to "Chrome" in V2rayNG (new release). Also Trojan configs seem to work well with custom uTLS fingerprints.

Or you could use a client like Sagetnet/Matsuri and try setting "browser forwarding" in the server's options.

irtux · 2022-10-17T03:50:32Z

It's all about interfering on TLS handshake by government. even I can not login to Gmail today by getting this msg (The server cannot process the request because it is malformed. It should not be retried.) blocking and manipulating on port 443 and TLS caused all of these problems.

wkrp · 2022-10-17T03:53:05Z

@ea0fdb thank you, that is super helpful. It looks to me like uTLS support is not working—it may be a bug in xray or possibly in uTLS.

The fingerprint 750e3f0f585283bd looks like the default Go crypto/tls fingerprint for platforms without AES acceleration hardware since go1.17. (Take the list cipherSuitesPreferenceOrderNoAES, append defaultCipherSuitesTLS13NoAES, and subtract disabledCipherSuites, and you get exactly the ciphersuite list of 750e3f0f585283bd.) So yeah, it looks like uTLS is not taking effect, for some reason.

@gaukas, you are more experienced. Is there any reason why uTLS might fail to take effect on ARM? Or is it more likely that xray is using uTLS ineffectively?

@ea0fdb, are you able to test uTLS support in xray on a non-ARM platform? If it changes the fingerprint on non-ARM but does not change the fingerprint on ARM, it would tell us something.
The fingerprint 750e3f0f585283bd is very similar to the fingerprint adfe55afa6f23950 which I observed being produced by snowflake-client in Tor Browser for Linux and Orbot for Android at Unexplained drop in Snowflake client polls and bandwidth, testers wanted #131 (comment). tlsfingerprint.io has a fingerprint comparison feature. The differences are:
- adfe55afa6f23950 additionally has "h2" in ALPN.
- adfe55afa6f23950 additionally supports TLS 1.0 and TLS 1.1.
I now suspect that the reason I got the fingerprint adfe55afa6f23950 with Tor Browser for Linux is that I was running it in a VM. The VM may not have indicated support for hardware acceleration of AES, which would cause Go crypto/tls to switch to the NoAES ciphersuites, the same as on ARM.

This all gives us a good hypothesis as to the features used in recent TLS blocking, in Iran at least. It includes the fingerprints of go1.17+ crypto/tls without AES acceleration, which are likely the most commonly occurring fingerprints among Go-based tools running on mobile platforms. These fingerprints include at least 750e3f0f585283bd and adfe55afa6f23950.

gaukas · 2022-10-17T04:16:13Z

Is there any reason why uTLS might fail to take effect on ARM

I am not aware of any problem that would cause the fingerprint parroting silently fail on ARM in uTLS.

I just ran a test on my Raspberry Pi (armv7 running armv6 go), uTLS worked fine and parrots both Chrome and Firefox fingerprints correctly. While being not very familiar with the code base of xray, I couldn't answer the question of what causes uTLS not functioning with xray.

gfw-report · 2022-10-17T04:22:54Z

I suspect that the 750e3f0f585283bd fingerprint causes the blocking.

Great analysis, @ea0fdb!

We have not been able to trigger the blocking with 750e3f0f585283bd fingerprint yet, however. In particular, we wrote a program that uses uTLS to make TLS connections with that fingerprint. We then did the following three experiments. In all three experiments, we captured the packets from both client and server to observe if there is any packet manipulation :

repetitively sending TLS client hello messages to both 443 and non-443 port of a host outside of China. The host is actually a TCP sink server, which only accepts the hello message but does not respond any data;
repetitively making TLS handshakes to both 443 and non-443 port of a host outside of China. The host is actually a TLS sink server, which only accepts the TLS handshakes but does not respond any data;
repetitively making TLS handshakes, and then follow by sending 517 bytes and then 126 bytes traffic, to mimic TLS-over-TLS's packet length through the TLS tunnel, to both 443 and non-443 port of a host outside of China. The host is still a TLS sink server, which only accepts the TLS handshakes and the following data but does not respond any data.

In all three cases, we cannot trigger the blocking immediately. Note that this does not necessarily mean that the censor does not consider TLS fingerprint when making a blocking decision, rather, it just says the TLS fingerprint is not a sufficient condition to trigger the blocking. There may be other factors the censor considers together with the fingerprint, for example:

The censor may only apply the blocking to certain IP addresses, like what we've seen for the dynamic blocking of Shadowsocks, and our server was not within those IP addresses.
The censor may consider in addition consider packet length in TLS handshake and the first several packets after the handshake.
The censor may require seeing more traffic than we sent to trigger the blocking.

In any case, since 750e3f0f585283bd is a very unique fingerprint, we should consider changing it.

BTW, we have received some reports that support the TLS fingerprint blocking hypothesis. In particular, the person setup two open ports on the same server, for one port, people used the trojan-go with latest uTLS, which is not blocked; for the other port, people used an android client, which was blocked.

//cc @2dust

ea0fdb · 2022-10-17T07:05:48Z

Hi everyone. Thanks for your replies.

Are you able to test uTLS support in xray on a non-ARM platform? If it changes the fingerprint on non-ARM but does not change the fingerprint on ARM, it would tell us something.

@wkrp For vmess+ws+tls/vless+ws+tls on a non-ARM device, using uTLS with any browser, produces the same TLS Fingerprint as not using uTLS. All of them are still connecting and working fine.

Now my concern is why doesn't uTLS change the TLS Fingerprint for vmess+ws+tls/vless+ws+tls. Actually I have already seen that uTLS is working with xray for vless+tls (without websocket).

It seems that the issue is not necessarily about ARM devices, as I suspect that it's about effectiveness of uTLS with xray when using websocket.

Note that I'm changing this key in the json configuration to use uTLS with xray 1.6.0.

{
  "outbounds": [
    {
      "streamSettings": {
        "tlsSettings": {
          "fingerprint": "chrome",
          ...
        }, ...
      }, ...
    },
  ], ...
}

gaukas · 2022-10-17T14:21:01Z

as I suspect that it's about effectiveness of uTLS with xray when using websocket

Nice catch. This reminds me that there's no H2 support for websocket libraries in Go(unfortunately). So possibly xray had to override ALPN with only http1.1.

wkrp · 2022-10-17T17:42:47Z

From #136 (comment), I found the recent commit XTLS/Xray-core@93c7ebe in xray, with commit message "Added utls to http2 transport". Maybe it helps?

klzgrad · 2022-10-18T03:19:09Z

uTLS should verify on its own its fingerprint on the wire and report error if not so, instead of relying on user tcpdumping. Though I don't know if this is technically doable.

gaukas · 2022-10-18T03:33:27Z

uTLS should verify on its own its fingerprint on the wire and report error if not so

Definitely possible, we have UnmarshalClientHello() to convert []byte into ClientHelloMsg. But would you like to elaborate on how could it be useful? Given that uTLS doesn't implement a hash-to-ClientHello converter, just simply verifying if clientHello == UnmarshalClientHello(clientHello.marshal()) looks redundant to me.

(Assuming all Extensions/FakeExtensions are implemented correctly, of course)

HirbodBehnam · 2022-10-18T06:48:06Z

Hello
I actually like to note that I did another pull request which adds utls to websocket connections of xray. You can read about the implementation here: XTLS/Xray-core#1256
Please do leave a comment if you think I did anything wrong!

P.S: As a workaround you can try specifying something like "cipherSuites":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" in the tls settings of xray client. This will change the fingerprint thus the traffic will be allowed.

gfw-report · 2022-10-18T08:54:44Z

This is a small script that we've been using to get the TLS fingerprint IDs from pcap files, which you may find handy:

https://github.com/gfw-report/get-tls-fingerprints

//cc @ea0fdb @tomac4t

Salarshafiei · 2022-10-20T22:38:42Z

@0xLem0nade I tried installing your .apk but it fails to install on my phone.

Salarshafiei · 2022-10-20T23:38:51Z

@0xLem0nade I uninstalled the PS version first and then I tried installing each and every file in the assets. Non of them could be installed unfortunately. It just says App not installed.
My phone is a Samsung Note 9 (exynos9810), updated to the latest version. Android V10, aarch64 (from CPU-Z)

irtux · 2022-10-21T05:10:26Z

@0xLem0nade I uninstalled the PS version first and then I tried installing each and every file in the assets. Non of them could be installed unfortunately. It just says App not installed. My phone is a Samsung Note 9 (exynos9810), updated to the latest version. Android V10, aarch64 (from CPU-Z)

Okay I did reset some signing configs and uploaded the new builds here. Get the v2rayNG_1.7.22_arm64-v8a.apk, reboot your device and then install the apk. Let me know if it works!

Hi, The problem still there, same as before, no problem with other Xray clients on Android, but v2rayNG 1.7.22 still not connecting .

ea0fdb · 2022-10-22T07:26:13Z

Hi,

With the new pre-release of v2rayNG 1.7.23, uTLS is now working fine and the issue is resolved.
We can now close the issue if there is not anything else to discuss.

Thanks everyone for your time and supports.
@HirbodBehnam Thanks for your contribution (دمت گرم 😁).

Ka6k · 2022-10-22T20:42:15Z

Still uTLS is not wotking for TCP+XTLS. For VLESS+TCP+XTLS & Trojan+TCP+XTLS the fingerprint remains the same for any browser option as not using uTLS.

ghost · 2022-10-23T19:23:56Z

Blog post expanding on the @0xLem0nade config file for v2rayNG + HTTP/2 + TLS from the other thread is at https://freejohn123.github.io/2022/10/23/v2rayNG-http2-tls.html

irtux · 2022-10-24T03:46:48Z

Blog post expanding on the @0xLem0nade config file for v2rayNG + HTTP/2 + TLS from the other thread is at https://freejohn123.github.io/2022/10/23/v2rayNG-http2-tls.html

Hi and thank you, but still have problem with this config, even try to change everything and testing ...
The time is correct for both server and client.

"TLS handshake error from MY_IP: read tcp SERVER_IP:443->MY_IP: i/o timeout"

HirbodBehnam · 2022-10-24T03:59:02Z

As a side note, you might want to use firefox instead of chrome when using the fingerprint option. Iran blocks chrome traffic to some IP addresses. For example my own server is from hetzner and I do have the exact problem; So I simply use firefox's fingerprint.
Probably related: #118

free-the-internet · 2022-10-24T14:17:38Z

Blog post expanding on the @0xLem0nade config file for v2rayNG + HTTP/2 + TLS from the other thread is at https://freejohn123.github.io/2022/10/23/v2rayNG-http2-tls.html

Hi and thank you, but still have problem with this config, even try to change everything and testing ... The time is correct for both server and client.

"TLS handshake error from MY_IP: read tcp SERVER_IP:443->MY_IP: i/o timeout"

This is also happened to me. @HirbodBehnam Changing fingerprint to Firefox is not working.

free-the-internet · 2022-10-24T22:08:05Z

Blog post expanding on the @0xLem0nade config file for v2rayNG + HTTP/2 + TLS from the other thread is at https://freejohn123.github.io/2022/10/23/v2rayNG-http2-tls.html

Hi and thank you, but still have problem with this config, even try to change everything and testing ... The time is correct for both server and client.
"TLS handshake error from MY_IP: read tcp SERVER_IP:443->MY_IP: i/o timeout"

This is also happened to me. @HirbodBehnam Changing fingerprint to Firefox is not working.

Hi, apparently, free domain was blocked. I moved to new domain and now it is working.

cross-hello · 2022-10-25T01:26:58Z

You have done a good job. תודה אתה

…

---------------------------------------- *From: *free-the-internet ***@***.***> *To: *net4people/bbs ***@***.***> *CC: *Subscribed ***@***.***> *Date: *Oct 25, 2022 06:08:31 *Subject: *Re: [net4people/bbs] vmess+ws+tls works on windows/mac, but NOT in Android (Issue #139)

Blog post expanding on the @0xLem0nade[https://github.com/0xLem0nade] config file for v2rayNG + HTTP/2 + TLS from the other thread is at https://freejohn123.github.io/2022/10/23/v2rayNG-http2-tls.html Hi and thank you, but still have problem with this config, even try to change everything and testing ... The time is correct for both server and client. "TLS handshake error from MY_IP: read tcp SERVER_IP:443->MY_IP: i/o timeout" This is also happened to me. @HirbodBehnam[https://github.com/HirbodBehnam] Changing fingerprint to Firefox is not working. Hi, apparently, free domain was blocked. I moved to new domain and now it is working. — Reply to this email directly, view it on GitHub[#139 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AKGBAYFASGTPSWNYIME25BLWE4CF5ANCNFSM6AAAAAARGNNA3M]. You are receiving this because you are subscribed to this thread.[Tracking image][https://github.com/notifications/beacon/AKGBAYEBV7LZ4CLZDDDVPKDWE4CF5A5CNFSM6AAAAAARGNNA3OWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSM34A7Q.gif]Message ID: ***@***.***>

poorp · 2022-10-26T15:25:35Z

Hi, I didn't really follow the discussion here but here is some added info:
I use vless+ws and it's working well but when I try to add TLS it doesn't work. I tried Cloudflare and Hostinger as my CDN.

SakuraSakuraSakuraChan · 2022-11-02T05:49:15Z

@wkrp Based on this issue, It seems that this is happening in China too. So, please add the China label too.

I think this issue is about configuration importing affairs, not about blocking

wkrp added the Iran label Oct 16, 2022

wkrp added the China label Oct 16, 2022

wkrp mentioned this issue Oct 17, 2022

Unexplained drop in Snowflake client polls and bandwidth, testers wanted #131

Open

GibMeMyPacket mentioned this issue Oct 17, 2022

section Censorship mullvad/mullvadvpn-app#4050

Open

gaukas mentioned this issue Oct 18, 2022

Optionally verifying the ClientHello sent on wire refraction-networking/utls#126

Open

wkrp mentioned this issue Oct 19, 2022

Shutdowns, intensified blocking in Iran since 2022-09-21 #125

Open

Ka6k mentioned this issue Oct 22, 2022

Add uTLS option to xray-plugin teddysun/xray-plugin#17

Open

wkrp mentioned this issue Nov 1, 2022

Connectivity problem over shadowsocks-v2ray-websocket and vless on wifi #151

Closed

wkrp mentioned this issue Nov 9, 2022

Chrome 107.0.5304.88 TLS fingerprint is (partly) blocked in Iran, what are the consequences for TLS based proxies? #153

Open

PanXXHH mentioned this issue Jun 10, 2023

电脑连接正常，手机连接报502错误 master-coder-ll/v2ray-web-manager#239

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vmess+ws+tls works on windows/mac, but NOT in Android #139

vmess+ws+tls works on windows/mac, but NOT in Android #139

ea0fdb commented Oct 16, 2022 •

edited

ea0fdb commented Oct 16, 2022 •

edited

wkrp commented Oct 16, 2022

ea0fdb commented Oct 16, 2022 •

edited

OnceUponATimeInAmerica commented Oct 17, 2022 •

edited

irtux commented Oct 17, 2022

wkrp commented Oct 17, 2022 •

edited

gaukas commented Oct 17, 2022

gfw-report commented Oct 17, 2022 •

edited

ea0fdb commented Oct 17, 2022 •

edited

gaukas commented Oct 17, 2022

wkrp commented Oct 17, 2022

klzgrad commented Oct 18, 2022

gaukas commented Oct 18, 2022 •

edited

HirbodBehnam commented Oct 18, 2022 •

edited

gfw-report commented Oct 18, 2022

Salarshafiei commented Oct 20, 2022

Salarshafiei commented Oct 20, 2022 •

edited

irtux commented Oct 21, 2022 •

edited

ea0fdb commented Oct 22, 2022

Ka6k commented Oct 22, 2022

ghost commented Oct 23, 2022

irtux commented Oct 24, 2022

HirbodBehnam commented Oct 24, 2022

free-the-internet commented Oct 24, 2022

free-the-internet commented Oct 24, 2022

cross-hello commented Oct 25, 2022 via email

poorp commented Oct 26, 2022

SakuraSakuraSakuraChan commented Nov 2, 2022

vmess+ws+tls works on windows/mac, but NOT in Android #139

vmess+ws+tls works on windows/mac, but NOT in Android #139

Comments

ea0fdb commented Oct 16, 2022 • edited

ea0fdb commented Oct 16, 2022 • edited

wkrp commented Oct 16, 2022

ea0fdb commented Oct 16, 2022 • edited

OnceUponATimeInAmerica commented Oct 17, 2022 • edited

irtux commented Oct 17, 2022

wkrp commented Oct 17, 2022 • edited

gaukas commented Oct 17, 2022

gfw-report commented Oct 17, 2022 • edited

ea0fdb commented Oct 17, 2022 • edited

gaukas commented Oct 17, 2022

wkrp commented Oct 17, 2022

klzgrad commented Oct 18, 2022

gaukas commented Oct 18, 2022 • edited

HirbodBehnam commented Oct 18, 2022 • edited

gfw-report commented Oct 18, 2022

Salarshafiei commented Oct 20, 2022

Salarshafiei commented Oct 20, 2022 • edited

irtux commented Oct 21, 2022 • edited

ea0fdb commented Oct 22, 2022

Ka6k commented Oct 22, 2022

ghost commented Oct 23, 2022

irtux commented Oct 24, 2022

HirbodBehnam commented Oct 24, 2022

free-the-internet commented Oct 24, 2022

free-the-internet commented Oct 24, 2022

cross-hello commented Oct 25, 2022 via email

poorp commented Oct 26, 2022

SakuraSakuraSakuraChan commented Nov 2, 2022

ea0fdb commented Oct 16, 2022 •

edited

ea0fdb commented Oct 16, 2022 •

edited

ea0fdb commented Oct 16, 2022 •

edited

OnceUponATimeInAmerica commented Oct 17, 2022 •

edited

wkrp commented Oct 17, 2022 •

edited

gfw-report commented Oct 17, 2022 •

edited

ea0fdb commented Oct 17, 2022 •

edited

gaukas commented Oct 18, 2022 •

edited

HirbodBehnam commented Oct 18, 2022 •

edited

Salarshafiei commented Oct 20, 2022 •

edited

irtux commented Oct 21, 2022 •

edited