New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
curl-impersonate: An alternative to utls? #345
Comments
related thread: refraction-networking/utls#103 |
You raised a very good question: what are the better approaches to TLS (client library) parroting than uTLS? One common challenge in maintaining parroting library such as uTLS is that popular implementations are getting highly volatile with new technologies being rolled out, and it could take huge effort in keeping up with the latest version. Also, there might be behaviors that we don't completely understand and/or cannot efficiently mimic (e.g., TLS handshake message fragmentation, etc), resulting in imperfect parrots. Per The Parrot is Dead, parroting will never be an efficient approach and instead we should try to use as much as the "real implementation": Instead of pretending to be doing a TLS handshake (shadow-tls), do a real TLS handshake and TLS communication (XTLS/Reality); Instead of pretending to be Chrome (uTLS), use Chrome (naiveproxy). So now the challenge is: how do we efficiently use these existing popular implementations in tools that are written in a different programming language? (also consider the cross-platform compatibility) |
I hope water can be one of the answers. (despite it could be heavier than native code) |
How about NSS? It's the crypto library of Firefox, Thunderbird, etc. |
It's good, but still facing the same challenge as naiveproxy(cronet) that you need a cross-platform, cross-language interface. That's true for all libraries no matter how well adopted it is. |
I think more important than "the best approach" in an academic sense (parroting vs not, full browser dialler vs naiveproxy approach vs only using BoringSSL/NSS) is which approach has the most resources thrown behind it, is most easily deployable, and therefore has a chance of continuous funding. I see curl-impersonate in an interesting position here where it is already sufficiently easy to use for commercial applications across a variety of languages, since bindings to curl exist for a bunch of them. E.g. just the Python bindings for curl-impersonate have 1.2k stars curl-impersonate provides builds for a variety of SSL libraries, one of them is NSS, necessary to impersonate Firefox. |
I guess we are almost on the same page. I totally agree with you that curl-impersonate shows very promising future with its great flexibility of being able to directly replace curl (including libcurl) which is very popular and has great support across different programming languages. However in practice, let's say how hard is it to integrate curl-impersonate with Go (since most of the circumvention tools are written in it)? Not very easy when without CGO, I guess. Here, a common challenge for such multi-binding library is they almost always rely on C, which is fine when being done with good flexibility on devices such as PC or full-scaled servers, but not necessarily for less flexible platforms like mobile phones, gateway/routers, IoT devices, etc. |
This is a problem I've been thinking about for over a year. While WATER and wasm is obviously an elegant solution, it requires embedding some kind of runtime in any software that intends to use the wasm binary. So it's fundamentally different from the traditional library/binding approach. So is the concern that libraries and software written in various languages are not exactly portable/buildable on all platforms that require it? Do we need better build and cross-build tools? Cross compilation is a massive pain, I solve this with my oven. I think through the use of Nix as the build environment and Zig as a C toolchain and cross-compiler, there's a staggering amount of libraries and tools that can be cross-compiled and linked for a lot of platforms. (wasm can of course be a build target too!) Would the perfect world be that transports can be written in various languages, and are cross-compiled to static libraries and wasm binaries? (sorry if this diverges from the topic) |
Ideally, yes, we want natively written transport modules. Pluggable Transports is a good example for that, which also reveals some inherent challenges in doing so such as high maintenance cost. Cross-compilation (into shared library) is on the other hand, shifting the challenge to how to efficiently integrating the library with various programming languages on various platforms. It is not trivial and if solved, the development and maintenance cost of different circumvention tools can be greatly reduced, leading to much better efficiency in developing and deploying new techniques. |
To me this sounds like a combination of a tooling problem (lack of widespread use of versatile build tools) and a code problem (lack of bindings for each library).
Indeed, we wouldn't be worried and limited by something like CGO if the libraries we intend to import were static and would build on any platform/arch. Reaching out to whatever project to improve their code/build system is a high barrier to entry for those who may only wish to write Go or Rust or whatever language they choose. |
It's one thing to have a cross compile toolchain for oneself, it's another thing to convince downstream application developers to migrate their original toolchains to your toolchain to link this library just to have a marginally better defense against the theoretical threat of being an imperfect parrot, which almost never works because being a better parrot is almost never a higher priority than the user facing features that the application developers deal with. |
I remember those conversations: @klzgrad proposed to integrate chromium network stack into v2ray (or was it another fork?), and @RPRX was pushing back against using CGO. If I find them again I will link them here.
I started a new project in Rust to play around with a few tls-fingerprinting ideas that can evolve independently of v2ray, but still bundled in a single project/"platform" so at least there are not too many "microservices" flying around. https://github.com/mmmray/minidialer I hope that with Rust as the basis, it's easier to link various C libraries. And since at least some parts of it are very generic and not married to specific v2ray transports, it can hopefully find use outside of censorship circumvention and attract contributors from other communities as well. (See also the Future Ideas section) Maybe the solution all along was to to bundle v2ray + chrome into a docker container, then converting the container back to a binary... |
I think regardless of Go or Rust, if we try hard enough, we can static link whatever we want. Just need to configure the linker correctly. It just sucks for massive projects. |
Not without CGO. One reason why we don't like CGO is that all the dependency libraries also need to be built for the target (which involves excessive amount of troubleshooting), and it is not realistic to automate the cross-build on update. Plus the fact that the interfaces are almost never designed to be C-compatible without explicitly keeping that in mind, thus a C-compatible wrapper for each dependency is usually required. |
I asked nekohasekai to integrate Cronet into Go, there was something half to almost done, and he seemed to have lost interest because this was all pain and no gain. Maybe he can explain why this is too much for a Go developer. I asked v2fly people about remote calling a Cronet process without messing with Go ABI. Haven't seen real interest in it. I don't know if RPRX did any pushback except his empty fork of naiveproxy.
If it sucks enough, it becomes intractable. If it's a smaller project CGO may be easier. Chromium's build system and Go's build system are two large beasts with extreme impedance mismatch. First it takes a large amount of klzgrad/naiveproxy@4e5adba special casing just to convince Go/CGO to complete the basic linking of libcronet.a into executable by "translating" the linker flags Chromium uses to link libcronet.a. Then these translated linker flags have to be transformed again into a Go package SagerNet/cronet-go@61c4e93 with hardcoded linker flags so it can be used by other Go packages. Then on each Chromium update or Go update, these two transformations of linker flags could break on any of the tens of platforms/architectures. The linker flags required by Chromium look like this: |
I mean, if there is already a separate cronet process and that's acceptable UX, I am not sure that v2fly or v2ray have to do anything for integration. You can re-implement xray's dialer (the JS part) in C, or wrap chromium network stack in a basic HTTP reverse proxy. Then it will work with all of them. |
I think people expect some tighter integration, something more efficient, or it's indistinguishable from just running naiveproxy.exe along with v2ray.exe with the latter being a thin wrapper. |
v2ray/discussion#754 (comment)
这个 empty fork 指的是 https://github.com/XTLS/naiveproxy-reality 吧,它是想 给 naiveproxy 的 TLS 加上 REALITY, 不过,这是件很简单的事情,客户端主要是修改 session id、修改验证证书的逻辑,所需代码很少,就像 给 uTLS 加上 REALITY This empty fork refers to https://github.com/XTLS/naiveproxy-reality. It's trying to add REALITY to naiveproxy's TLS, However, it's a very simple thing to do, the client side is mainly to modify the session id, modify the logic of the authentication certificate, the required code is very little, just like add REALITY to uTLS |
此外:
|
I think this traffic characteristic could be solved if the browser dialer was reversed. Instead of having the browser open
I am also optimistic that browser dialers are not yet a dead end with regard to performance, even if they are JS. I get 1 Mbps download speed with the one from xray, but minidialer can do 300 Mbps (saturating bandwidth). I am hoping for somebody to reproduce those results. |
continuing some xray-specific questions here: XTLS/Xray-core#3263 |
In a similar theme as #336, I see significant overlap between the technical solutions the scraping/botting community comes up with, and the needs for censorship circumvention. This is interesting because there is some commercial/industrial power (and therefore developer time) behind developing web scraping techniques, more so than behind censorship circumvention.
This week I came across https://github.com/lwthiker/curl-impersonate, which appears to be identical in function to uTLS, except written by completely different people, written for a different purpose, and using the same TLS libraries as the mimicked browsers. It seems to me that the level of impersonation it provides might lie somewhere between naiveproxy and utls because of that.
I wonder if anybody has the resources to try this tool in the field and see how well it performs against e.g. xray's integration of utls.
The text was updated successfully, but these errors were encountered: