rustls client flagged as bot by Cloudflare

[darind]

I am using rustls 0.21.7 on MacOS 13.5.2 to make a TLS request to chat.openai.com:443:

use std::io::{Read, Write};
use std::{sync::Arc, net::TcpStream};
use rustls::{Certificate, RootCertStore, ClientConfig, Stream, ClientConnection};

fn main() {
    let mut root_store = RootCertStore::empty();
    for cert in rustls_native_certs::load_native_certs().expect("could not load platform certs") {
        root_store
            .add(&Certificate(cert.0))
            .unwrap();
    }

    let mut config = ClientConfig::builder()
        .with_safe_defaults()
        .with_root_certificates(root_store)
        .with_no_client_auth();

    config.alpn_protocols.push("http/1.1".as_bytes().to_vec());

    let mut conn = ClientConnection::new(
        Arc::new(config),
        "chat.openai.com".try_into().unwrap()
    ).unwrap();

    let mut sock = TcpStream::connect("chat.openai.com:443").unwrap();
    let mut tls = Stream::new(&mut conn, & mut sock);

    tls.write("GET / HTTP/1.1\r\nHost: chat.openai.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36\r\nConnection: close\r\n\r\n".as_bytes()).unwrap();

    let mut buf = vec![0; 2048];
    let len = tls.read(&mut buf).unwrap();
    std::io::stdout().write_all(&buf[..len]).unwrap();
}

This returns HTTP/1.1 403 Forbidden.

I suspect that the JA3 fingerprint that rustls is generating is not matching with the one generated by popular web browsers and Cloudflare bot protection is detecting that the ClientHello is not coming from a whitelisted source.

Is it possible to control the list of tls extensions and cipher suites sent by a rustls client?

[djc]

I think this is a duplicate of #1421. Have you done some testing to ascertain that the TLS-level fingerprinting is what is causing the issues here? (Like running the same minimal test with native-tls.) Your fake HTTP request (or even the fact that this is a HTTP/1.1 request rather than H2 or H3) seem much more likely candidates.

[darind]

Not sure about native-tls, but a simple curl on the command line works fine:

curl --http1.1 \
     -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" \
     https://chat.openai.com \
     -vp

I guess curl is using openssl under the hood?

[djc]

curl has like 9 different TLS backends, so it really depends where you got your curl. curl -V should tell you more.

[darind]

Tried on Windows (schannel):

curl 8.0.1 (Windows) libcurl/8.0.1 Schannel WinIDN
Release-Date: 2023-03-20
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS HTTPS-proxy IDN IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI threadsafe Unicode UnixSockets

and on MacOS (I have a custom built curl here):

curl 7.85.0-DEV (x86_64-apple-darwin21.6.0) libcurl/7.85.0-DEV (SecureTransport) BoringSSL zlib/1.2.11 brotli/1.0.9 zstd/1.5.5 libidn2/2.3.4 libssh2/1.11.0 nghttp2/1.54.0 quiche/0.14.0 librtmp/2.3
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets zstd

curl works fine on both Windows and MacOS (assuming the propert User-Agent header is provided).

MacOS:

curl --http1.1 -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" https://chat.openai.com -vp

Windows:

curl --http1.1 -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" https://chat.openai.com -vp

The rustls example I showed previously fails on both Windows and MacOS with 403.

[djc]

All this isn't really that relevant to the rustls maintainers. I just think that until you have a test that differs across a small number of factors (I'm still not sure what curl sends on the wire as a HTTP/1.1 request) it's not obvious why your first suspicion is that this is because of rustls -- even if it is, we have an issue for this already. Going to close this now.

[darind]

Well, one example in which rustls differs from BringSSL on MacOS is that it doesn't send the Grease TLS extension, among other differences. All the differences can be seen quite well if you make a request here: https://tls.peet.ws/api/all

All those differences account for rustls not behaving as the native TLS clients on the corresponding OS and CDNs flagging requests from it as spam.

[cpu]

In general robust TLS fingerprint evasion is a cat and mouse game that, in my experience, requires significant investment. That investment is often at odds with the goals of a general purpose TLS library, which is why it's very common for censorship avoidance projects to fork a TLS library to customize the lowest levels of the handshake to avoid fingerprinting. E.g. see this fork of the Go TLS library. That project is already discussing Rustls.

[darind]

Yeah, speaking about go-lang, I also found this https://github.com/CUCyber/ja3transport. But I understand that this is not something that you wish to be dealing with in rustls and that is fine for me - as long as it is clearly stated.

I wanted to use rustls in a forward proxy server but that clearly is a no-go for me with this limitation.

Thanks for the clarifications.