Request made with reqwest blocked by Cloudflare's protection but succeeds with curl

[filips123]

A request made with reqwest to https://www.crunchyroll.com/manifest.json is blocked by Cloudflare's protection/challenge. However, a request to the same URL with the same headers using curl works.

This is a minimal reproducible example for this issue:

use anyhow::Result;
use reqwest::blocking::Client;
use reqwest::header::{HeaderMap, HeaderValue};

pub fn create_client() -> reqwest::Result<Client> {
    let mut headers = HeaderMap::new();
    headers.insert("Sec-Fetch-Site", HeaderValue::from_static("none"));
    headers.insert("Sec-Fetch-Dest", HeaderValue::from_static("manifest"));

    let builder = Client::builder()
        .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0")
        .default_headers(headers);

    builder.build()
}

fn main() -> Result<()> {
    let url = "https://www.crunchyroll.com/manifest.json";

    let response = create_client()?
        .get(url.to_owned())
        .header(reqwest::header::REFERER, url.to_owned())
        .send()?
        .text()?;

    println!("{}", response);

    Ok(())
}

The client is configured with some additional headers, but removing them does not fix the issue. This example has been tested with these dependencies:

[dependencies]
anyhow = "1.0.81"
reqwest = { version = "0.12.2", features = ["blocking", "socks", "gzip", "brotli", "deflate", "native-tls"] }

Note that the exact versions do not seem to matter, as this issue also happens in my actual project with older dependencies (and reqwest 0.11.23).

This code should download the URL and print it. When tested, it downloads an HTML page which seems to be Cloudflare's challenge, instead of the actual JSON manifest.

However, when I try to download the URL with the same headers using curl from the command line, it works:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0" -H "Sec-Fetch-Site: none" -H "Sec-Fetch-Dest: manifest" --referer "https://www.crunchyroll.com/manifest.json" "https://www.crunchyroll.com/manifest.json"

{
  "background_color": "#23252b",
  "display": "minimal-ui",
  "name": "Crunchyroll - Watch Popular Anime",
  "orientation": "any",
  "scope": "/",
  "short_name": "Crunchyroll",
  "start_url": "/?utm_medium=pwa_app_launch&utm_source=pwa",
  "theme_color": "#f47521",
  "description": "Stream the world’s largest anime library. Watch over 1,000 titles—from past seasons to new episodes fresh from Japan",
  "categories": ["entertainment", "video"],
  "icons": [
    {
      "src": "https://crunchyroll.com/build/assets/img/pwa/512.png",
      "sizes": "512x512",
      "type": "image/png",
      "purpose": "any"
    },
    {
      "src": "https://crunchyroll.com/build/assets/img/pwa/512-maskable.png",
      "sizes": "512x512",
      "type": "image/png",
      "purpose": "maskable"
    }
  ],
  "scope_extensions": [
    {
      "origin": "www.crunchyroll.com"
    },
    {
      "origin": "sso.crunchyroll.com"
    }
  ],
  "shortcuts": [
    {
      "name": "View Watchlist",
      "short_name": "Watchlist",
      "url": "/watchlist?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "Pick up where you left off with your favorite anime series and movies",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/watchlist-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "Browse Anime",
      "short_name": "Browse",
      "url": "/videos/popular?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "See all anime Crunchyroll has to offer by popularity, freshness, and genre",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/browse-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "Search",
      "short_name": "Search",
      "url": "/search?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "Search for anime on Crunchyroll",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/search-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "View Crunchylists",
      "short_name": "Crunchylists",
      "url": "/crunchylists?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "Create and manage your custom Crunchylists",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/crunchylists-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "View History",
      "short_name": "History",
      "url": "/history?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "See your most recently watched videos on Crunchyroll",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/history-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    }
  ],
  "id": "com.crunchyroll.pwa",
  "dir": "auto"
}

I tested both ways multiple times (from the same computer), and the results were always the same. This issue also happens to some users of my project (filips123/PWAsForFirefox#482). It also seems to be specific to Crunchyroll, as it works fine with other Cloudflare-hosted websites.

Does reqwest send some different hidden headers than curl that make Cloudflare block the request? Or does it maybe do some advanced detection that can differentiate between curl and reqwest? So, is it possible to somehow configure reqwest client so it is not detected and blocked by Cloudflare?