v0.4.0

Req v0.4.0 changes headers to be maps, adds request & response streaming, and improves steps.

Change Headers to be Maps

Previously headers were lists of name/value tuples, e.g.:

[{"content-type", "text/html"}]

This is a standard across the ecosystem (with minor difference that some Erlang libraries use charlists instead of binaries.)

There are some problems with this particular choice though:

• We cannot use headers[name]
• We cannot use pattern matching

In short, this representation isn't very ergonomic to use.

Now headers are maps of string names and lists of values, e.g.:

%{"content-type" => ["text/html"]}

This allows headers[name] usage:

response.headers["content-type"]
#=> ["text/html"]

and pattern matching:

case Req.request!(req) do
  %{headers: %{"content-type" => ["application/json" <> _]}} ->
    # handle JSON response
end

This is a major breaking change. If you cannot easily update your app or your dependencies, do:

# config/config.exs
config :req, legacy_headers_as_lists: true

This legacy fallback will be removed on Req 1.0.

There are two other changes to headers in this release.

Header names are now case-insensitive in functions like Req.Response.get_header/2.

Trailer headers, or more precisely trailer fields or simply trailers, are now stored in a separate trailers field on the %Req.Response{} struct as long as you use Finch 0.17+.

Add Request Body Streaming

Req v0.4 adds official support for request body streaming by setting the request body to an enumerable. Here's an example:

iex> stream = Stream.duplicate("foo", 3)
iex> Req.post!("https://httpbin.org/post", body: stream).body["data"]
"foofoofoo"

The enumerable is passed through request steps and they may change it. For example, the compress_body step gzips the request body on the fly.

Add Response Body Streaming

Req v0.4 also adds response body streaming, via the :into option.

Here's an example where we download the first 20kb (by making a range request, via the put_range step) of Elixir release zip. We stream the response body into a function and can handle each body chunk. The function receives a {:data, data}, {req, resp} and returns a {:cont | :halt, {req, resp}} tuple.

resp =
  Req.get!(
    url: "https://github.com/elixir-lang/elixir/releases/download/v1.15.4/elixir-otp-26.zip",
    range: 0..20_000,
    into: fn {:data, data}, {req, resp} ->
      IO.inspect(byte_size(data), label: :chunk)
      {:cont, {req, resp}}
    end
  )

# output: 17:07:38.131 [debug] redirecting to https://objects.githubusercontent.com/github-production-release-asset-2e6(...)
# output: chunk: 16384
# output: chunk: 3617

resp.status #=> 206
resp.headers["content-range"] #=> ["bytes 0-20000/6801977"]
resp.body #=> ""

Notice we only stream response body, that is, Req automatically handles HTTP response status and headers. Once the stream is done, Req passes the response through response steps which allows following redirects, retrying on errors, etc. Response body is set to empty string "" which is then ignored by decompress_body, decode_body, and similar steps. If you need to decompress or decode incoming chunks, you need to do that in your custom into: fun function.

As the name :into implies, we can also stream response body into any [Collectable]. Here's a similar snippet to above where we stream to a file:

resp =
  Req.get!(
    url: "https://github.com/elixir-lang/elixir/releases/download/v1.15.4/elixir-otp-26.zip",
    range: 0..20_000,
    into: File.stream!("elixit-otp-26.zip.1")
  )

# output: 17:07:38.131 [debug] redirecting to (...)
resp.status #=> 206
resp.headers["content-range"] #=> ["bytes 0-20000/6801977"]
resp.body #=> %File.Stream{}

Full CHANGELOG

• Change request.headers and response.headers to be maps.

• Ensure request.headers and response.headers are downcased.

  Per RFC 9110: HTTP Semantics, HTTP headers should be case-insensitive. However, per RFC 9113: HTTP/2 headers must be sent downcased.

  Req headers are now stored internally downcased and all accessor functions like [Req.Response.get_header/2] are downcasing the given header name.

• Add trailers field to [Req.Response] struct. Trailer field is only filled in on Finch 0.17+.

• Make request.registered_options internal representation private.

• Make request.options internal representation private.

  Currently request.options field is a map but it may change in the future. One possible future change is using keywords lists internally which would allow, for example, Req.new(params: [a: 1]) |> Req.update(params: [b: 2]) to keep duplicate :params in request.options which would then allow to decide the duplicate key semantics on a per-step basis. And so, for example, put_params would merge params but most steps would simply use the first value.

  To have some room for manoeuvre in the future we should stop pattern matching on request.options. Calling request.options[key], put_in(request.options[key], value), and update_in(request.options[key], fun) is allowed.

• Fix typespecs for some functions

• Deprecate output step in favour of into: File.stream!(path).

• Rename follow_redirects step to redirect

• redirect: Rename :follow_redirects option to :redirect.

• redirect: Rename :location_trusted option to :redirect_trusted.

• redirect: Change HTTP request method to GET only on POST requests that result in 301..303.

  Previously we were changing the method to GET for all 3xx except 307 and 308.

• decompress_body: Remove support for deflate compression (which was broken)

• decompress_body: Don't crash on unknown codec

• decompress_body: Fix handling HEAD requests

• decompress_body: Re-calculate content-length header after decompresion

• decompress_body: Remove content-encoding header after decompression

• decode_body: Do not decode response with content-encoding header

• run_finch: Add :inet6 option

• retry: Support retry: :safe_transient which retries HTTP 408/429/500/502/503/504 or exceptions with reason field set to :timeout/:econnrefused.

  :safe_transient is the new default retry mode. (Previously we retried on 408/429/5xx and any exception.)

• retry: Support retry: :transient which is the same as :safe_transient except it retries on all HTTP methods

• retry: Use retry-after header value on HTTP 503 Service Unavailable. Previously only HTTP 429 Too Many Requests was using this header value.

• retry: Support retry: &fun/2. The function receives request, response_or_exception and returns either:

  • true - retry with the default delay

  • {:delay, milliseconds} - retry with the given delay

  • false/nil - don't retry

• retry: Deprecate retry: :safe in favour of retry: :safe_transient

• retry: Deprecate retry: :never in favour of retry: false

• Req.request/2: Improve error message on invalid arguments

• Req.update/2: Do not duplicate headers

• Req.update/2: Merge :params

• Req.Request: Fix displaying redacted basic authentication

• Req.Request: Add [Req.Request.get_option/3]

• Req.Request: Add [Req.Request.fetch_option/2]

• Req.Request: Add [Req.Request.fetch_option!/2]

• Req.Request: Add [Req.Request.delete_option/2]

• [Req.Response]: Add [Req.Response.delete_header/2]

• [Req.Response]: Add [Req.Response.update_private/4]