support following 302 redirects #60

ruckc · 2021-01-22T02:40:40Z

I'm using data "http" "someyaml" {} to fetch official crds/yamls for deploying to kubernetes. Some of these yamls send a 302 redirect that the http client is not following. It would be nice to expose a follow redirects argument to provide this control.

toabi · 2021-07-27T15:31:04Z

I'm using http provider v2.1.0 with terraform v0.15.3 and it does follow a redirect. I tested it with an output.

data "http" "hcloud_ccm_manifest" {
  url = "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.11.1/ccm-networks.yaml"
}

output "cmm_manifest" {
  value = data.http.hcloud_ccm_manifest.body
}

GitHub releases respond with HTTP 302 to some other location. The output contains the correct YAML, but it also prints a warning Warning: Content-Type is not recognized as a text type, got "application/octet-stream" which is a bit annoying. It would be nice to have the option to ignore this warning.

Edit: found an actually more fitting issue here: #72 - sorry

louhnmn · 2023-05-04T10:39:24Z

I have a similiar problem with a google sheet integration:

for example this test sheet: gsheet

variable "gsheet" {
  type = object({
    base_url = string
    pub_id   = string
    gid      = string
  })
  default = {
    base_url = "https://docs.google.com/spreadsheets/d/e/"
    pub_id   = "2PACX-1vRzZjUzim7pcO9Q3xBOCf9_GmMEANBLVeGpPbUOX-GzfQLU6bIgEkOQsXOuuM-M9ephWi3TSSO9vpWH"
    gid      = "0"
  }
}


data "http" "get_sheet" {
  url    = format("%s%s/pub?gid=%s&output=csv", var.gsheet.base_url, var.gsheet.pub_id, var.gsheet.gid)
  method = "GET"

  request_headers = {
    Accept = "text/csv"
  }
}

output "sheet_response_body" {
  value = data.http.get_sheet.response_body
}

then after a terraform plan in response body there will be no csv data, but something like this:

data.http.get_sheet: Reading...
data.http.get_sheet: Read complete after 1s [id=https://docs.google.com/spreadsheets/d/e/2PACX-1vRzZjUzim7pcO9Q3xBOCf9_GmMEANBLVeGpPbUOX-GzfQLU6bIgEkOQsXOuuM-M9ephWi3TSSO9vpWH/pub?gid=0&output=csv]

Changes to Outputs:
  + sheet_response_body = <<-EOT
        <HTML>
        <HEAD>
        <TITLE>Temporary Redirect</TITLE>
        </HEAD>
        <BODY BGCOLOR="#FFFFFF" TEXT="#000000">
        <H1>Temporary Redirect</H1>
        The document has moved <A HREF="https://doc-0s-88-sheets.googleusercontent.com/pub/n4743ueaoiib6is67cgugk097k/35pr69hukspeo9k26g4nlvkarg/1683195800000/105210521312657442095/*/e@2PACX-1vRzZjUzim7pcO9Q3xBOCf9_GmMEANBLVeGpPbUOX-GzfQLU6bIgEkOQsXOuuM-M9ephWi3TSSO9vpWH?gid=0&amp;output=csv">here</A>.
        </BODY>
        </HTML>
    EOT

If you do a curl -L on the initial url, it will grab the redirect url and output the csv data:

colA,colB
123,456
789,12

If you skip the -L Location option curl will also "fail" and give the same message as the http-provider.

So the http provider is not doing this redirect.
Altough it was working two weeks ago but then stopped working this date 2023-04-25 13:04:01.44.
I think google changed something in the way of redirect here.

Here a more detailed view of the curl with curl -Li:

A curl -Li will output:

HTTP/2 307 
content-type: text/html; charset=UTF-8
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Mon, 01 Jan 1990 00:00:00 GMT
date: Thu, 04 May 2023 10:33:15 GMT
location: https://doc-0s-88-sheets.googleusercontent.com/pub/n4743ueaoiib6is67cgugk097k/8ae3sg2p0gv6016ai6lm64n7p4/1683196395000/105210521312657442095/*/e@2PACX-1vRzZjUzim7pcO9Q3xBOCf9_GmMEANBLVeGpPbUOX-GzfQLU6bIgEkOQsXOuuM-M9ephWi3TSSO9vpWH?gid=0&single=true&output=csv
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
server: GSE
set-cookie: NID=511=llanR2tu9k6ac9<...redacted...>_QuFpih9Z5Om4; expires=Fri, 03-Nov-2023 10:33:14 GMT; path=/; domain=.google.com; HttpOnly
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
accept-ranges: none
vary: Accept-Encoding

HTTP/2 200 
content-type: text/csv
x-frame-options: ALLOW-FROM https://docs.google.com
x-robots-tag: noindex, nofollow, nosnippet
expires: Thu, 04 May 2023 10:33:18 GMT
date: Thu, 04 May 2023 10:33:18 GMT
cache-control: private, max-age=300
content-disposition: attachment; filename="test_table-data1.csv"; filename*=UTF-8''test_table%20-%20data1.csv
access-control-allow-origin: *
access-control-expose-headers: Cache-Control,Content-Disposition,Content-Encoding,Content-Length,Content-Type,Date,Expires,Server,Transfer-Encoding
content-security-policy: frame-ancestors 'self' https://docs.google.com
content-security-policy: base-uri 'self';object-src 'self';report-uri https://doc-0s-88-sheets.googleusercontent.com/spreadsheets/cspreport;script-src 'nonce-e1_RvbtZa_0Z_GdQwIpG9g' 'unsafe-inline' 'strict-dynamic' https: http: 'unsafe-eval';worker-src 'self' blob:
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
server: GSE
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
accept-ranges: none
vary: Accept-Encoding

colA,colB
123,456
789,12

So could it be that http-provider only supports 302 redirects (as in the example from @toabi ) but not 307 ones?

azend · 2023-06-30T18:51:35Z

I have this issue as well.

louhnmn · 2023-07-24T07:37:56Z

I found a workaround for 307 that should basically work for any redirect that gives you the redirect url in the output response.
see #264

relvacode mentioned this issue Aug 8, 2022

Do not follow HTTP redirects #170

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support following 302 redirects #60

support following 302 redirects #60

ruckc commented Jan 22, 2021

toabi commented Jul 27, 2021 •

edited

louhnmn commented May 4, 2023 •

edited

azend commented Jun 30, 2023

louhnmn commented Jul 24, 2023

support following 302 redirects #60

support following 302 redirects #60

Comments

ruckc commented Jan 22, 2021

toabi commented Jul 27, 2021 • edited

louhnmn commented May 4, 2023 • edited

azend commented Jun 30, 2023

louhnmn commented Jul 24, 2023

toabi commented Jul 27, 2021 •

edited

louhnmn commented May 4, 2023 •

edited