What am I doing wrong?

[Ethan-Arrowood]

Hi folks, I'm trying to use undici with the GitHub api but I'm hitting something strange and struggling to debug it effectively.

Is it something to do with https?

I'm running the following script with node 16:

'use strict'

import undici from 'undici'
import http from 'http'

undici.request(
  'https://api.github.com/zen',
  (error, { statusCode, headers, body }) => {
    if (error) {
      console.error(error)
      process.exit(1)
    }

    console.log(`Staus: ${statusCode}`)
    console.log(`Status Message: ${http.STATUS_CODES[statusCode]}`)
    console.log('Headers', headers)
    const bufs = []
    body.on('data', buf => { bufs.push(buf) })
    body.on('end', () => {
      console.log(Buffer.concat(bufs).toString('utf-8'))
    })
  }
)

This is the Hello World example for the github rest api

[Ethan-Arrowood]

Heres the same endpoint with curl:

❯ curl -v https://api.github.com/zen
*   Trying 140.82.113.5...
* TCP_NODELAY set
* Connected to api.github.com (140.82.113.5) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.com
*  start date: Mar 25 00:00:00 2021 GMT
*  expire date: Mar 30 23:59:59 2022 GMT
*  subjectAltName: host "api.github.com" matched cert's "*.github.com"
*  issuer: C=US; O=DigiCert, Inc.; CN=DigiCert High Assurance TLS Hybrid ECC SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f7f4e80f600)
> GET /zen HTTP/2
> Host: api.github.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< server: GitHub.com
< date: Thu, 29 Apr 2021 18:17:48 GMT
< content-type: text/plain;charset=utf-8
< access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'
< vary: Accept-Encoding, Accept, X-Requested-With
< x-ratelimit-limit: 60
< x-ratelimit-remaining: 54
< x-ratelimit-reset: 1619722045
< x-ratelimit-used: 6
< accept-ranges: bytes
< content-length: 37
< x-github-request-id: DA54:6ABC:1691546:4969760:608AF84C
< 
* Connection #0 to host api.github.com left intact
Non-blocking is better than blocking.* Closing connection 0

[ronag]

Looks like GitHub is denying the request. Do they not allow http/1.1? Do you have the same problem with the node core client?

[Ethan-Arrowood]

Forcing http 1.1 with curl works fine:

curl -v https://api.github.com/zen --http1.1 
*   Trying 140.82.113.5...
* TCP_NODELAY set
* Connected to api.github.com (140.82.113.5) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.com
*  start date: Mar 25 00:00:00 2021 GMT
*  expire date: Mar 30 23:59:59 2022 GMT
*  subjectAltName: host "api.github.com" matched cert's "*.github.com"
*  issuer: C=US; O=DigiCert, Inc.; CN=DigiCert High Assurance TLS Hybrid ECC SHA256 2020 CA1
*  SSL certificate verify ok.
> GET /zen HTTP/1.1
> Host: api.github.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: GitHub.com
< Date: Thu, 29 Apr 2021 18:28:42 GMT
< Content-Type: text/plain;charset=utf-8
< Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
< Access-Control-Allow-Origin: *
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< X-Frame-Options: deny
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 0
< Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
< Content-Security-Policy: default-src 'none'
< Vary: Accept-Encoding, Accept, X-Requested-With
< X-RateLimit-Limit: 60
< X-RateLimit-Remaining: 50
< X-RateLimit-Reset: 1619722045
< X-RateLimit-Used: 10
< Accept-Ranges: bytes
< Content-Length: 26
< X-GitHub-Request-Id: DAEC:062E:60EA3B8:A6C0333:608AFAEA
< 
* Connection #0 to host api.github.com left intact
Practicality beats purity.* Closing connection 0

[ronag]

Btw, here is the output of your script:

Status: 403
Headers {
  'cache-control': 'no-cache',
  'content-type': 'text/html; charset=utf-8',
  'strict-transport-security': 'max-age=31536000',
  'x-content-type-options': 'nosniff',
  'x-frame-options': 'deny',
  'x-xss-protection': '0',
  'content-security-policy': "default-src 'none'; style-src 'unsafe-inline'",
  connection: 'close'
}
Error SocketError: other side closed
    at TLSSocket.onSocketEnd (/Users/ronagy/GitHub/public/undici/lib/client.js:935:22)
    at TLSSocket.emit (node:events:377:35)
    at endReadableNT (node:internal/streams/readable:1312:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  code: 'UND_ERR_SOCKET'
}

[ronag]

I get the same problem with the node core client. I don't think this is strictly problem with undici. I recommend opening an issue on the Node core repo. If we find any problem there I'm sure we can port that over to undici.

[ronag]

If I read the error body I get the following:

Request forbidden by administrative rules. Please make sure your request has a User-Agent header (http://developer.github.com/v3/#user-agent-required). Check https://developer.github.com for other possible causes.

[ronag]

It looks to me like github doesn't provide a content-length nor do they use chunked encoding.

[ronag]

As far as I can see undici does the correct thing here. But I'm unsure whether we should fail if the server responds with neither content-length nor transfer-encoding chunked header?

[ronag]

@szmarczak @ShogunPanda wdyt think from a spec perspective?

[ronag]

#784

[ronag]

@indutny might also be appropriate to ping. Shouldn’t llhttp invoke onCompleteMessage here?

[Ethan-Arrowood]

Issue is not node or undici related. Its a GitHub api requirement I was missing (you have to specify a user-agent):

'use strict'

import undici from 'undici'
import http from 'http'

undici.request(
  'https://api.github.com/zen',
  {
    headers: [
      'user-agent', 'undici'
    ]
  },
  (error, { statusCode, headers, body }) => {
    if (error) {
      console.error(error)
      process.exit(1)
    }

    console.log(`Staus: ${statusCode}`)
    console.log(`Status Message: ${http.STATUS_CODES[statusCode]}`)
    console.log('Headers', headers)
    const bufs = []
    body.on('data', buf => { bufs.push(buf) })
    body.on('end', () => {
      console.log(Buffer.concat(bufs).toString('utf-8'))
    })
  }
)