"Failed to decode latest license data" when reaching out to replicated.app to update license

[collinbachi]

On multiple online installations, we have observed issues reaching out to replicated.app to update the license. I have not yet been able to consistently reproduce, but I want to start compiling evidence here and look for feedback on what could be causing this.

The error appears to correspond to this code path:

kots/pkg/license/license.go

Line 43 in a177cc1

return nil, errors.Wrap(err, "failed to decode latest license data")

This error implies that replicated.app is reachable and returns a status code <400.

To help debug, we asked our customer to run the following from within the kotsadm pod:

curl -ivv -H "Authorization: Basic [redacted]" https://replicated.app/license/[redacted]

The response returned looked correct:

< HTTP/2 200 
HTTP/2 200 
< date: Thu, 29 Oct 2020 21:22:49 GMT
date: Thu, 29 Oct 2020 21:22:49 GMT
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< content-length: 4963
content-length: 4963
< x-powered-by: Express
x-powered-by: Express
< access-control-allow-origin: *
access-control-allow-origin: *
< etag: W/"1363-oELXt3W6sXxcfcFH5rqxSDfGLnU"
etag: W/"1363-oELXt3W6sXxcfcFH5rqxSDfGLnU"

< 
apiVersion: kots.io/v1beta1
kind: License
metadata:
  name: [redacted]
spec:
  licenseID: [redacted]
  licenseType: trial
  customerName: [redacted]
  appSlug: [redacted]
  channelID: [redacted]
  channelName: Stable
  licenseSequence: 1
  endpoint: 'https://replicated.app'
  entitlements:
    expires_at:
      title: Expiration
      description: License Expiration
      value: '2020-12-01T23:19:09Z'
      valueType: String
    max_guests:
      title: Max Guests
      value: 100
      valueType: Integer
    max_members:
      title: Max Members
      value: 50
      valueType: Integer
    max_projects:
      title: Max Projects
      value: 10000
      valueType: Integer
    max_users:
      title: Max Users
      value: 10
      valueType: Integer
      isHidden: true
  isAirgapSupported: true
    isGitOpsSupported: true
  isSnapshotSupported: true
  signature: >-
    [redacted]
* Curl_http_done: called premature == 0
* Connection #0 to host replicated.app left intact