Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Receiving chart pull error on environment with a proxy - EOF #1485

Open
Valgueiro opened this issue May 13, 2024 · 9 comments · May be fixed by #1490
Open

Receiving chart pull error on environment with a proxy - EOF #1485

Valgueiro opened this issue May 13, 2024 · 9 comments · May be fixed by #1490
Assignees
@souleb
Labels
area/helm Helm related issues and pull requests bug Something isn't working

Comments

@Valgueiro
Copy link

Valgueiro commented May 13, 2024 •
edited

Environment

I have my k8s cluster deployed behind a firewall, that only allows connections from a proxy that is on the same network.

image

Setup

Flux version: v2.1.2
Source controller version: 1.1.2
I've setup the gotk as such to be able to use the proxy to fetch things.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
patches:
  - patch: |
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: all
      spec:
        template:
          spec:
            containers:
              - name: manager
                env:
                  - name: "HTTPS_PROXY"
                    value: "http://proxy.com:3128"
                  - name: "NO_PROXY"
                    value: ".cluster.local.,.cluster.local,cluster.local,.svc,127.0.0.0/8,10.0.0.0/8"  
                  - name: "https_proxy"
                    value: "http://proxy.com:3128"
                  - name: "no_proxy"
                    value: ".cluster.local.,.cluster.local,cluster.local,.svc,127.0.0.0/8,10.0.0.0/8"     
    target:
      kind: Deployment
      labelSelector: app.kubernetes.io/part-of=flux

And I have HelmRelease and helmrepo configured like this:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: keda
  namespace: keda
spec:
  interval: 5m0s
  releaseName: keda
  install:
    createNamespace: true
  chart:
    spec:
      chart: keda
      version: '2.12.1'
      sourceRef:
        kind: HelmRepository
        name: charts
        namespace: keda
  valuesFrom:
  - kind: ConfigMap
    name: keda-values
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: charts
  namespace: keda
spec:
  type: "oci"
  interval: 5m0s
  url: oci://<acr>/sre/charts/
  secretRef:
    name: registry-pull-secret
  certSecretRef:
    name: tls-ca

My HelmRepo is showing as active, but the HelmChart is showing as "Reconciling" and I can see the following error:

chart pull error: failed to download chart for remote reference: failed to get 'oci://<acr>/sre/charts/keda:2.12.1': failed to do request: Head "https://<acr>/v2/sre/charts/keda/manifests/2.12.1": EOF

I thought that this could be related to this issue about http_proxy on busybox images: mirror/busybox#21 , and after that I tried with this docker image as source-controller:

FROM <acr>/sre/fluxcd/source-controller:v1.1.2
USER root

COPY zscaler.crt /etc/ssl/certs/
RUN update-ca-certificates

RUN apk --no-cache -U add openssl wget ca-certificates
# wget https://httpbin.org/get

USER 65534:65534

But I continued to receive the same error.

Do you guys have any idea of what I can do to fix this?
@Valgueiro
Copy link
Author

Valgueiro commented May 13, 2024 •
edited

Other things that can be useful here:

  1. The same setup works when I remove the firewall and proxy from the architecture.
  2. This is the output of the command when I try to do a HEAD request from the source-controller container
~ $ wget --spider https://<acr>/v2/sre/rancher-alerting-drivers/manifests/102.1.0
Spider mode enabled. Check if remote file exists.
--2024-05-13 22:07:41--  https://<acr>/v2/sre/rancher-alerting-drivers/manifests/102.1.0
Resolving proxy.com ( proxy.com)... <proxy-ip>
Connecting to proxy.com (proxy.com)|<proxy-ip>|:3128... connected.
Proxy request sent, awaiting response... 401 Unauthorized
  1. I tried to debug the code myself but I couldn't get much further. From what I could understand the error is popped from here:

    source-controller/internal/helm/chart/builder_remote.go

    Line 161 in f8eea53

    res, err := remote.DownloadChart(cv)

    more specifically when calling Client.get

    source-controller/internal/helm/repository/chart_repository.go

    Line 281 in f8eea53

    return r.Client.Get(resolvedUrl, clientOpts...)

@Valgueiro Valgueiro changed the title Receiving chart pull error on environment with a proxy configuration. Receiving chart pull error on environment with a proxy - EOF May 13, 2024
@Valgueiro
Copy link
Author

Just confirmed here with tcpdump that source-controller is sending requests directly to the OCI URL without using proxy. This should not be happening since the proxy is setup on the flux services like the doc suggests

@stefanprodan
Copy link
Member

Can you please try with an OCIRepository and see if that works, example here https://fluxcd.io/blog/2024/05/flux-v2.3.0/#enhanced-helm-oci-support

@souleb
Copy link
Member

souleb commented May 16, 2024 •
edited

This is fixed in helm/helm@94c1dea I believe. Upgrading helm to v3.15.0 in source-controller should resolve this

@stefanprodan
Copy link
Member

Switching to OCIRepo and HelmRelease v2 should work as we don’t use the Helm getter in OCIRepo.

@Valgueiro
Copy link
Author

I tried to just update to the latest flux version which uses a version of helm that was already fixed ( 1.3.0 source controller points to 3.14.4) but still maintaining the HelmRepository and I did not have success. I will give the OCIRepo a try.

@souleb
Copy link
Member

souleb commented May 16, 2024

As I wrote above, it is fixed in helm v3.15.0. We have not updated Flux to that version yet. I would try Stefan suggestion on Flux v2.3.0.

@Valgueiro
Copy link
Author

Valgueiro commented May 16, 2024 •
edited

As I wrote above, the fix is already on flux version 2.3.0. Even the guy who made the fix himself bumped another repository to 3.14.4 to fix his issue. As you can see on the link to the code on 3.14.4, it is already there! Which means that 2.3.0 already have this fix.

image

So, bumping the version of helm 3.15 in the future must not solve the issue that I am facing.

@souleb souleb self-assigned this May 16, 2024
@souleb souleb added bug Something isn't working area/helm Helm related issues and pull requests labels May 16, 2024
@souleb
Copy link
Member

souleb commented May 16, 2024 •
edited

Thanks @Valgueiro, indeed we instantiate our own http.Transport. This will be fixed in the next Flux minor.

@souleb souleb linked a pull request May 17, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@souleb souleb

Labels
area/helm Helm related issues and pull requests bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use the transport pool fluxcd/source-controller
3 participants
@stefanprodan @souleb @Valgueiro