Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"GET https://example.com/users/sign_in: unexpected status code 200 OK" when pushing to self-hosted GitLab with Kaniko #3146

Open
andrew2758 opened this issue May 1, 2024 · 5 comments
Open

"GET https://example.com/users/sign_in: unexpected status code 200 OK" when pushing to self-hosted GitLab with Kaniko #3146

andrew2758 opened this issue May 1, 2024 · 5 comments

Comments

@andrew2758
Copy link

Actual behavior

When attempting to publish a Kaniko-built image to a self-hosted GitLab server (which I'll call example.com in this issue), we get the following error:

$ /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "registry-example.com:443/me/registry-test:v0.0.14": GET https://example.com/users/sign_in: unexpected status code 200 OK: <!DOCTYPE html>
<HTML of the of the user login page>
...

The build stage is successful if we provide --no-push. It looks like what might be happening here is that as Kaniko tries logging into the registry at registry-example.com/v2/, and it at some point decides to visit example.com. That automatically redirects it to the user login page, which doesn't contain the expected data. (Our GitLab instance requires a login to view the / path on that site, but other public groups and repos are visible if you know their URLs).

I looked at the output of the env command in our runner, and that shows some entries that contain 'example.com', but it's not clear whether any of those are influencing Kaniko's behavior.

Is Kaniko looking for some data on the main page of our GitLab instance?

Expected behavior

The expected behavior is for Kaniko to log into the container registry, so it can push an image to it.

To Reproduce
Steps to reproduce the behavior:

  1. Create a self-hosted GitLab server installed via Omnibus.
  2. Block anonymous viewing of the base path of the GitLab application.
  3. Configure a separate domain to serve the registry API.
  4. Use the following CI scripts in a new repo:

Additional Information

gcr.io/kaniko-project/executor:v1.22.0-debug (likely c7c1f8d3d464)

Dockerfile:

FROM ubuntu:mantic
RUN DEBIAN_FRONTEND=noninteractive apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y git yamllint

CMD /bin/bash

.gitlab-ci.yml:

---
build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.22.0-debug
    entrypoint: ["/bin/sh"]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > /kaniko/.docker/config.json
    - cat /kaniko/.docker/config.json
    - echo "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
    - env
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
  rules:
    - if: $CI_COMMIT_TAG

Triage Notes for the Maintainers

Description Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile
@andrew2758
Copy link
Author

Here are the env lines that match 'example.com':

CI_DEPENDENCY_PROXY_SERVER=example.com:443
CI_SERVER_HOST=example.com
CI_SERVER_URL=https://example.com
CI_COMPONENT_FQDN=example.com
CI_REGISTRY_IMAGE=registry-example.com:443/me/registry-test
CI_PIPELINE_URL=https://example.com/me/registry-test/-/pipelines/3612
CI_SERVER_FQDN=example.com
CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@example.com/me/registry-test.git
CI_API_GRAPHQL_URL=https://example.com/api/graphql
CI_REGISTRY=registry-example.com:443
CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX=example.com:443/me/dependency_proxy/containers
CI_API_V4_URL=https://example.com/api/v4
CI_SERVER_SHELL_SSH_HOST=example.com
CI_JOB_URL=https://example.com/me/registry-test/-/jobs/7576
CI_PROJECT_URL=https://example.com/me/registry-test
CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX=example.com:443/me/dependency_proxy/containers

@andrew2758
Copy link
Author

I'm trying to compile a modified version of Kaniko for debugging purposes:

DOCKER_BUILDKIT=1 docker image build . -f deploy/Dockerfile --tag=kaniko

How can I build a -debug version / flavor so it includes /bin/sh, which is required by GitLab's CI runner?

@andrew2758
Copy link
Author

I'm trying to compile a modified version of Kaniko for debugging purposes:

DOCKER_BUILDKIT=1 docker image build . -f deploy/Dockerfile --tag=kaniko

How can I build a -debug version / flavor so it includes /bin/sh, which is required by GitLab's CI runner?

I was able to add busybox with an extra Dockerfile:

FROM busybox:musl AS busybox

FROM kaniko:latest AS kaniko

ENV PATH="/bin:/kaniko"

COPY --from=busybox /bin /bin

Which allowed me to run my locally compiled version with GitLab. I'm still not sure why Kaniko is failing to log in though.

@andrew2758
Copy link
Author

I added and used the following debug patch to Kaniko:

diff --git a/pkg/executor/push.go b/pkg/executor/push.go
index c95aecc3..393d345d 100644
--- a/pkg/executor/push.go
+++ b/pkg/executor/push.go
@@ -91,6 +91,7 @@ var (
 // push to every specified destination.
 func CheckPushPermissions(opts *config.KanikoOptions) error {
        targets := opts.Destinations
+       fmt.Printf("destinations: %s\n", opts.Destinations)
        // When no push and no push cache are set, we don't need to check permissions
        if opts.SkipPushPermissionCheck {
                targets = []string{}

Which prints the following, when adding a container name to the command, which should be necessary because we need more levels in the URL path:

$ /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}/foo:${CI_COMMIT_TAG}"
destinations: [registry-example.com:443/leaf-node/registry-test/foo:v0.0.19]
...
(same error seen originally)

The destination URL is correct. I get the same error with and without a the following in .gitlab-ci.yml:

    - mkdir -p /kaniko/.docker/
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > /kaniko/.docker/config.json

@andrew2758
Copy link
Author

I found a workaround for this issue: explicitly enter the registry URL so it's possible to leave off the port number :443.

      # working:
      --destination "registry-example.com/me/registry-test/my-container:${CI_COMMIT_TAG}"
      # not working:
      --destination "registry-example.com:443/me/registry-test/my-container:${CI_COMMIT_TAG}"

It may also be possible to configure GitLab to not include the port in the ${CI_REGISTRY_IMAGE} variable:

/etc/gitlab/gitlab.rb:

## comment out this line:
#gitlab_rails['registry_port'] = "443"

I haven't tested that yet, because our dev instance is currently facing another issue.

In any case, you'll still likely need to add something like /my-container to the image URL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@andrew2758