Quarkus k8s image push does not work with kubectl external authentication

[vishalgoel1988]

Describe the bug
Hi,

I am trying to deploy a quarkus application in kubernetes cluster (EKS) using quarkus kubernetes extension.

During mvnw package, I see below error in mvn output (though build is shown as success). I have "aws" in my path and even on same console, I am able to run aws command.

[INFO] --- quarkus-maven-plugin:1.4.2.Final:build (default) @ security-keycloak-authorization-quickstart ---
[WARNING] [io.quarkus.deployment.QuarkusAugmentor] Using Java versions older than 11 to build Quarkus applications is deprecated and will be disallowed in a future release!
[ERROR] Failed to parse the kubeconfig.
java.io.IOException: Cannot run program "aws": CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start (ProcessBuilder.java:1048)
at io.fabric8.kubernetes.client.Config.loadFromKubeconfig (Config.java:570)
at io.fabric8.kubernetes.client.Config.tryKubeConfig (Config.java:491)
at io.fabric8.kubernetes.client.Config.autoConfigure (Config.java:230)

In users section of .kube/config, I have below:

users:

• name: <eks_cluster_arn>
  user:
  exec:
  apiVersion: client.authentication.k8s.io/v1alpha1
  args:
  - --region
  - us-west-2
  - eks
  - get-token
  - --cluster-name
  - <cluster_name>
  command: aws
  env: null

Expected behavior
There should not be any build errors.

Actual behavior
There are exceptions in build.

To Reproduce
Steps to reproduce the behavior:

1. Create quarkus kubernetes project.
2. Try to do the build with .kube/config having external auth with aws

Configuration

# Add your application.properties here, if applicable.
quarkus.container-image.build=true

Screenshots
(If applicable, add screenshots to help explain your problem.)

Environment (please complete the following information):

• Output of uname -a or ver:
• Output of java -version:
• GraalVM version (if different from Java):
• Quarkus version or git rev:
• Build tool (ie. output of mvnw --version or gradlew --version):

Additional context
(Add any other context about the problem here.)
Discussion going on here:
https://groups.google.com/g/quarkus-dev/c/TchHYnLC-3I

[geoand]

@iocanel @manusa this looks like a bug in the Kubernetes, right?

[manusa]

It sure seems a problem in Kubernetes Client.

It seems the aws command cannot be found when starting the process to retrieve the OAuth token.

Is aws available in the $PATH? (just saw in issue description) If it's not and this is some additional context/user configuration, is that the current-context in your config?

https://groups.google.com/g/quarkus-dev/c/TchHYnLC-3I/m/VvpdaWe_AQAJ:

I had aws as command in my .kube/config which mvnw was picking during build. Strangely, it could not resolve aws (though it is there in PATH), so I provided full path in .kube/config and it worked out correctly.
Quarkus is pushing images to ECR and even deploying it on k8s cluster.

Does this mean that if we fix the env, your use-case would work (i.e. instead of running aws going for sh -c aws)?

See also: client-go-credential-plugins

[vishalgoel1988]

My kubectl cli works perfectly with just "aws" in kube config. Just a note that I am running all this on windows.
If I put my full aws.cmd path in my kube config, quarkus works perfectly.

[manusa]

My kubectl cli works perfectly with just "aws" in kube config. Just a note that I am running all this on windows.
If I put my full aws.cmd path in my kube config, quarkus works perfectly.

👍 So it's definitely a problem with ProcessBuilder not using your env PATH.
This could probably get fixed by wrapping the command in the ProcessBuilder with sh -c aws (*nix) or cmd /C start aws (windows).

[vishalgoel1988]

Hi @manusa, would this get fixed?

[manusa]

Hi @manusa, would this get fixed?

Hi @vishalgoel1988 we've added fabric8io/kubernetes-client#2308 to our current Sprint, and we'll try to take care of it soon. It would also be good if we could count on you to verify our fix works (once we have it), since our team doesn't have access to an environment with the same conditions as yours.

[vishalgoel1988]

Thanks @manusa.
Yes, I can do the testing for this.

[manusa]

Hi @vishalgoel1988

Your issue was tackled in scope of fabric8io/kubernetes-client#2308 and fix implementation was included in our latest release: https://github.com/fabric8io/kubernetes-client/releases/tag/v4.11.0

Please, check if everything works as expected now.

[vishalgoel1988]

Thanks. I will give it a try.

[geoand]

I'll close this as it seems to have been fixed. If that is not the case, feel free to comment or reopen