Application fails to start on local when Kubernetes config is present in .kube directory #434

sujeet100 · 2022-02-22T12:44:00Z

Expected Behavior

Hi people,
Recently we upgraded to the latest version of Micronuat Kubernetes and we are facing following error on startup in local environment
Note: For local environment we do not use kubernetes, however there is a config file in ~/.kube directory which causes micronaut to think that it's running in a Kubernetes environment.
Is there a way to disable this for local environment?

Actual Behaviour

The application fails to start with an exception

2:51:04.286 [main] ERROR io.micronaut.runtime.Micronaut -
                        Error starting Micronaut server: Error instantiating bean of type  [io.kubernetes.client.openapi.ApiClient]

Message: Could not refresh token
Path Taken: new ResourceEventHandlerBeanListener([SharedIndexInformerFactory sharedIndexInformerFactory],InformerApiGroupResolver apiGroupResolver,InformerResourcePluralResolver resourcePluralResolver,InformerNamespaceResolver namespaceResolver,InformerLabelSelectorResolver labelSelectorResolver) --> new DefaultSharedIndexInformerFactory(InformerConfiguration informerConfiguration,[ApiClient apiClient]) --> ApiClient.apiClient([ClientBuilder clientBuilder],ExecutorService executorService)
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type  [io.kubernetes.client.openapi.ApiClient]

Message: Could not refresh token
Path Taken: new ResourceEventHandlerBeanListener([SharedIndexInformerFactory sharedIndexInformerFactory],InformerApiGroupResolver apiGroupResolver,InformerResourcePluralResolver resourcePluralResolver,InformerNamespaceResolver namespaceResolver,InformerLabelSelectorResolver labelSelectorResolver) --> new DefaultSharedIndexInformerFactory(InformerConfiguration informerConfiguration,[ApiClient apiClient]) --> ApiClient.apiClient([ClientBuilder clientBuilder],ExecutorService executorService)
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2363)
	at io.micronaut.context.DefaultBeanContext.createAndRegisterSingletonInternal(DefaultBeanContext.java:3281)
	at io.micronaut.context.DefaultBeanContext.createAndRegisterSingleton(DefaultBeanContext.java:3267)
	at io.micronaut.context.DefaultBeanContext.getBeanForDefinition(DefaultBeanContext.java:2820)
	at io.micronaut.context.DefaultBeanContext.getBeanInternal(DefaultBeanContext.java:2782)
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1638)
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:1933)
	at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1189)
	at io.micronaut.kubernetes.client.$ApiClientFactory$ApiClient1$Definition.build(Unknown Source)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2336)
	at io.micronaut.context.DefaultBeanContext.createAndRegisterSingletonInternal(DefaultBeanContext.java:3281)
	at io.micronaut.context.DefaultBeanContext.createAndRegisterSingleton(DefaultBeanContext.java:3267)
	at io.micronaut.context.DefaultBeanContext.getBeanForDefinition(DefaultBeanContext.java:2820)
	at io.micronaut.context.DefaultBeanContext.getBeanInternal(DefaultBeanContext.java:2782)
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1638)
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:1933)
	at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1189)
	at io.micronaut.kubernetes.client.informer.$DefaultSharedIndexInformerFactory$Definition.build(Unknown Source)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2336)
	at io.micronaut.context.DefaultBeanContext.createAndRegisterSingletonInternal(DefaultBeanContext.java:3281)
	at io.micronaut.context.DefaultBeanContext.createAndRegisterSingleton(DefaultBeanContext.java:3267)
	at io.micronaut.context.DefaultBeanContext.getBeanForDefinition(DefaultBeanContext.java:2820)
	at io.micronaut.context.DefaultBeanContext.getBeanInternal(DefaultBeanContext.java:2782)
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1638)
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:1933)
	at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1189)
	at io.micronaut.kubernetes.client.informer.$ResourceEventHandlerBeanListener$Definition.build(Unknown Source)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2336)
	at io.micronaut.context.DefaultBeanContext.createAndRegisterSingletonInternal(DefaultBeanContext.java:3281)
	at io.micronaut.context.DefaultBeanContext.createAndRegisterSingleton(DefaultBeanContext.java:3267)
	at io.micronaut.context.DefaultBeanContext.initializeEventListeners(DefaultBeanContext.java:1838)
	at io.micronaut.context.DefaultBeanContext.readAllBeanDefinitionClasses(DefaultBeanContext.java:3452)
	at io.micronaut.context.DefaultBeanContext.finalizeConfiguration(DefaultBeanContext.java:3883)
	at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:329)
	at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:183)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:72)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:313)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:299)
	at com.foo.Application.main(Application.java:34)
Caused by: java.lang.RuntimeException: Could not refresh token
	at io.kubernetes.client.util.authenticators.OpenIDConnectAuthenticator.refreshOidcToken(OpenIDConnectAuthenticator.java:243)
	at io.kubernetes.client.util.authenticators.OpenIDConnectAuthenticator.refresh(OpenIDConnectAuthenticator.java:165)
	at io.kubernetes.client.util.KubeConfig.getAccessToken(KubeConfig.java:215)
	at io.kubernetes.client.util.credentials.KubeconfigAuthentication.<init>(KubeconfigAuthentication.java:57)
	at io.kubernetes.client.util.ClientBuilder.kubeconfig(ClientBuilder.java:297)
	at io.kubernetes.client.util.ClientBuilder.getClientBuilder(ClientBuilder.java:129)
	at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:108)
	at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:100)
	at io.micronaut.kubernetes.client.ApiClientFactory.clientBuilder(ApiClientFactory.java:84)
	at io.micronaut.kubernetes.client.$ApiClientFactory$ClientBuilder0$Definition.build(Unknown Source)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2336)
	... 38 common frames omitted
Caused by: java.lang.RuntimeException: Invalid response code for token retrieval - 401
	at io.kubernetes.client.util.authenticators.OpenIDConnectAuthenticator.refreshOidcToken(OpenIDConnectAuthenticator.java:234)
	... 48 common frames omitted

If the config file is removed from the ~/.kube directory, then the application starts properly.

Steps To Reproduce

Add micronaut kubernetes dependency implementation("io.micronaut.kubernetes:micronaut-kubernetes-discovery-client")
Put a kubernetes config file with name config in ~/.kube directory
Start the application

Environment Information

OS: mac
JDK: OpenJDK 17

Example Application

No response

Version

3.3.3

The text was updated successfully, but these errors were encountered:

pgressa · 2022-02-23T16:16:09Z

@sujeet100 you can disable the environment deduction: https://docs.micronaut.io/latest/guide/#environments

Automatic detection of environments can be disabled by setting the micronaut.env.deduction system property or the MICRONAUT_ENV_DEDUCTION environment variable to false. This prevents Micronaut from detecting current environments, while still using any environments that are specifically provided as shown above.

pgressa · 2022-02-23T16:16:19Z

Does that address your issue?

sujeet100 · 2022-02-24T11:16:49Z

setting MICRONAUT_ENV_DEDUCTION = false did not fix the issue, still seeing the same exception.
The environment deduced is local only and does not include k8s.

Established active environments: [local]

alvarosanchez · 2022-02-24T12:00:58Z

I think the problem is triggered by https://github.com/micronaut-projects/micronaut-kubernetes/blob/master/kubernetes-client/src/main/java/io/micronaut/kubernetes/client/ApiClientFactory.java#L84

As per their Javadoc:

Creates a builder which is pre-configured in the following way

If $KUBECONFIG is defined, use that config file.

If $HOME/.kube/config can be found, use that.

If the in-cluster service account can be found, assume in cluster config.

Default to localhost:8080 as a last resort.

sujeet100 · 2022-03-29T10:36:06Z

@pgressa any update on this issue?

alvarosanchez · 2022-03-29T10:46:17Z

So one thing that could be done is to add an opt-out property, something like kubernetes.client.enabled, which would default to true but that you could set to false when running locally.

sujeet100 · 2022-04-25T06:24:51Z

@alvarosanchez can we annotate clientBuilder singleton

micronaut-kubernetes/kubernetes-client/src/main/java/io/micronaut/kubernetes/client/ApiClientFactory.java

Line 65 in 56c0ad7

    
           public ClientBuilder clientBuilder(ApiClientConfiguration apiClientConfiguration) throws IOException {

with @requires(env = Environment.KUBERNETES)

alvarosanchez · 2022-04-25T07:32:14Z

The annotation should go to the factory class instead. I would do the job, yes, but feels to me like a breaking change.

matt-snider · 2022-10-07T09:01:18Z

I ran into this issue in a GitLab CI Pipeline test stage. Locally gradle test works but in the CI pipeline it seems to detect the Kubernetes environment and this bean gets loaded.

Disabling MICRONAUT_ENV_DEDUCTION did not help.

I took a look at the dependency path and saw that the ApiClient bean is being required by DefaultSharedIndexInformerFactory which has a @Requires annotation on the property kubernetes.client.informer.enabled.

Once I disabled kubernetes.client.informer.enabled, the pipeline stage succeeded.

rdamus · 2023-03-27T16:34:17Z

was having the same problem with minikube. could only run tests when minikube was active on the machine.

in my application-test.yml the following allows my tests to complete without having minikube started

kubernetes:
  client:
    informer:
      enabled: false

pgressa self-assigned this Feb 23, 2022

pgressa added type: bug Something isn't working and removed type: bug Something isn't working labels Feb 23, 2022

pgressa added the status: awaiting feedback label Feb 23, 2022

alvarosanchez assigned alvarosanchez and unassigned pgressa Mar 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Application fails to start on local when Kubernetes config is present in .kube directory #434

Application fails to start on local when Kubernetes config is present in .kube directory #434

sujeet100 commented Feb 22, 2022 •

edited

pgressa commented Feb 23, 2022

pgressa commented Feb 23, 2022

sujeet100 commented Feb 24, 2022

alvarosanchez commented Feb 24, 2022

sujeet100 commented Mar 29, 2022 •

edited

alvarosanchez commented Mar 29, 2022

sujeet100 commented Apr 25, 2022

alvarosanchez commented Apr 25, 2022

matt-snider commented Oct 7, 2022

rdamus commented Mar 27, 2023

Application fails to start on local when Kubernetes config is present in .kube directory #434

Application fails to start on local when Kubernetes config is present in .kube directory #434

Comments

sujeet100 commented Feb 22, 2022 • edited

Expected Behavior

Actual Behaviour

Steps To Reproduce

Environment Information

Example Application

Version

pgressa commented Feb 23, 2022

pgressa commented Feb 23, 2022

sujeet100 commented Feb 24, 2022

alvarosanchez commented Feb 24, 2022

sujeet100 commented Mar 29, 2022 • edited

alvarosanchez commented Mar 29, 2022

sujeet100 commented Apr 25, 2022

alvarosanchez commented Apr 25, 2022

matt-snider commented Oct 7, 2022

rdamus commented Mar 27, 2023

sujeet100 commented Feb 22, 2022 •

edited

sujeet100 commented Mar 29, 2022 •

edited