Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock when there is a wrongly configured client #5824

Open
AndresSanchezG opened this issue Mar 21, 2024 · 2 comments
Open

Deadlock when there is a wrongly configured client #5824

AndresSanchezG opened this issue Mar 21, 2024 · 2 comments
Labels
Waiting on feedback Issues that require feedback from User/Other community members

Comments

@AndresSanchezG
Copy link

Describe the bug

Deadlock seems to happen when a jenkins kubernetes agent is trying to be created but it cannot due to some error. In this case having wrongly configured a container.

Fabric8 Kubernetes Client version

6.4.1

Steps to reproduce

Example:

podTemplate(containers: [
    containerTemplate(name: 'maven', image: 'wrong_name:3.0.0', command: 'sleep', args: '99'),
  ]) {
    retry(count: 100, conditions: [kubernetesAgent(), nonresumable()]) {
    node(POD_LABEL) {
        stage('Get a Maven project') {
            container('maven') {
                stage('Hello')

{                     sh 'echo Hello'                 }
            }
        }
    }
    }
}

"wrong_name" image does not exists and then it retries 100 times.

While this pod is being created, when going to manage/configure and pushing save button; configuration is not saved and an error 504 appears in browser.

Performing a thread dump, this thread appears not to finish:

 Handling POST /manage/configSubmit from 10.133.236.30 : https-jsse-nio-8443-exec-10
java.base@11.0.17/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.17/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@11.0.17/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
java.base@11.0.17/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
java.base@11.0.17/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
java.base@11.0.17/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:162)
io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:185)
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.watch(BaseOperation.java:611)
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.watch(BaseOperation.java:598)
org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper.watchCloud(Reaper.java:221)
org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper.watchClouds(Reaper.java:194)
org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$ReaperSaveableListener.onChange(Reaper.java:532)
hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:82)
jenkins.model.Jenkins.save(Jenkins.java:3591)
hudson.BulkChange.commit(BulkChange.java:98)
jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:4036)
java.base@11.0.17/java.lang.invoke.LambdaForm$DMH/0x00000008004c6c40.invokeVirtual(LambdaForm$DMH)
java.base@11.0.17/java.lang.invoke.LambdaForm$MH/0x00000008004c3c40.invoke(LambdaForm$MH)
java.base@11.0.17/java.lang.invoke.LambdaForm$MH/0x00000008004c5040.invoke(LambdaForm$MH)
java.base@11.0.17/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
java.base@11.0.17/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
org.kohsuke.stapler.SelectionInterceptedFunction$Adapter.invoke(SelectionInterceptedFunction.java:36)
org.kohsuke.stapler.verb.HttpVerbInterceptor.invoke(HttpVerbInterceptor.java:48)
org.kohsuke.stapler.SelectionInterceptedFunction.bindAndInvoke(SelectionInterceptedFunction.java:26)
org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:836)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
org.kohsuke.stapler.Stapler.service(Stapler.java:248)
javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:163)
org.jenkinsci.plugins.corsfilter.AccessControlsFilter.doFilter(AccessControlsFilter.java:79)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:76)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:118)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:660)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.base@11.0.17/java.lang.Thread.run(Thread.java:829)

And blocks other threads:

"Handling GET /tcpSlaveAgentListener/ from 10.133.236.30 : https-jsse-nio-8443-exec-2 TcpSlaveAgentListener/index.jelly" Id=36 Group=main BLOCKED on hudson.model.Hudson@42d24189 owned by "Handling POST /manage/configSubmit from 10.133.236.30 : https-jsse-nio-8443-exec-10" Id=44
	at jenkins.model.Jenkins.getAgentProtocols(Jenkins.java:1255)
	-  blocked on hudson.model.Hudson@42d24189
	at hudson.TcpSlaveAgentListener.getAgentProtocolNames(TcpSlaveAgentListener.java:163)
	at java.base@11.0.17/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base@11.0.17/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base@11.0.17/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base@11.0.17/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
	at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:76)
	at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:259)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99)
	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:66)
	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:55)
	at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)
	at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)
	at org.kohsuke.stapler.Facet.handleIndexRequest(Facet.java:284)
	at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:104)
	at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:32)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:172)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:163)
	at org.jenkinsci.plugins.corsfilter.AccessControlsFilter.doFilter(AccessControlsFilter.java:79)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
	at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
	at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:76)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:118)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:145)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:660)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	-  locked org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@44b1df47
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base@11.0.17/java.lang.Thread.run(Thread.java:829)

	Number of locked synchronizers = 1
	- org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker@6e02fcd8

Checking the calls it seems to be a problem happening Kubernetes client code in https://github.com/fabric8io/kubernetes-client/blob/v6.8.1/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/BaseOperation.java#L611 introduced by 0ebd164

THis problem started appearing after change in Jenkins core: https://github.com/jenkinsci/jenkins/blame/jenkins-2.414.2/core/src/main/java/jenkins/model/Jenkins.java#L1254 introduced by jenkinsci/jenkins@81bf589

Expected behavior

No deadlock should appear.

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

1.24

Environment

Linux

Fabric8 Kubernetes Client Logs

No response

Additional context

No response
@shawkins
Copy link
Contributor

shawkins commented Mar 21, 2024 •
edited

@AndresSanchezG is it a deadlock, or just that one of your theads is holding a lock that is needed by another? A deadlock isn't clear from what you provided.

Also the lock involved "blocked on hudson.model.Hudson@42d24189" isn't coming from fabric8. Since fabric8 can make blocking calls you may want to reconsider how you are holding that lock.

Are you expecting the watch call to fail fast? That may have to do with the number of retries that are allowed - if the misconfiguration is producing an error that seems recoverable, then we may keep trying the operation.

@AndresSanchezG
Copy link
Author

@shawkins, so from the threads and traces it seems that while a pod is trying to start up the lock is not released. May be not a deadlock but there is a situation where the lock is never released and therefore it impacts the system. I have already spoken with Jenkins core developers and it seems to be on the part of the plugin.

@manusa manusa added the Waiting on feedback Issues that require feedback from User/Other community members label May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Waiting on feedback Issues that require feedback from User/Other community members
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@manusa @shawkins @AndresSanchezG