Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retain default group membership when configuring additional paths on probe health groups #40268

Open
tharakadesilva opened this issue Apr 9, 2024 · 4 comments
Labels
type: enhancement A general enhancement
Milestone

Comments

@tharakadesilva
Copy link

tharakadesilva commented Apr 9, 2024

When configuring additional paths for Spring Boot actuator health groups ('readiness', 'liveness', etc.), unrelated health indicators to those groups unexpectedly start failing. This behavior presents risks for production environments, potentially leading to unnecessary restarts or pod terminations.

Impact:

This issue introduces the risk of unnecessary pod restarts or service disruptions within environments using health endpoints for availability checks (e.g., Kubernetes). A failing health indicator on one path could lead to cascading failures within liveness or readiness probes.

Steps to reproduce:

  1. Bootstrap a new project with SpringBoot 3.2.4.

  2. Create a custom health indicator that always reports "DOWN":

    @Bean
    HealthIndicator myHealthIndicator() {
      return () -> Health.down().build();
    }
  3. Add an additional paths to the readiness and liveness groups:

    management.endpoint.health.group.readiness.additional-path=server:/readyz
    management.endpoint.health.group.liveness.additional-path=server:/livez
    

Expected behavior (and this works without the additional path configs in step (3)):

  • /actuator/health - should fail
  • /actuator/health/readiness - should pass
  • /readyz - should pass
  • /actuator/health/liveness - should pass
  • /livez - should pass

Actual behavior:

  • /actuator/health - fails
  • /actuator/health/readiness - fails (unexpectedly)
  • /readyz - fails (unexpectedly)
  • /actuator/health/liveness - fails (unexpectedly)
  • /livez - fails (unexpectedly)

I would have expected readiness and livenss to have failed if I only set the following props:

management.endpoint.health.group.readiness.include=my
management.endpoint.health.group.liveness.include=my

Demo Project (see tests):

demo.zip

Potential Workaround (Temporary):

Avoid using additional-path on health groups where this side-effect could be disruptive.

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Apr 9, 2024
@philwebb
Copy link
Member

This is a little confusing, but what's happening here is when you declare the following:

management.endpoint.health.group.readiness.additional-path=server:/readyz
management.endpoint.health.group.liveness.additional-path=server:/livez

Spring Boot is creating two new groups called readiness and liveness which by default include all health indicators. If you want to only include the liveness and readiness probes you have two options. You can either set the following property to create /readyz and /livez additional paths:

management.endpoint.health.probes.add-additional-paths=true

or you can update the properties to include the correct health indicators:

management.endpoint.health.group.readiness.include=readinessState
management.endpoint.health.group.readiness.additional-path=server:/readyz
management.endpoint.health.group.liveness.include=livenessState
management.endpoint.health.group.liveness.additional-path=server:/livez

@philwebb
Copy link
Member

Flagging for a team discussion since I wonder if we should do more to improve things. Perhaps if management.endpoint.health.probes.enabled=true is set and management.endpoint.health.group.liveness doesn't change the include or exclude we should keep the probe defaults.

@philwebb philwebb added for: team-meeting An issue we'd like to discuss as a team to make progress type: enhancement A general enhancement and removed status: waiting-for-triage An issue we've not yet triaged for: team-meeting An issue we'd like to discuss as a team to make progress labels Apr 13, 2024
@philwebb philwebb changed the title Actuator health groups fail for unrelated indicators when additional paths are configured Retain default group membership when configuring additional paths on probe health groups Apr 17, 2024
@philwebb
Copy link
Member

We're going to look to see if we can improve the default behavior so that liveness and readiness groups have sensible memberships unless the user has specifically configured them otherwise. This is a breaking change so we can't consider it a bug.

@philwebb philwebb added this to the 3.x milestone Apr 17, 2024
@tharakadesilva
Copy link
Author

Thanks @philwebb!! I've applied the workaround (1) that you recommended and it is working as expected, thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: enhancement A general enhancement
Projects
None yet
Development

No branches or pull requests

3 participants