You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since we're deploying on Kubernetes, we're making use of liveness and readiness probes. If these probes fail too often, the container is deemed unhealthy and is restarted.
Sometimes we have long-running data migrations, which are executed as part of the regular migrate step in the docker command script. These can run so long that the liveness probes fail so often, resulting in a container being restarted. If this happens over and over again, it can mean that the new version/container is never coming up.
One way to deal with this, would be to use the same container image in an init container which only runs the migrations. These containers don't have the "time restriction" that the "main" containers have.
No code changes are particularly needed for this, except for maybe a script/entrypoint similar to bin/docker_start.sh which stops at running the migrations, so we can adapt the infrastructure to that.
The text was updated successfully, but these errors were encountered:
@damm89 It's indeed good practice to run the migrations from an init-container. This is currently not the case, which in practice means that you'll bump into the issue described by Sergei when your migrations + application start-up run longer than 55 seconds.
Since we're deploying on Kubernetes, we're making use of liveness and readiness probes. If these probes fail too often, the container is deemed unhealthy and is restarted.
Sometimes we have long-running data migrations, which are executed as part of the regular
migrate
step in the docker command script. These can run so long that the liveness probes fail so often, resulting in a container being restarted. If this happens over and over again, it can mean that the new version/container is never coming up.One way to deal with this, would be to use the same container image in an init container which only runs the migrations. These containers don't have the "time restriction" that the "main" containers have.
No code changes are particularly needed for this, except for maybe a script/entrypoint similar to
bin/docker_start.sh
which stops at running the migrations, so we can adapt the infrastructure to that.The text was updated successfully, but these errors were encountered: