Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Show pod logs when pod never become available #1251

Open
lahabana opened this issue Mar 2, 2023 · 0 comments · May be fixed by #1252
Open

Show pod logs when pod never become available #1251

lahabana opened this issue Mar 2, 2023 · 0 comments · May be fixed by #1252
Labels
enhancement New feature or request

Comments

@lahabana
Copy link
Contributor

lahabana commented Mar 2, 2023

Currently when you use k8s.WaitUntilPodAvailable() and your pod fails to start because of a container failing you get an output like:

   2023-03-02T09:35:04Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:04Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:04Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:07Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:07Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:07Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:10Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:10Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:10Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:13Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:13Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:13Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:16Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:16Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:16Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:19Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:19Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:19Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:22Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:22Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:22Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:25Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:25Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:25Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:28Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:28Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:28Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:31Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:31Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:31Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:34Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:34Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:34Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:37Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:37Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:37Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:40Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:40Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:40Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:43Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:43Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:43Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:46Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:46Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:46Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:49Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:49Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:49Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:52Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:52Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:52Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:55Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:55Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:55Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:35:58Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:35:58Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:35:58Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:01Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:01Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:01Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:04Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:04Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:04Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:07Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:07Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:07Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:10Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:10Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:10Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:13Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:13Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:13Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:16Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:16Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:16Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:19Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:19Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:19Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:22Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:22Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:22Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:25Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:25Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:25Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:28Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:28Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:28Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:31Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:31Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:31Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.
   2023-03-02T09:36:34Z retry.go:91: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned.
   2023-03-02T09:36:34Z client.go:42: Configuring Kubernetes client using config file /root/.kube/kind-kuma-2-config with context
   2023-03-02T09:36:34Z retry.go:103: Wait for pod test-server-57d5fdcbc8-tt2cq to be provisioned. returned an error: Pod test-server-57d5fdcbc8-tt2cq is not available, reason: , message: . Sleeping for 3s and will try again.

It would be nice to:

  1. Get the conditions of the pod
  2. If the pod.Status.Phase == running dump the logs of the default container.
lahabana added a commit to lahabana/terratest that referenced this issue Mar 2, 2023
@lahabana
add logs when pod failed to start when using WaitUntilPodAvailableE
632e07d 
Fix gruntwork-io#1251

Signed-off-by: Charly Molter <charly.molter@konghq.com>
@lahabana lahabana linked a pull request Mar 2, 2023 that will close this issue
Open
4 tasks
lahabana added a commit to lahabana/terratest that referenced this issue Mar 10, 2023
@lahabana
add logs when pod failed to start when using WaitUntilPodAvailableE
c37568a 
Fix gruntwork-io#1251

Signed-off-by: Charly Molter <charly.molter@konghq.com>
@denis256 denis256 added the enhancement New feature or request label Apr 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add logs when pod failed to start when using WaitUntilPodAvailableE lahabana/terratest
2 participants
@lahabana @denis256