Post deployment hook cpod does not share network namespace with main pod

[jorgemoralespou]

I have a post-deployment hook that want to connect to the application being started in a container in the main pod (the deployment) but it's failing to connect.
If I get into the main pod's container and do curl http://localhost:8080/xxx I get a response, while from the post-deployment hook I get a curl error curl: (7) Failed connect to 127.0.0.1:8080; Connection refused

Version

1.3.0-alpha.3

Steps To Reproduce

oc create -f https://raw.githubusercontent.com/jorgemoralespou/ose3-parks/6dfdc2f8394d8673d8ae92dd5b5ec7acad461a34/ose3/application-template-wildfly.json
oc new-app parks-map

Current Result

Failed to connect

Expected Result

Connect succesful

Additional Information

This is 100% reproducible. Since I have set por the post deployment hook Retry, at the end the post-deployment ends in CrashLoop which is also strange

cc/ @Kargakis

[jorgemoralespou]

Using the servicename instead of localhost works, as I assume the endpoint is already there, but then this means that the deployment has finished when doing the post-deployment hook.

[0xmichalis]

@jorgemoralespou I don't see any post-hook defined in your template.

Using the servicename instead of localhost works, as I assume the endpoint is already there, but then this means that the deployment has finished when doing the post-deployment hook.

Have you noticed something different? We are waiting for pods to become ready before executing the post-hook.

I don't think we are doing anything for configuring network namespaces for the deployer pods. @mfojtik @smarterclayton @knobunc got any ideas on this?

[jorgemoralespou]

@Kargakis true. Had the changes not commited. You can see it here: https://raw.githubusercontent.com/jorgemoralespou/ose3-parks/6dfdc2f8394d8673d8ae92dd5b5ec7acad461a34/ose3/application-template-wildfly.json

In any case, what I see is that this is executed as a new pod, and hence there is no way that network namespace can be shared with main pod, unless something is done ad-hoc.

I think there is an issue (#9924) that could change how post-deployment hooks work in that instead of launching a new pod would launch a container in the same pod, which make more sense, as all the namespaces and filesystem would be shared by default. Also, the quota for pod (when available) could help on defining proper limits. Right now the post-deployment takes all the same quota (or is BestEffort) while it should be quoted and counted against the pod that executes the post-deployment IMHO.

[smarterclayton]

If you want to deeply control the deployment process, use a custom deployment. Hook pods are for when you need to run code that is easily made independent of the deployment process. If you can't do it with a custom deployment, describe why and we'll try to address that.

[jorgemoralespou]

@smarterclayton I guess you haven't read the thread. What I want is to be able to install data in the database from a post deployment hook, but instead of executing a script to sed the database connecting remotely i want to call my app (in localhost) to do it, since this is a common practice in javaland

[smarterclayton]

That would be what you can do in a custom deployment. The only
question is whether you should do that.

[jorgemoralespou]

@smarterclayton then what are post deployment hooks for? This should be the easy way to solve things. It's just a matter of whether this is a new pod or a container attached and removed to the main pod. IMHO

[smarterclayton]

Not the same thing, really. You can't mix run once pods and run
forever containers in a pod. At best we could add an ExecHook that
runs in an existing container, but you wouldn't have the appropriate
CPU / memory quota, and we don't support dynamic pod changes to either
of those (and wouldn't).

So your options are

1. New pod
2. In the deployer pod as a custom deployment
3. As an exec into an existing container, and odds are you'll run out
   of memory unless you account for extra space (not recommended).
4. We could create an init container for exactly one pod, but
   replication controllers won't allow that easily so I'm not inclined to
   break that abstraction.

If this is an app that can tolerate > 1 instance of your app, exec pod
is the way to go. If you only support 1 instance, use an init
container. When we add deployment behavior to petsets, we might have
a way to do this per pet, but it'll be harder. If you want to risk
the memory limits we can potentially try the exec in pod option, but
won't be before 1.5

[jorgemoralespou]

@smarterclayton can you describe option 3. Sorry for my ignorance.

As a note, and I think we've already talked about this, launching a
complete pod for a deployment hook is too much overhead since they get same
requests, and for a main pod where the hook might be very low resource
intensive this can be a problem. I've seen hooks failing because of this.
I think it is better to execute the process in the container or make
possible to attach/detach a new container with its own requests. Once
requests per pod lands this will be something users will account for, the
extra between the containers and the pod for the hooks.
This also simplifies quota management.

Also being able to mix in a pod a runonce container could be helpful in
some other cases, but mostly for things that could be done in pods as
sidecar.
Right now, you need to also set s volume if files need to be shared between
main pod and hook pod, which is cumbersome.

El 25 ago. 2016 1:26, "Clayton Coleman" notifications@github.com escribió:

Not the same thing, really. You can't mix run once pods and run
forever containers in a pod. At best we could add an ExecHook that
runs in an existing container, but you wouldn't have the appropriate
CPU / memory quota, and we don't support dynamic pod changes to either
of those (and wouldn't).

So your options are

1. New pod
2. In the deployer pod as a custom deployment
3. As an exec into an existing container, and odds are you'll run out
   of memory unless you account for extra space (not recommended).
4. We could create an init container for exactly one pod, but
   replication controllers won't allow that easily so I'm not inclined to
   break that abstraction.

If this is an app that can tolerate > 1 instance of your app, exec pod
is the way to go. If you only support 1 instance, use an init
container. When we add deployment behavior to petsets, we might have
a way to do this per pet, but it'll be harder. If you want to risk
the memory limits we can potentially try the exec in pod option, but
won't be before 1.5

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10603 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEyDoYBRC88UWG9DAr43JbiZA75r93Jks5qjNMkgaJpZM4JrX3U
.

[smarterclayton]

@smarterclayton can you describe option 3. Sorry for my ignorance.

We could in some future release add a "exec in" hook that invokes exec
on one of the new deployment cntainers.

As a note, and I think we've already talked about this, launching a
complete pod for a deployment hook is too much overhead since they get same
requests

If you need to start your entire JVM to run DB tests, then your
request in the hook pod is going to likely match the target pod. If
you don't, then hook pods should let you specify resources.

, and for a main pod where the hook might be very low resource
intensive this can be a problem. I've seen hooks failing because of this.
I think it is better to execute the process in the container or make
possible to attach/detach a new container with its own requests.

Execute process in container is worse - that means every pod in your
deployment has to have extra wiggle room for memory, meaning you waste
resources.

We will not be adding the ability to dynamically add / remove
containers to an individual pod anytime soon (philosophically its
against a bunch of core behaviors) so it's not worth trying to solve
things that way.

Once
requests per pod lands this will be something users will account for, the
extra between the containers and the pod for the hooks.

I don't think requests per pod will solve this problem. Adding an
extra 100M slack for every pod is just as wasteful as of your quota.
Some pods will have burst, but not all.

This also simplifies quota management.

Also being able to mix in a pod a runonce container could be helpful in
some other cases, but mostly for things that could be done in pods as
sidecar.

It won't happen (we've said we will not do it for various reasons) so
to solve your near term problem it's important to focus on what can
work.

Right now, you need to also set s volume if files need to be shared between
main pod and hook pod, which is cumbers

If all your hooks depend on a "main pod" to work you're probably
shouldn't be using a replicaset / deployment. Replicasets and
deployments don't offer that sort of guarantee. We aren't going to
have special pods under a replica set, so if your hook pod needs local
disk shared with your app code you need a different solution (possibly
exec, or possibly just a differently structured app).

[jorgemoralespou]

Would doing a curl http://localhost:8080/ws/load be one of those things that could produce a big overhead into the container?

This is one of the places I see more confusion on users of our platform, how to best do capacity planning and limit enforcement for the workloads. SO far, most of the users (a.k.a customers) I know, use Best effort containers.

In an ideal world, when users would use the 3 QoS for containers (end within time) to pods, I think there will need to account for whatever is executed in the containers and hooks as these are part of the application (otherwise they would not be implemented as hooks, but as something else, like jobs or ...).

And about adding/removing containers to pods, I was mentioning this as an option cause it's been brought up in many discussions around builds with @bparees.

To be honest, I still don't see a reason (as you haven't explained them) why having a runonce container in a pod is a problem and why it will not be solved.

In any case, as I said, for this issue, I can use servicename to access networking on the main application pod instead of localhost. Now I understand is working as designed, although not properly documented.

I'll close this issue, since it seems not to be one, but I'll keep bugging on the "hooks" should run in the pod not in a different pod.
:-D

[smarterclayton]

Would doing a curl http://localhost:8080/ws/load
http://localhost:8080/ws/load be one of those things that could produce
a big overhead into the container?

Probably not. So an exec may be reasonable for your use case. Although
"load" for some frameworks might be incredibly expensive, and thus cause
OOM in the cluster.

You could also use a postStart pod hook.

This is one of the places I see more confusion on users of our platform,
how to best do capacity planning and limit enforcement for the workloads.
SO far, most of the users (a.k.a customers) I know, use Best effort
containers.

In an ideal world, when users would use the 3 QoS for containers (end
within time) to pods, I think there will need to account for whatever is
executed in the containers and hooks as these are part of the application
(otherwise they would not be implemented as hooks, but as something else,
like jobs or ...).

I don't think that's the case - hook execution for some people might be
"another JVM". Just depends on your workload. Combining workloads is an
antipattern.

And about adding/removing containers to pods, I was mentioning this as an
option cause it's been brought up in many discussions around builds with
@bparees https://github.com/bparees.

It's been discussed, but there are lots of reasons not to do it (it changes
the scheduling decision, can lead to resource contention, makes security
harder to reason about, makes it hard for naive clients to react, etc). I
don't think we will add this for a long time, and maybe never.

To be honest, I still don't see a reason (as you haven't explained them)
why having a runonce container in a pod is a problem and why it will not be
solved.

kubernetes/kubernetes#127 (comment)

We've continually reinforced that decision via other additions (pod level
cgroups, init containers, etc)

In any case, as I said, for this issue, I can use servicename to access
networking on the main application pod instead of localhost. Now I
understand is working as designed, although not properly documented.

I'll close this issue, since it seems not to be one, but I'll keep bugging
on the "hooks" should run in the pod not in a different pod.
:-D

Open one for "ExecInContainer"