-
Notifications
You must be signed in to change notification settings - Fork 18.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add docker run --net=network:<net-name> #14143
Conversation
As per networking BoF, add the missing UI to have a container join an extant network. This a wart whereby a service is actually created and published with the same name as the container. Before committing this, I would argue to fix that and hide that detail of the implementation. Signed-off-by: Tim Hockin <thockin@google.com>
+1 based on today's discussion. Adding @abhat |
@@ -803,6 +803,11 @@ func (container *Container) AllocateNetwork() error { | |||
return fmt.Errorf("conflicting options: publishing a service and network mode") | |||
} | |||
|
|||
if mode.IsNetwork() { | |||
parts := strings.SplitN(string(mode), ":", 2) | |||
networkName = parts[1] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't the networkDriver need be recalculated or am I missing something.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rajat,
I don't think so - this invocation path will never create a network - it
just references an existing one. If you refer to a network that does not
exist, this will fail (which is what we want, right?)
On Tue, Jun 23, 2015 at 7:49 PM, Rajat Chopra notifications@github.com
wrote:
In daemon/container_linux.go
#14143 (comment):@@ -803,6 +803,11 @@ func (container *Container) AllocateNetwork() error {
return fmt.Errorf("conflicting options: publishing a service and network mode")
}
- if mode.IsNetwork() {
parts := strings.SplitN(string(mode), ":", 2)
networkName = parts[1]
Doesn't the networkDriver need be recalculated or am I missing something.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/14143/files#r33113600.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. You are right. I didn't follow the code path correctly.
+1 based on the feature based on our discussions today and the feedback we have from our users. +1 on fixing the wart before committing if possible. (Happy to consider alternative syntax if people have strong views so long as the syntax are the same.) |
Also a +1, and thanks to Tim, from me. |
+1 |
+1! |
Thanks Tim! +1. |
+1 |
Not sure I understand the difference with --publish-service, is it simply that the service name is not specified by the user, but defaults to the container name or id instead? |
The argument is that the concept of "service" should not be part of the UX There was some ... let's call it "strong" sentiment about this at the The fact that this PR still implies a service and that the implied service On Fri, Jun 26, 2015 at 9:47 PM, Solomon Hykes notifications@github.com
|
Service discovery is a fundamental aspect of this design, and has been for 6 months now. I remember clearly explaining it at the dockercon eu ecosystem meeting. The reason is simple: if we don't define a standard interface for service discovery, we allow for fragmentation and incompatibility between certain plugins and certain apps. Anything that breaks "every plugin works with every app" is a non-starter. This design is popular with users. To my knowledge it does not introduce any new problems for users, and it solves many. If you have arguments to the contrary I am happy to hear them. |
Your explanation of "container that consumes services but does not expose them" is a good argument for finding a word other than "service". My favorite other candidate is "endpoint" since it works for bi-directional connectivity. I think it makes sense to have an "invisible" setting for consume-only endpoints, which removes them from discovery. They should still be endpoints, since certain properties are useful for both directions (for example port redirect rules). It's also useful for endpoints to have names even if they are not discoverable: if only to designate them for management. how about: docker run --connect-as db.myapp Note that invisibility of endpoints is a nice to have. Even if a db can discover the web node that consumes it, there is no security consequence since only exposed ports are allowed through anyway. This maps the reality of the system where usually on a given network every node can resolve the hostame of every other node, even if they can't reach them. |
One use case I've heard come up multiple times with our users is for short lived containers that may have a lifetime of <100ms. (e.g. Spark containers.) Forcing each of these to be available via service discovery is likely to impact scalability for large deployments. I think this is a reasonable real world example, and backs the idea that having an "invisible only endpoint" option to get a container onto a network without publishing as a service has value. |
Yeah that makes sense to me. Note that you can run containers without connecting them to a shared network at all, which would solve that particular problem. |
Only if the container is completely self-sufficient and isn't expected to consume any services. |
On separate note, if the intent of consuming a service has to be expressed explicitly, there has to be some kind of wildcard approach as well. Moreover, what is there to consumers advantage that such explicit expression of dependencies will bring? Also, @shykes, you said:
Do you mean connect as a client to That said, there probably exists some grand idea of service discovery semantics, but before climbing that high, a whole bunch of folks here are in need of a simple low-level solution that provides a way of associating containers with new networking subsystem, which is meant to be pluggable and extendible... However, at the moment it looks like one needs to go indirectly through this discovery subsystem that is anything but mature. |
I'm not arguing that service discovery is not important, I'm arguing that not every endpoint is a service. You've conflated two (important, good) ideas into one franken-concept. There is a cost to participating in service discovery - that cost should be optionally borne by only those things that actually want to offer service (not getting into what that means here). It's fine to make the implementation of both discoverable and non-discoverable endpoints use the same basic mehcanism, as long as the publish part of it is an extra step. An "invisible" endpoint that bypasses publishing is what I was proposing, and it sounds like that is basically acceptable to you?
Maybe I misunderstand - why make this two operations? The invisibility isn't really something users want on its own, it's an implementation detail. I think the default could go the other way - assume private unless explicitly published. The layering is much simpler to comprehend if "publish service" is As it is today, every container is on a network, right? If you don't say otherwise you get put on the default network with a made-up name (which by the Now being able to publish for discovery can be added purely on top of this - as a 2nd imperative step or as part of the same run command. The layering is cleaner, the concepts are additive and map to things the industry already understands. |
Yes an invisible endpoint is totally acceptable. The only question is how do we manage their names. I understand what you mean with the layering. We had assumed that the name is always present, and given in the same way, whether the EP is invisibleor not. An alternative would be to give a unique ID to each endpoint, and only assign a name when it is (optionally) made visible. Wdyt madhu @crosbymichael @icecrime ? |
@errordeveloper no, you completely misunderstood. There is no concept of dependency. The word "connect" was meant as "connect to the network" but the fact that you didn't understand tells me that it's probably not the right word. The "immature" system you speak of is basicay this: associate names to IPs and make them available to containers via hostname lookups. It's straightforward and useful, and is not forcing you to "work around" anything. |
I like that thinking, gives a nice parallel with other parts of Docker. Would end points also be accessible/discoverable through their ID? If so, have IDs priority over names (as is the case for container ID vs name) |
There seems to be two (unrelated) objectives between the participants here. 1) how do we logically describe the relationships between components in an application. 2) how we expose low level networking to containers. For "2" I have no clue what the UX should be. I understand very well the implementation details but I've yet to get many users actually ask me for this capability. So I shall punt to those more invested in the topic. For "1" I'm very interested in the UX. All that I would like to see is that containers can publish or consume a named service in a scope. Done and done. So for each container you just say it consumes X or publishes Y in scope Z. |
Alright, I think I found the solution to everyone’s problems (I’m joking, I know how complex this topic is). Back to my previous comment, I feel there are two concerns here. First how do I expose low level networking concepts to containers. Second, how do I express relationships between components in an application. Low Level NetworkingI really struggle with what the UI should be for low level networking in containers. In the grand scheme of an application platform, low level networking probably shouldn’t exist. I put the configuration of low level networking more on par with namespace control. Something like
The ServicesIn order to express relationships between components I do believe Let walk through some examples.
In the default case the container is neither publishing or consuming any service. For backwards compatibility this container will publish service
This container will publish service
This container will consume
In this situation the container will consume all services in the scope ScopesWe do not explicitly need scope operations. Scopes can dynamically come and go as they are used. If we add Link compatibilityIn order to support links in a backwards compatible way all containers will by default publish a service to scope
becomes shorthand for
The only different is that in this special case we add environment variables in the container to emulate the old behavior of links. Publishing multiple containers of the same serviceWhat happens when two containers publish the same service? I know there will be a bit of a debate on this one. I personally don't like magic and I think the safest solution is usually the simplest. In the situation that two containers publish the same name, then when looking up a service by name, you should get two results. In the terms of DNS this means multiple A records. An alternative idea is that they should be automatically load balanced. I am not in favor of this. Instead if somebody wants 'web' to be load balanced they should publish a load balancer as But what happens if the load balancer dies. I think a separate concept should exist that gives somebody the ability to have a VIP. Something like |
@shykes @thaJeztah We already provide a unique ID to endpoints for every endpoint that is created. So if an endpoint needs to be created implicitly then the changes in this PR are the only ones needed to achieve that. But right now we assign the container name to such endpoints and they become discoverable using that name. The only additional thing that would be preferable is to make these endpoints entirely not discoverable and completely hidden from the service discovery layer. It's only a simple change to achieve that. In essence I do agree with the layering that this will provide i.e container which just needs connectivity will connect to the network with implicit endpoints and the container which need to be network wide discoverable will attach to an already published service endpoint. |
@ibuildthecloud It seems you are proposing something very different from what is being discussed here. Let's not derail this discussion with that proposal since this discussion is getting somewhere. You could create another issue and document your proposal there so we can discuss there. WDYT? |
The architecture of libnetwork does not privilege the network named "bridge" above any other network created via the bridge driver, or any other driver, including plugin drivers. It seems natural to extend this into the user interface by letting a user refer to any network where they would refer to "bridge". I don't see how this represents a conflation of high-level and low-level. In any case, let me state my interest: I think users should be able to join containers to an plugin-provided network, without reference to services. Not everyone wants or needs service discovery -- for instance people who are driving docker via other tooling -- and those who do, don't necessarily want to name every container on the network as a service. As @lxpollitt pointed out, discoverability is different to reachability. Expanding the meaning of |
I think neither solution is mutually exclusive with each other, but it feels like we’re trying to force it in one direction instead of accepting that. I don’t see the point in that. Whether it’s a best practice or not, lots of people are going to want flat networks and direct addressability. Castrating the feature does not really serve anyone, as this discussion has indicated; nobody is really saying “this doesn’t work”, they’re saying “this isn’t right for docker” and I think that’s inaccurate simply by the number of people fighting for this feature in this thread. I would really like to see both service discovery and direct network management in docker and I really don’t see why we can’t have both, or why the UX has to be perfect; I think in practice most of the people who will want these kind of features want config files instead of CLI arguments anyway. I certainly understand there’s a lot of resistance to new arguments/syntax at docker because of “you have to support it forever”. I don’t think this is bad at all (hi, volume syntax), but we’re splitting hairs over a 10-line patch largely for people who will type whatever they need into bash to make docker use a specific network (I’m looking at you, pipework). Let’s not fight the tide of desire and instead embrace it.
|
I've heard over and over again that we should separate networking from service discovery and I agree. The problem with this PR is that it is presenting networks as a networking concept but I believe the real intention of "networks" in the UI is really a misnomer for "scope" in the context of service discovery. We should absolutely give people the ability to control the networking as an administrative function, but the current approach is going to cause problems. Let me give a more concrete example. Imagine an enterprise has a requirement that they want all container to be placed on VLAN 42 but then production containers would be put on VLAN 43. In the DockerCon hackathon somebody already started working on a VLAN extension to libnetwork and you can expect a real VLAN driver will appear. So now that we have networks it would make sense that I would create two networks in Docker called "dev" that is VLAN 42 and is the default and another called "prod" that is VLAN 43. Depending on where your container should run the user then does The problem with this scenario is that networks are not being exposed to the developer as a tool they can use. They are controlled by NetOps/Sysadmins. In every single exchange with @shykes regarding networks is the reoccurring topic is the we are exposing a concept to developer so that they can describe relationships between their containers. You hear ideas thrown around like "every docker compose file will create a network" and such. Those concepts can not be done if networking is controlled by an infrastructure team and not the developer. @shykes has asserted that networks is a concept that makes sense to developers. I would tend to agree with @shykes as really most developers don't understand networking, so their concept of a network is something logical like "dev", "prod", "the internet." The problem though with the term "network" is that it confuses the hell out of networking and infrastructure folks. They immediately think of VLANs, subnets, etc. I propose the current UI should be services and scopes. This more correctly describes the real intent of the current UI. Now this does leave real networking folks without a solution though. I think we should pursue a low level simpler approach that just exposes all the knobs. I like @erikh idea of maybe something that is config file based. We have a lot more flexibility in the config file because that is not really a developer facing interface. |
@ibuildthecloud I agree that your VLAN example is problematic, but my view is that VLAN as a technology is simply a poor choice for implementing different reachability domains for precisely the reason you mentioned: NetOps and sysadmins control it. VLAN IDs are a scarce resource and already have meanings within the enterprise network. VLAN is just a bad technology for this requirement. People being what they are, some people might use it. And yet, developers still need a tool that controls what containers can talk to what other containers, not just what names resolve to. There exist network technologies (not based on VLAN) that can provide those arbitrary reachability domains ("network" in the current CNM) for the developer, c.f. Calico, Weave, the new Overlay driver, etc. I'm fine with the service name "scope" being the same as network reachability "scope." But we do need a way to connect a container to networking and select the reachability without insisting it publish a service name. |
+1 @erikh @thockin @ibuildthecloud @squaremo et. al. Saying that "developers don't understand networking" and forcing the exposure of low level networking primitives via a "service" -- which is as vague as a concept as you can get, as we all agreed at the round table at DockerCon-- is prescriptive at best, condescending at worst (fine line there...) (As a side note, I'd rather point out to the fact that in @erikh original CNM proposal -- #9983 -- concept of "service" was not included, and rightfully so) Please keep separate "network" (==collection of endpoints able to communicate with each others e.g. via a "flat network") and "service" (whatever that means). To yours truly (and several above) they are fundamentally different as layers of abstraction -- very useful in their own respect, but different. So please do not conflate them in the name of "simplicity" and "avoid fragmentation and incompatibility". Afterall, the mantra always was " batteries included but removable " Thanks. |
This PR makes sense to me mainly because it's the expected behavior I had in mind when I heard about Docker gaining support for advanced networking features. I always imagined Docker supporting both the low level case where I want a container to join a specific network, and the high level case that would improve the UX around Docker links and service discovery. I feel this PR is a step in the right direction where Docker continues to offer "power" users and "platform" builders the flexibility to try new ideas without working around the Docker APIs (http and cli). |
+1 for this PR |
I've had some discussions with a number of libnetwork contributors and we've created #14593 to cover the immediate changes we'd like to see to the UI to handle a number of use cases with both Docker and Docker Compose. This includes using To all those who have already commented here, it would be fantastic to get your feedback on the issue that we raised as we move towards stabilising the UI! |
+1 for #14593 and removing the "network:" prefix. The only restriction this puts on users is that they cannot create networks using names that conflict with the existing built in names (such as "bridge"). That seems reasonably to me. |
@dave-tucker Does it mean that the subject will continue being discussed on docker/libnetwork side? Do we still need this PR against docker? @thockin Is it ok with you? |
I have been away for a while, I was literally just reading this and On Thu, Jul 23, 2015 at 11:12 AM, Arnaud Porterie notifications@github.com
|
I was pretty busy the last few, weeks, I am going to comment as I catch up @ibuildthecloud said
I am in favor of more control BUT I think that's a different topic than
I am not sure I agree with this. There are reasons to have "global" The examples you provided omit all mention of the network CLI, which is OK
If I want an HA service, and that is pretty much expected these days, I The other alternative is a load-balancer which fronts backends. A large So you introduce VIP. But it has to be multi-endpoint VIP. Which is a If you accept that lots of apps want HA and that load-balancers are the There are cases for which LB is not right, but they appear to be the @mavenugo said
I'm confused - I am trying to UN-conflate those layers. What in this PR @ibuildthecloud said
It's worse than that it's a half-solution because all of the other network If you want developers to talk in terms of scopes, make scopes imply @dave-tucker said
How do you disambiguate a network named "container" from --net=container:id I'll go look at #14593 now On Mon, Jul 13, 2015 at 10:31 AM, Alex Pollitt notifications@github.com
|
Ping @mavenugo @dave-tucker @mrjana |
@thockin said
+1 here. This was the consensus of everyone at the table @ dockercon in June. Allowing for this will enable more external usage of the libnetworking rather than having folks work around it. |
This looks like it is happening as part of #16645 |
As per networking BoF, add the missing UI to have a container join an
extant network. There is a wart whereby a service is actually created and
published with the same name as the container. Before committing this,
I would argue to fix that and hide that detail of the implementation.
Signed-off-by: Tim Hockin thockin@google.com
@squaremo @mavenugo @mrjana @lxpollitt @ibuildthecloud @errordeveloper @vbatts @bfirsh @mrunalp @rajatchopra