TLS support

[jedisct1]

Hi,

Sorry for opening a lame ticket instead of a pull request.

Is TLS (with client cert authentication) something that may eventually be implemented in oklog?

[peterbourgon]

Sure, I'm open to it, of course.

edit: It would make sense both for query/store communication, as well as ingest/store replication.

[squiddle]

Since there seems to be no protocol between forwarder and ingester couldn't the ingester just take connections from a local domain socket and then the transfers could be done by anything which can send lines of text to a socket.

e.g. netcat, stunnel, socat, ...

Using stunnel would provide a TLS solution.

[peterbourgon]

Yep, to be explicit the communication paths are

Components	Protocol	Security
Forwarder–Ingester	plain socket	whatever you want
Ingester–Store	HTTP/1.1	TLS applicable
Store–Query	HTTP/1.1	TLS applicable

[squiddle]

stunnel can also act as a proxy for http so it might be able to use between the backend services as well.

[jtarchie]

Instead of implementing multiple solutions. Would it be possible to abstract the transport mechanism of how things talk to each other.

For example, a lot of this is just plain old TCP right now. It would be nice to have the components talk to each other different transport mechanisms based on the user requirements.

An good example is Mangos transports abstractions. That way users could add transport mechanisms as they need them.

[peterbourgon]

Any transport abstraction beyond plain TCP or HTTP, especially something like ZeroMQ or mangos, would be severe overengineering at this stage of the project.

[jtarchie]

I am not saying to use mangos for a transport mechanism. I'm just referencing their implementation of different transport protocols. I agree mangos is over engineering.

It would be nice to implement the transport mechanism as an interface. It implements methods, Start, Accept, Stop, (for example) and knows how to pass incoming and outgoing messages to the server it is running on. That abstraction would then worry about retry logic, encryption, etc.

Our systems require mutual TLS, not just TLS.

[peterbourgon]

Ah, okay. I'm on board with this!

[msm595]

It would be nice to have a single setting to bind oklog to a certain ip (for instance, 127.0.0.1) so a TLS tunnel like ghostunnel can be put in front of it securely. Right now I think the ips are spread across api, ingest.fast, ingest.durable, ingest.bulk, and cluster.