Privilege separation for vault as a system service

[ketzacoatl]

Seeing the error message in 637549b, I wonder: if Vault requires root privs, is it feasible to implement privilege separation in Vault such that root privs are dropped when they are not needed?

[armon]

This is a painful one in Go in general. See: golang/go#1435
Basically there is no easy way to "setuid" down from root. The simplest way is to disable mlock and run at a lower privilege.

[ketzacoatl]

That is rough. @armon, can you help me understand the implications of disabling mlock?

[armon]

Using mlock prevents pages from being swapped to disk, effectively "locking" virtual memory into physical memory. This requires root privilege to invoke (at the syscall level), so a non-root user must disable that feature.

However, if you disable swapping entirely (pretty standard practice for production servers), then the OS should never swap to disk anyways.

[ketzacoatl]

Thank you for the explanation!

[MadsRC]

Swapping to disk would be counter-productive to what Vault tries to accomplish? If Vault swaps, you risk writting data to persistent local storage that is not yet encrypted, or did I miss something?

Saying that disabling swap on production server is "pretty standard" is kinda bold, don't you think?
I mean, it really depends on what you are running. In some cases swap actually gives slightly better performance, and in some cases what gives that little bit of extra performance actually decreases performance (With tiered storage).

[armon]

@MadsRC I agree, swapping is counter-productive, which is why we attempt to mlock by default. In situations where that can't be done (no OS support, non-root user, etc), you can disable swapping at the system level. I don't think disabling swap is particularly bold.

[ketzacoatl]

@armon, I think he meant it was bold to say disabling swap is common in production. It is.. but so is running swap in production (often as a bandaid).

Either way.. I think it is fair to say if you want to run as non-root user, you must disable mlock, but I would like to know.. if swap is used, are we potentially leaking unencrypted secrets? Is vault's memory more or less cleartext or protected in some form?

Thanks for putting up with my questions :)

[armon]

@ketzacoatl You could potentially be leaking secrets yes. There is very little data that is cleartext in memory, but anything that is currently in the request path being processed as well as the master encryption keys are necessarily in clear text. The majority of data is LRU caches of the physical backend, and all that data is still encrypted in memory. There is also the GC to consider, as previously decrypted data may be unused but still paged in.

[sheldonh]

I think this can be closed when the website is next deployed. By default, vault server doesn't listen on privileged ports, and the next deploy of the website documentation will offer 3 approaches to avoiding swap leak in production (none of which require running vault as root).

[armon]

I'm just going to close this now, as it's already committed in master!

[directionless]

This is in the docs committed in #268 but since this thread shows up in the google results, I wanted to note it here as well. if you're on linux, you can use setcap to work around this. See the docs