OpenSSH privilege separation directory is not created before exec sshd

[dannysauer]

determined/harness/determined/launch/horovod.py

Lines 40 to 48 in 78505d6

	run_sshd_command = [
	"/usr/sbin/sshd",
	"-p",
	str(DTRAIN_SSH_PORT),
	"-f",
	"/run/determined/ssh/sshd_config",
	"-D",
	]
	return pid_server_cmd, run_sshd_command

Starting with OpenSSH 7.5, the UsePrivilegeSeparation config parameter for sshd was deprecated, making separation mandatory. There is an explanation of what the directory does here, but the important part is that there's a check in OpenSSH which verifies that directory to exist.

Normally, either the init script installed with sshd on a system or one of the "create tempfiles" startup scripts creates this directory, so users don't see it. But since we directly launch sshd and containers often don't have those features, that directory can be missing on startup. This leads to the error message "Missing privilege separation directory: %s". The directory is a compile-time option, so can potentially vary by distribution.

I'm not positive if that check is bypassed when sshd runs non-privileged. Nor am I sure how to easily determine the compiled-in default. But I'm opening this issue with a link to the relevant code block so it can be further investigated. :)

[dannysauer]

This was initially reported in a Slack #support thread at https://determined-community.slack.com/archives/CV3MTNZ6U/p1653688666785279?thread_ts=1653679454.491989&cid=CV3MTNZ6U

[dannysauer]

Per openssh/openssh-portable@d13281f, the chroot directory check only happens when sshd runs as uid 0 since OpenSSH 7.6p1 (hopefully people have upgraded by now). This makes some sense, as the directory is only useful when the user has the need (and capability) to chroot and drop privileges.

It doesn't look like there's a super-clean way to get the configured path from the binary; running strings or parsing error output is not great. Just around my house - Ubuntu uses /run/sshd, SUSE uses /var/lib/empty, and Fedora appears to use /usr/share/empty.sshd. I don't think it sounds like much fun checking for every possibility.

Do we ever actually need to run sshd as UID 0? Perhaps the most reasonable fix is to just always run processes as non-root, and get a workaround for this issue as a free side-effect?

[rb-determined-ai]

Yes, we do need to if the user chooses to run containers as UID 0. Telling users "you can't run as root because we can't guess what the privsep directory is in your sshd binary" is much worse than telling users "you have to have a functional sshd installation in your container", which after reviewing the code you showed, seems like the only reasonable solution here.

[dannysauer]

After some internal discussion, documenting this directory requirement (along with other requirements) in the custom container documentation seems like the most feasible solution. Maintaining an exhaustive list of every possible vendor's privsep directory doesn't seem like a very good solution.

Users who derive their container from our container as the base get the directory pre-created, and we'll have a list available for the remaining users who prefer to fully roll their own. This ticket will be closed when that checklist is prepared.