LIBFFI_TMPDIR should point somewhere that permits executables

[DaveCTurner]

JNA requires a temporary directory which is not mounted noexec, but JNA also uses libffi which itself requires a temporary directory that isn't mounted noexec. If none of the usual suspects work, libffi will fall back to $HOME which will often work, but in general might also not permit executables (or even exist).

We override the temporary directory that JNA uses by setting the java.io.tmpdir system property to ES_TMPDIR. I believe we should do the same thing for libffi, by setting the LIBFFI_TMPDIR environment variable. We should also document that this can be overridden much as we do for the jna.tmpdir system property (which trumps java.io.tmpdir if set).

Fixing this would, I believe, also fix #73309.

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[pugnascotia]

This sounds like a reasonable change. Do we have any ideas around testing this?

[DaveCTurner]

I see a couple of options:

1. We could have packaging tests that run on a SELinux installation that's sufficiently well-locked-down to reproduce the crash, and then confirm that setting LIBFFI_TMPDIR does resolve the problem. I now believe I have a decent handle on why the crash happens and how to reproduce it, so it's just™ a matter of setting up SELinux packaging tests 😉

2. We could just verify that Elasticsearch sees the LIBFFI_TMPDIR variable we expect it to see. Maybe we could even fail the node if running on Linux and that variable is not set, given that the startup scripts should be setting it?

[DaveCTurner]

After a bit of digging, it turns out that support for $LIBFFI_TMPDIR was added in libffi-3.4.2 but the latest release of JNA is using libffi-3.3.0, so we'll need to wait for JNA to pick up the newer version, then pick up the new JNA, and only then can we proceed with this.

[mark-vieira]

We could have packaging tests that run on a SELinux installation that's sufficiently well-locked-down to reproduce the crash

Yeah, seems we don't have such a testing environment right now but probably should. Do we have an idea of the specific controls that need to be enabled to reproduce?

[DaveCTurner]

Do we have an idea of the specific controls that need to be enabled to reproduce?

We know at least some of them I think:

• SELinux enabled & configured to prevent mmap() from creating anonymous executable & writeable mappings
• /tmp, /var/tmp, /dev/shm all nonexistent or mounted noexec
• $HOME nonexistent for the elasticsearch user
• All mount points in /etc/mtab not writable by elasticsearch, or mounted noexec
• $ES_TMPDIR (and hence java.io.tmpdir and/or jna.tmpdir) pointing somewhere writeable and not mounted noexec.
• $TMPDIR unset

That might not be everything, but this problem comes up frequently enough that I don't think it's going to need anything especially weird to reproduce. If the list above isn't enough then we can use strace to see what we're still missing I think (by comparing to the log shared in #77053 (comment)).

[DaveCTurner]

Didn't intend to close this, reopening.

[spricer]

It seems that -Djava.io.tmpdir alone is not enough to allow elasticsearch startup.
I analyzed running tasks of elasticsearch when /tmp is mounted with noexec removed and there is additional path required for elasticsearch to work.
This variable is called TMPDIR.
So in order to make it work the best way is to append next in /etc/sysconfig/elasticsearch file:

ES_TMPDIR=/var/lib/elasticsearch/tmp
TMPDIR=$ES_TMPDIR

If you have issue with above path (e.g. also mounted with noexec flag) you can introduce new path that is not under
noexec flag:

mkdir /opt/tmp
chown elasticsearch:elasticsearch /opt/tmp
chmod 755 /opt/tmp

Then update /etc/sysconfig/elasticsearch file accordingly:

ES_TMPDIR=/opt/tmp
TMPDIR=$ES_TMPDIR

I think developers need to hardcode somewhere TMPDIR=$ES_TMPDIR

[DaveCTurner]

That's one possible workaround @spricer, but TMPDIR has a wide impact and it's not always acceptable to redirect all temporary files to a location that permits executables. Instead we plan to set LIBFFI_TMPDIR instead as described above.

[DaveCTurner]

Ugh Github keeps misinterpreting commit messages and closing this. Reopening again...

[spricer]

That's one possible workaround @spricer, but TMPDIR has a wide impact and it's not always acceptable to redirect all temporary files to a location that permits executables. Instead we plan to set LIBFFI_TMPDIR instead as described above.

Hi Dave I understand your concern but please also consider that elasticsearch is executing under elasticsearch user session (at least Linux side) with its own env vars. That means that TMPDIR would be affected for that session/user only and if no other JAVA app is running noting else would be affected.

[DaveCTurner]

Sorry for the delay in getting back to you on this @spricer. I'm not sure I understand your last message. Are you saying that the fix in 7.16 (now released) doesn't work for you? If so, can you share more details of how it fails?