Use new Linux 5.12 openat2 with RESOLVE_CACHED flag

[wmanley]

Is your feature request related to a problem? Please describe.

We don't know whether opening a file will block or not, so we must hand it off to a thread-pool. This is inefficient if opening it wouldn't have required blocking.

Describe the solution you'd like

Try opening a file using openat2 passing the RESOLVE_CACHED flag only handing it off to the threadpool if it would block or if the flag is not supported. This is available on the recently released Linux 5.12. See LWN: https://lwn.net/Articles/843163/ .

Describe alternatives you've considered

Alternatively users could use the openat2 library directly like:

let res = openat2(None, path, OpenHow {flags: O_CLOEXEC | ..., 0, ResolveFlags::CACHED});
let err = match res {
    Ok(fd) => Ok(unsafe {File::from_raw_fd(fd)}),
    Err(e) => tokio::fs::File::open(path).await,
}

But this requires specific handling by the user and requires unsafe.

Additional context

This would be to open what #3518 was for read.

It looks like it might be tricky to do. Tokio's File::open wraps std::File::open which takes a tokio::OpenOptions. This in turn wraps std::OpenOptions, but std::OpenOptions doesn't provide any getters, only setters, so we can't get the flags and mode out to pass to openat2. We also can't change tokio::OpenOptions to not be implemented in terms of std::OpenOptions because it implements From<std::OpenOptions>, and the only thing you can do is store that std::OpenOptions to be used later.

Options:

1. Add accessors to std::fs::OpenOptions
2. Add openat2 support into std directly by enhancing std::OpenOptions
3. Add an additional open API to tokio that doesn't use OpenOptions.

[Darksonn]

I mean, the only place where we can't do this is if the user created the OpenOptions through the From impl. In any other case, we can keep track of the options ourselves.

[wmanley]

Good point. It would involve duplicating OpenOptions from std, but that's fine. We wouldn't have to add any additional public API, so it'd be entirely an implementation detail.