fsync/fdatasync running on main thread

[jannes]

I recently wrote a few scripts to benchmark different sizes for the tokio pool
that is used for running blocking disk IO calls such as write, read.
I noticed that fsync/fdatasync system calls are made from the main thread.

As these are blocking calls, I believe they should also be executed by the Tokio pool.
Otherwise for workloads that make heavy use of these system calls
the main thread will be constantly blocked.

[bartlomieju]

CC @caspervonb

Related #7400
Blocked by #7984

[caspervonb]

Kind-of blocked on #8416 (altho not impossible, tedious to make this async with the current resource table).

[bartlomieju]

Let's continue this topic as part of #7400