Support Direct I/O

[tabokie]

Simple benchmark shows more than 20% throughput improvement from direct I/O.

This task is preferred to be stacked over #165.

[sticnarf]

Besides direct I/O, is there plan to use Linux AIO and provide asynchronous read and write interfaces in the near future?

[tabokie]

No plan yet. But I think it would be trivial to implement once this task (and parallel writing) is completed.

[sticnarf]

For direct I/O, I think one thing we should do before GA is change the file header to occupy the whole 4K block. I think it will be helpful to forward compatibility, so we don't need a format v2 to support direct I/O.

[tabokie]

@sticnarf Sadly, we will default enable it for TiDB Cloud dev tier. Can't change it now.

[LykxSassinator]

Unfortunately, while this enhancement may yield positive results in other scenarios, its impact on TiKV is not as significant.

Based on practical statistics gathered from TiKV across various workloads such as sysbench, tpcc, and ycsb, it has been observed that the typical size during the flushing of raft logs is less than 8kb/s. Consequently, the benefits of introducing DIO are not as impressive as initially expected.

workloads	Bytes per second	pic
tpcc-1k	avg 3KB/s (P99 < 16KB/s)