Run cargo build
or cargo build --release
to build in release mode.
Some features take advantage of SSE4.2 instructions, which can be
enabled by adding RUSTFLAGS="-C target-feature=+sse4.2"
before the
cargo build
command.
Run cargo test
for unit tests. To also run tests related to the binaries, use cargo test --features cli
.
The following binaries are provided (use cargo install --features cli
to install them):
-
parquet-schema for printing Parquet file schema and metadata.
Usage: parquet-schema <file-path>
, wherefile-path
is the path to a Parquet file. Use-v/--verbose
flag to print full metadata or schema only (when not specified only schema will be printed). -
parquet-read for reading records from a Parquet file.
Usage: parquet-read <file-path> [num-records]
, wherefile-path
is the path to a Parquet file, andnum-records
is the number of records to read from a file (when not specified all records will be printed). Use-j/--json
to print records in JSON lines format. -
parquet-rowcount for reporting the number of records in one or more Parquet files.
Usage: parquet-rowcount <file-paths>...
, where<file-paths>...
is a space separated list of one or more files to read.
If you see Library not loaded
error, please make sure LD_LIBRARY_PATH
is set properly:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(rustc --print sysroot)/lib
Run cargo bench
for benchmarks.
To build documentation, run cargo doc --no-deps
.
To compile and view in the browser, run cargo doc --no-deps --open
.
To generate the parquet format code run
$ git clone https://github.com/apache/thrift
$ cd thrift
$ git checkout v0.16.0
# docker build just builds a docker image with thrift dependencies
$ docker build -t thrift build/docker/ubuntu-bionic
# build/docker/scripts/cmake.sh actually compiles thrift
$ docker run -v $(pwd):/thrift/src -it thrift build/docker/scripts/cmake.sh && wget https://raw.githubusercontent.com/apache/parquet-format/apache-parquet-format-2.9.0/src/main/thrift/parquet.thrift && ./cmake_build/compiler/cpp/bin/thrift --gen rs parquet.thrift
Then copy the generated parquet.rs
into src/format.rs
and commit changes.