Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add API to Retrieve Finished Writer from Parquet Writer #2498

Merged
merged 3 commits into from Aug 18, 2022
Merged

Add API to Retrieve Finished Writer from Parquet Writer #2498

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
25aceeb
add into_inner to take inner writer out
jiacai2050 Aug 18, 2022
a0cb64a
flush writer before into_inner
jiacai2050 Aug 18, 2022
5d5fba2
Apply suggestions from code review
tustvold Aug 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions parquet/src/arrow/arrow_writer/mod.rs
View file
Open in desktop
Expand Up @@ -223,6 +223,11 @@ impl<W: Write> ArrowWriter<W> {
Ok(())
}

/// Returns the underlying writer.
tustvold marked this conversation as resolved.
Show resolved Hide resolved
pub fn into_inner(self) -> W {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should either call flush, or mention the need for this in the doc comment. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reminding.

At first, I think caller should call flush before into_inner, but since this method consume ownership, so it' best we call flush inside it just like what close does.

self.writer.into_inner()
}

/// Close and finalize the underlying Parquet writer
pub fn close(mut self) -> Result<parquet_format::FileMetaData> {
self.flush()?;
Expand Down
10 changes: 10 additions & 0 deletions parquet/src/file/writer.rs
View file
Open in desktop
Expand Up @@ -60,6 +60,11 @@ impl<W: Write> TrackedWrite<W> {
pub fn bytes_written(&self) -> usize {
self.bytes_written
}

/// Returns the underlying writer.
pub fn into_inner(self) -> W {
self.inner
}
}

impl<W: Write> Write for TrackedWrite<W> {
Expand Down Expand Up @@ -306,6 +311,11 @@ impl<W: Write> SerializedFileWriter<W> {
Ok(())
}
}

/// Returns the underlying writer.
tustvold marked this conversation as resolved.
Show resolved Hide resolved
pub fn into_inner(self) -> W {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs to call self.write_metadata like close does, otherwise the file will not have a footer

self.buf.into_inner()
}
}

/// Parquet row group writer API.
Expand Down