Inline Generated Thift Code Into Parquet Crate

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently parquet-format, which contains the generated code from the parquet thrift definitions, is published as a separate crate from a separate repo

There is discussion about potentially donating parquet-format and I wanted to create an issue to discuss this

Describe the solution you'd like

It's necessarily an incomplete picture, but looking at the reverse dependencies of parquet-format they all already have a dependency on parquet. As such I would like to propose inlining the generated code in a format module within parquet. This would avoid maintenance burden associated with an additional crate

Describe alternatives you've considered

We could not do this or we could donate the code, but still vend it as a separate crate. This would not address the maintenance burden of additional dependencies, but perhaps updates are infrequent enough that this doesn't matter

Additional context

If we are happy to proceed with this, I would be willing to help with the mechanics of moving the code across

FYI @sunchao @nevi-me

[nevi-me]

It's necessarily an incomplete picture, but looking at the reverse dependencies of parquet-format they all already have a dependency on parquet. As such I would like to propose inlining the generated code in a format module within parquet. This would avoid maintenance burden associated with an additional crate

We've had this discussion before, and I support this approach (sunchao/parquet-format-rs#10).

I think what could also bolster the argument for making parquet::format is that in reality we (I) haven't seen people depend on a specific version of parquet-format. So whatever the latest version we support, should be authoritative. In the sense that it'd be lower maintenance if we generate the thrift code whenever there's a change (which is very infrequent), and expose that as part of parquet.

@sunchao @tustvold I'd be in favour of deprecating and eventually archiving parquet-format-rs, then generating the code here. thrift-rs has come a long way since the early days, so it should be easier for external users to build their own format file if they don't want to depend on parquet::format.

[alamb]

I also agree that incorporating the code generated from thrift into the parquet crate and removing the dependency on parquet-format would be a good idea because:

1. I think it would easier to use for users (can't have a version mismatch with parquet and another crate)
2. The generated thrift code seems like an implementation detail.

[sunchao]

+1. Let me know if you need any help.

I'd be in favour of deprecating and eventually archiving parquet-format-rs, then generating the code here.

I'd prefer to do the deprecating & archiving later, since we have internal projects that are still using parquet-format-rs.

[alamb]

Automatically added labels {'api-change'} from #2626