feat: {Field,DataType}::size
#3149
Conversation
github-actions
bot
added
arrow
Add a way to calculate in-memory size of `Field` and `DataType`. Closes apache#3147.
crepererum
force-pushed
the
crepererum/issue3147
branch
from
9cd5a23
to
f86c9e1
Compare
| std::mem::size_of_val(self) - std::mem::size_of_val(&self.data_type) | ||
| + self.data_type.size() | ||
| + self.name.capacity() | ||
| + (std::mem::size_of::<(String, String)>() * self.metadata.capacity()) |
There was a problem hiding this comment.
I'm not sure how accurate this approximation is, but I imagine probably good enough to a first order
|
Benchmark runs are scheduled for baseline = f091cbb and contender = a110004. a110004 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Closes #3147.
Rationale for this change
In DataFusion, it would be nice to know how much data is allocated so we can bail out early instead of OOMing (a slight over-allocation is OK, so this can be measured after the fact). For that purpose, it would be nice to know how much memory a specific instance of
Field/DataTyperequires.What changes are included in this PR?
DataType::sizeandSchema::size.Are there any user-facing changes?
New methods, not breaking.