Fix parsing null counts for struct type columns in the struct stats #714
+39
−13
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When reading the struct column stats (as opposed to json column stats) the null counts where ignored for struct type columns. Completely ignoring is probably less bad than parsing them wrongly but if the user was filtering on a struct column this may have impacted performance. It was also spamming error logs:
I probably should have done this as part of #656. Sorry for introducing this slight regression. I think some logging was getting suppressed during the unittest so I didn't notice it was an issue.
Related Issue(s)
Relates to #653 but that was for the most part already solved by #656
Changes