Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disallow cast from other datatypes to NullType #1923

Closed
viirya opened this issue Jun 22, 2022 · 2 comments · Fixed by #1942
Closed

Disallow cast from other datatypes to NullType #1923

viirya opened this issue Jun 22, 2022 · 2 comments · Fixed by #1942
Assignees
Labels
arrow Changes to the arrow crate bug

Comments

@viirya
Copy link
Member

viirya commented Jun 22, 2022

Describe the bug

We allow casting from other datatypes to NullType (see can_cast_types). I suspect that this is a mistake.

For example, Spark doesn't allow casting from any datatypes to NullType.

I also checked C++ Cast kernels. Mostly the datatypes cannot cast to NullType (e.g. https://github.com/apache/arrow/blob/53752adc6b81166cd4ee7db5a819494042f29197/cpp/src/arrow/compute/kernels/scalar_cast_test.cc#L187-L211). I said "mostly" because I don't see explicit tests for other cases like Decimal, but I suspect that it is not allowed but just lack of test coverage.

I don't see there is cast kernel in C++ for casting to NullType except for a special case (dict<null> -> null): https://github.com/apache/arrow/blob/53752adc6b81166cd4ee7db5a819494042f29197/cpp/src/arrow/compute/kernels/scalar_cast_numeric.cc#L748-L751. Even for that, there is a comment questioning about it.

To Reproduce
Steps to reproduce the behavior:

Expected behavior

Cast kernel should not allow casting from any datatypes to NullType.

Additional context
Add any other context about the problem here.

@liukun4515
Copy link
Contributor

The result of discussion: #1922 (comment)

@liukun4515
Copy link
Contributor

I will give a follow-up pr to remove casting other data type to Null Data type.

@alamb alamb changed the title Disallow cast from other datatypes to NullType Disallow cast from other datatypes to NullType Jul 7, 2022
@alamb alamb added the arrow Changes to the arrow crate label Jul 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants