arrow_cast to cast to timestamptz #5914
Conversation
| fn parse_timezone(&mut self, context: &str) -> Result<Option<String>> { | ||
| match self.next_token()? { | ||
| Token::None => Ok(None), | ||
| Token::Some => { | ||
| self.expect_token(Token::LParen)?; | ||
| let timezone = self.parse_double_quoted_string("Timezone")?; | ||
| self.expect_token(Token::RParen)?; | ||
| Ok(Some(timezone)) | ||
| } | ||
| tok => Err(make_error( | ||
| self.val, | ||
| &format!("finding Timezone for {context}, got {tok}"), | ||
| )), | ||
| } | ||
| } |
There was a problem hiding this comment.
add a parser to parse timezone Option<String>
either None, or Some("SomeTimezone")
| /// Parses the next double quoted string | ||
| fn parse_double_quoted_string(&mut self, context: &str) -> Result<String> { | ||
| match self.next_token()? { | ||
| Token::DoubleQuotedString(s) => Ok(s), | ||
| tok => Err(make_error( | ||
| self.val, | ||
| &format!("finding double quoted string for {context}, got '{tok}'"), | ||
| )), | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
add parser to parse the double quoted string e.g. "+00:00"
IMO this behaviour is a bug in arrow-cast that should be fixed, I'm not sure if we wish to expose this. The "correct" behaviour would be |
There was a problem hiding this comment.
Looks great @waitingkuo -- I had some small suggestions on some other tests but we can also add them as a follow on PR
| DataType::Timestamp(TimeUnit::Nanosecond, Some("+08:00".into())), | ||
| DataType::Timestamp(TimeUnit::Microsecond, Some("+08:00".into())), | ||
| DataType::Timestamp(TimeUnit::Millisecond, Some("+08:00".into())), | ||
| DataType::Timestamp(TimeUnit::Second, Some("+08:00".into())), |
There was a problem hiding this comment.
Some other test cases that would be good:
"" (empty timezone)
Also errors:
Three double quotes
"Timestamp(Nanosecond, Some("+00:00""))"
There was a problem hiding this comment.
thanks @alamb , more error handling and test cases added
|
apache/arrow-rs#4034 should fix the timestamp handling in arrow-cast, if we are quick it may still make the release this afternoon |
| r#"Timestamp(Nanosecond, Some(+00:00))"#, | ||
| "Error unrecognized word: +00:00", | ||
| ), | ||
| ( | ||
| r#"Timestamp(Nanosecond, Some("+00:00))"#, | ||
| r#"parsing "+00:00 as double quoted string: last char must be ""#, | ||
| ), | ||
| ( | ||
| r#"Timestamp(Nanosecond, Some(""))"#, | ||
| r#"parsing "" as double quoted string: empty string isn't supported"#, | ||
| ), | ||
| ( | ||
| r#"Timestamp(Nanosecond, Some("+00:00""))"#, | ||
| r#"parsing "+00:00"" as double quoted string: escaped double quote isn't supported"#, |
There was a problem hiding this comment.
@alamb
added error cases for
empty string and
double quoted string that contains double quote
| # once https://github.com/apache/arrow-rs/issues/1936 fixed, please update this query | ||
| query P | ||
| select arrow_cast(timestamp '2000-01-01T00:00:00Z', 'Timestamp(Nanosecond, Some( "+08:00" ))'); | ||
| ---- | ||
| 2000-01-01T08:00:00+08:00 |
There was a problem hiding this comment.
@tustvold thank you
added this case to notify users once the kernel casting issue fixed
| } | ||
| } | ||
|
|
||
| if len == 2 { |
Which issue does this PR close?
Closes #5913
cc @alamb
Rationale for this change
What changes are included in this PR?
allow arrow_cast to cast to timestamptz
Are these changes tested?
yes
Are there any user-facing changes?