Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage/mysql: add option for interpreting zero datetime values #27028

Open
Tracked by #15901
chuck-alt-delete opened this issue May 10, 2024 · 1 comment
Open
Tracked by #15901
Labels
C-feature Category: new feature or request T-mysql Theme: MySQL source

Comments

@chuck-alt-delete
Copy link
Contributor

chuck-alt-delete commented May 10, 2024

Feature request

Unfortunately, MySQL has a date 0000-00-00. This affects the Date type as well as the timestamp type. This 0 date is absurd because it never actually happened. It seems to me users are really trying to use NULL. Maybe this allows them to avoid some COALESCE statement down the line in some query, I don’t know. But the fact remains people use this absurd value for timestamps, and it is impractical to update production tables with hundreds of millions of rows to fix it on the MySQL side.

The current best approach is to ingest these as text columns and do casting in a CASE WHEN, something like this (using the currently hypothetical try_parse_mysql_timestamp)

case
  when my_timestamp like '0000-00-00%'
    then null
  else
    try_parse_mysql_timestamp(my_timestamp)

Timestamps are very important in materialize. They are used for temporal filters with mz_now(), as well as for predicate pushdown. It’s a little bit of a rough user experience to require this kind of parsing for all MySQL users who use timestamps (aka all MySQL users) and I suspect this rake will be stepped on frequently.

I would prefer an option in the MySQL source that allows users to configure how to interpret the 0 date/timestamp. By default, I think it should be interpreted as NULL, but optionally users can specify some valid date.

@chuck-alt-delete chuck-alt-delete added the C-feature Category: new feature or request label May 10, 2024
@benesch
Copy link
Member

benesch commented May 10, 2024

Recording my suggestion on syntax from Slack.

Could we add an option like MAP ZERO DATE VALUES = {NULL | }, where users can explicitly specify a value to use for any zero dates?

I think the name needs some workshopping, but that's the gist.

@morsapaes morsapaes changed the title Add option for how to interpret 0000-00-00 Date in MySQL Source storage/mysql: add option for interpreting zero datetime values May 16, 2024
@morsapaes morsapaes added the T-mysql Theme: MySQL source label May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature Category: new feature or request T-mysql Theme: MySQL source
Projects
None yet
Development

No branches or pull requests

3 participants