Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH] Add support for non-UTC timezone-aware scalars #1193

Open
charlesbluca opened this issue Jun 30, 2023 · 0 comments
Open

[ENH] Add support for non-UTC timezone-aware scalars #1193

charlesbluca opened this issue Jun 30, 2023 · 0 comments
Labels
enhancement New feature or request needs triage Awaiting triage by a dask-sql maintainer

Comments

@charlesbluca
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
While working on #1184, I noticed that we were getting a misleading test pass on test_filter_cast_timestamp, and that currently we don't have support for things like

import pandas as pd
from dask_sql import Context

c = Context()
c.create_table(
    "df",
    pd.DataFrame({
        "d": pd.date_range(start="2014-08-01 09:00", freq="8H", periods=6, tz="Europe/Berlin")
    })
)

c.sql("select * from df where d >= timestamp '2014-08-01 23:00:00+00'")

Because this would require us to support timezone-aware datetime scalars other than UTC, which is currently unsupported.

Describe the solution you'd like
Two potential solutions I think might work here:

  • adding support for non-UTC datetimes on the Python end
  • making modifications to the construction of the logical plan such that in situations like this, non-UTC datetime columns are localized to some common scalar-supported timezone; for example, the offending plan above looks like:
TableScan: df projection=[d], full_filters=[df.d >= TimestampNanosecond(1406934000000000000, Some("Europe/Berlin"))]

But would pass if we were to change it to something like:

TableScan: df projection=[d], full_filters=[CAST(df.d AS Timestamp(Nanosecond, None)) >= TimestampNanosecond(1406934000000000000, None)]
@charlesbluca charlesbluca added enhancement New feature or request needs triage Awaiting triage by a dask-sql maintainer labels Jun 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request needs triage Awaiting triage by a dask-sql maintainer
Projects
None yet
Development

No branches or pull requests

1 participant