Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

in UI, populate asset event "Source data" using event tags #21602

Open
sryza opened this issue May 2, 2024 · 1 comment
Open

in UI, populate asset event "Source data" using event tags #21602

sryza opened this issue May 2, 2024 · 1 comment

Comments

@sryza
Copy link
Contributor

sryza commented May 2, 2024

On the "Events" tab of the asset details page, each event has a "Source data" section which contains links to upstream materializations that the event is based on.

These links are populated using a heuristic that can be inaccurate and also deviate from the materialization lineage information that we include in asset event tags.

The heuristic: look at the timestamp of the event and find the latest materialization of the upstream asset that's before that timestamp. One way this can be inaccurate is if the upstream asset was re-materialized after the downstream asset started the process of being materialized.

Events have a tag prefixed with "input_event_pointer" that encodes Dagster's best understanding of the upstream event they're derived from.

image
@sryza
Copy link
Contributor Author

sryza commented May 2, 2024

Related issues:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant