New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DS][35/n] Replace asset_subsets_with_metadata with asset_slices_with_metadata #21793
[DS][35/n] Replace asset_subsets_with_metadata with asset_slices_with_metadata #21793
Conversation
4946a2f
to
eb565e8
Compare
@@ -263,9 +265,22 @@ def create( | |||
context: "SchedulingContext", | |||
true_slice: AssetSlice, | |||
subsets_with_metadata: Sequence[AssetSubsetWithMetadata] = [], | |||
slices_with_metadata: Sequence[Tuple[AssetSlice, MetadataMapping]] = [], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should introduce a new object here that capture an AssetSlice enrich with metadata, rather than relying on a tuple.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's add a lightweight abstraction to model this. I think it will be useful. I also want to get a general pattern going for associating arbitrary objects with an AssetSlice, because I believe we will want to do that quite a bit. I'm not saying build a super flexible, generalized abstract now, but a nice baby step is a lightweight object here.
8ac28d7
to
8e6c879
Compare
eb565e8
to
b9a986c
Compare
8e6c879
to
06e29fe
Compare
b9a986c
to
97839e1
Compare
from dagster._model import DagsterModel | ||
from dagster._utils import utc_datetime_from_timestamp | ||
|
||
|
||
class AssetSliceWithMetadata(NamedTuple): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok we can workshop this name later and generalize it but this is a good intermediate step.
06e29fe
to
a4c99d7
Compare
97839e1
to
fc4c2a0
Compare
a4c99d7
to
d8ab103
Compare
fc4c2a0
to
a5d3f72
Compare
d8ab103
to
ded5c44
Compare
a5d3f72
to
407cd58
Compare
ded5c44
to
516f396
Compare
407cd58
to
b6d8453
Compare
Merge activity
|
d91911f
to
4fdd85a
Compare
b6d8453
to
e374d83
Compare
Summary & Motivation
As title -- we want to hide AssetSubset from the internals as much as possible.
How I Tested These Changes