from __future__ import annotations causes flow initialization to fail with issubclass arg must be a class
#7502
Labels
bug
Something isn't working
great writeup
This is a wonderful example of our standards
needs:research
Blocked by investigation into feasibility and cause
orchestration
Related to Orchestration features
status:accepted
We may work on this; we will accept work from external contributors
Comments
zanieb
added
status:upstream
|
Thanks for the issue! I originally thought this was an upstream issue (see pydantic/pydantic#2678) but with further thought this does look to be caused by the way we are reading type hints to generate a model. We'll probably need to look into how Pydantic solves this problem, which is perhaps discussed in the linked issue. |
|
I ran into this exact same issue yesterday which causes me a lot of debugging head-scratching :). This also seems to happen with other types, e.g. Interestingly, this bug seemed to have caused my flow orchestration to go haywire, as my flow runs were crashing with an infrastructure failure and all I could see was: Prefect Prefect which doesn't seem to have anything to do with the issue at all (and was quite hard to debug)! I'm not a Python guru, but this smells like it might have something to do with dynamic imports... |
|
Hm I thought perhaps you were encountering #8688 but a fix is out in 2.10.7 |
zanieb
added
needs:research
|
@zanieb: I have some more details. This is still a significant issue. In Prefect 2.14.3, Pydantic 2.4.2, Python 3.10.13 run the following: from __future__ import annotations
from prefect import flow
class Test:
pass
@flow
def foo(x: Test):
print(x)This yields a similar error as originally reported. Removing the TypeError Traceback (most recent call last)
Cell In[2], line 7
3 class Test:
4 pass
6 @flow
----> 7 def foo(x: Test):
8 print(x)
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/prefect/flows.py:1339, in flow(__fn, name, version, flow_run_name, retries, retry_delay_seconds, task_runner, description, timeout_seconds, validate_parameters, persist_result, result_storage, result_serializer, cache_result_in_memory, log_prints, on_completion, on_failure, on_cancellation, on_crashed)
1237 """
1238 Decorator to designate a function as a Prefect workflow.
1239
(...)
1334 >>> pass
1335 """
1336 if __fn:
1337 return cast(
1338 Flow[P, R],
-> 1339 Flow(
1340 fn=__fn,
1341 name=name,
1342 version=version,
1343 flow_run_name=flow_run_name,
1344 task_runner=task_runner,
1345 description=description,
1346 timeout_seconds=timeout_seconds,
1347 validate_parameters=validate_parameters,
1348 retries=retries,
1349 retry_delay_seconds=retry_delay_seconds,
1350 persist_result=persist_result,
1351 result_storage=result_storage,
1352 result_serializer=result_serializer,
1353 cache_result_in_memory=cache_result_in_memory,
1354 log_prints=log_prints,
1355 on_completion=on_completion,
1356 on_failure=on_failure,
1357 on_cancellation=on_cancellation,
1358 on_crashed=on_crashed,
1359 ),
1360 )
1361 else:
1362 return cast(
1363 Callable[[Callable[P, R]], Flow[P, R]],
1364 partial(
(...)
1384 ),
1385 )
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/prefect/context.py:185, in PrefectObjectRegistry.register_instances.<locals>.__register_init__(__self__, *args, **kwargs)
183 registry = cls.get()
184 try:
--> 185 __init__(__self__, *args, **kwargs)
186 except Exception as exc:
187 if not registry or not registry.capture_failures:
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/prefect/flows.py:299, in Flow.__init__(self, fn, name, version, flow_run_name, retries, retry_delay_seconds, task_runner, description, timeout_seconds, validate_parameters, persist_result, result_storage, result_serializer, cache_result_in_memory, log_prints, on_completion, on_failure, on_cancellation, on_crashed)
289 self.retries = (
290 retries if retries is not None else PREFECT_FLOW_DEFAULT_RETRIES.value()
291 )
293 self.retry_delay_seconds = (
294 retry_delay_seconds
295 if retry_delay_seconds is not None
296 else PREFECT_FLOW_DEFAULT_RETRY_DELAY_SECONDS.value()
297 )
--> 299 self.parameters = parameter_schema(self.fn)
300 self.should_validate_parameters = validate_parameters
302 if self.should_validate_parameters:
303 # Try to create the validated function now so that incompatibility can be
304 # raised at declaration time rather than at runtime
305 # We cannot, however, store the validated function on the flow because it
306 # is not picklable in some environments
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/prefect/utilities/callables.py:336, in parameter_schema(fn)
333 # Generate a Pydantic model at each step so we can check if this parameter
334 # type supports schema generation
335 try:
--> 336 create_schema(
337 "CheckParameter", model_cfg=ModelConfig, **{name: (type_, field)}
338 )
339 except ValueError:
340 # This field's type is not valid for schema creation, update it to `Any`
341 type_ = Any
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/prefect/utilities/callables.py:296, in create_v1_schema(name_, model_cfg, **model_fields)
292 def create_v1_schema(name_: str, model_cfg, **model_fields):
293 model: "pydantic.BaseModel" = pydantic.create_model(
294 name_, __config__=model_cfg, **model_fields
295 )
--> 296 return model.schema(by_alias=True)
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/pydantic/v1/main.py:664, in BaseModel.schema(cls, by_alias, ref_template)
662 if cached is not None:
663 return cached
--> 664 s = model_schema(cls, by_alias=by_alias, ref_template=ref_template)
665 cls.__schema_cache__[(by_alias, ref_template)] = s
666 return s
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/pydantic/v1/schema.py:188, in model_schema(model, by_alias, ref_prefix, ref_template)
186 model_name_map = get_model_name_map(flat_models)
187 model_name = model_name_map[model]
--> 188 m_schema, m_definitions, nested_models = model_process_schema(
189 model, by_alias=by_alias, model_name_map=model_name_map, ref_prefix=ref_prefix, ref_template=ref_template
190 )
191 if model_name in nested_models:
192 # model_name is in Nested models, it has circular references
193 m_definitions[model_name] = m_schema
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/pydantic/v1/schema.py:582, in model_process_schema(model, by_alias, model_name_map, ref_prefix, ref_template, known_models, field)
580 s['description'] = doc
581 known_models.add(model)
--> 582 m_schema, m_definitions, nested_models = model_type_schema(
583 model,
584 by_alias=by_alias,
585 model_name_map=model_name_map,
586 ref_prefix=ref_prefix,
587 ref_template=ref_template,
588 known_models=known_models,
589 )
590 s.update(m_schema)
591 schema_extra = model.__config__.schema_extra
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/pydantic/v1/schema.py:623, in model_type_schema(model, by_alias, model_name_map, ref_template, ref_prefix, known_models)
621 for k, f in model.__fields__.items():
622 try:
--> 623 f_schema, f_definitions, f_nested_models = field_schema(
624 f,
625 by_alias=by_alias,
626 model_name_map=model_name_map,
627 ref_prefix=ref_prefix,
628 ref_template=ref_template,
629 known_models=known_models,
630 )
631 except SkipField as skip:
632 warnings.warn(skip.message, UserWarning)
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/pydantic/v1/schema.py:256, in field_schema(field, by_alias, model_name_map, ref_prefix, ref_template, known_models)
253 s.update(validation_schema)
254 schema_overrides = True
--> 256 f_schema, f_definitions, f_nested_models = field_type_schema(
257 field,
258 by_alias=by_alias,
259 model_name_map=model_name_map,
260 schema_overrides=schema_overrides,
261 ref_prefix=ref_prefix,
262 ref_template=ref_template,
263 known_models=known_models or set(),
264 )
266 # $ref will only be returned when there are no schema_overrides
267 if '$ref' in f_schema:
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/pydantic/v1/schema.py:528, in field_type_schema(field, by_alias, model_name_map, ref_template, schema_overrides, ref_prefix, known_models)
526 else:
527 assert field.shape in {SHAPE_SINGLETON, SHAPE_GENERIC}, field.shape
--> 528 f_schema, f_definitions, f_nested_models = field_singleton_schema(
529 field,
530 by_alias=by_alias,
531 model_name_map=model_name_map,
532 schema_overrides=schema_overrides,
533 ref_prefix=ref_prefix,
534 ref_template=ref_template,
535 known_models=known_models,
536 )
537 definitions.update(f_definitions)
538 nested_models.update(f_nested_models)
File ~/software/miniconda/envs/prefect/lib/python3.10/site-packages/pydantic/v1/schema.py:927, in field_singleton_schema(field, by_alias, model_name_map, ref_template, schema_overrides, ref_prefix, known_models)
924 if lenient_issubclass(getattr(field_type, '__pydantic_model__', None), BaseModel):
925 field_type = field_type.__pydantic_model__
--> 927 if issubclass(field_type, BaseModel):
928 model_name = model_name_map[field_type]
929 if field_type not in known_models:
File ~/software/miniconda/envs/prefect/lib/python3.10/abc.py:123, in ABCMeta.__subclasscheck__(cls, subclass)
121 def __subclasscheck__(cls, subclass):
122 """Override for issubclass(subclass, cls)."""
--> 123 return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class |
|
Thanks for the details Andrew — I'm not working on this project anymore perhaps @desertaxle can help instead. |
|
Sorry about that! Hope you're enjoying the new adventures :) @desertaxle, if you need any additional details, don't hesitate to ping me. |
|
When the The two ways of parsing the parameters (for v1 or v2 of pydantic) are here and here Instead of returning |
|
@jamesdow21: Thanks, that is a great observation!! Are you interested in opening a PR for this? If not, I'll try to give it a go when I have some time since this is an important one for me (I'm not quite sure why more Prefect users aren't running into this...). |
I can give it a shot in a couple days when I'll have some free time. I'm also surprised that this isn't occurring more commonly, I always default to using the PEP563 annotations (at least until my minimum version will be 3.13 with PEP649 style annotations) |
|
If we don't want to copy source code, I suppose we could also just do a check if Python 3.10 is used and then call the |
This was referenced Jan 7, 2024
|
@jamesdow21 -- FYI, I opened a PR in #11578. |
This was referenced Jan 10, 2024
6 tasks
Andrew-S-Rosen
added a commit
to Quantum-Accelerators/quacc
that referenced
this issue
Jan 11, 2024
## Summary of Changes Closes #1382. TODO: - The `@subflow` decorator should ideally be doing a `gather`-type operation like in Dask so nobody is iterating through a list to resolve futures. - Add documentation. REQUIRES: - PrefectHQ/prefect#7502 - PrefectHQ/prefect#11616 --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
First check
Bug summary
When using
from __future__ import annotationsflow initialization fails onpydantic.create_modelwith messageissubclass() arg must be a class.Reproduction
Error
Versions
Additional context
This is caused by PEP 563, specifically:
This PEP proposes changing function annotations and variable annotations so that they are no longer evaluated at function definition time. Instead, they are preserved in __annotations__ in string form.When using this import statement the value of
type_in the below statement inparameter_schema.pyis a string with the value'date'.If I manually change the value of
type_todate(the class), then no error is raised. I don't have enough familiarity withtyping.get_type_hintsto provide suggestions, but I did confirm that if I usetyping.get_type_hints(fn)I do get thedateclass, which we could then pass topydantic.create_model.The text was updated successfully, but these errors were encountered: