type hints wanted!

[xdewx]

Summary

scrapy.Item is just like a placeholder, neither item.field=value nor v=item.field is supported.

It would be nice if scrapy.Item is based on pydantic.BaseModel.

Motivation

For nice coding experience.

[wRAR]

You can already use pydantic (or other libraries) via ItemAdapter.

[xdewx]

You can already use pydantic (or other libraries) via ItemAdapter.

Thank you very much. However, i am curious that why need scrapy.Item? It seems that use pydantic.BaseModel directly is totally fine, then there is no need to use ItemAdapter.

[wRAR]

I don't think changing scrapy.Item to make it a pydantic.BaseModel is backwards-compatible.

[xdewx]

I don't think changing scrapy.Item to make it a pydantic.BaseModel is backwards-compatible.

Oh ,i see. That's to say it is feasible to declare pydantic model in items.py and yield it from Spider at present, but maybe fail in the future.

And I would suggest that :

1. scrapy could generate code with extra option for example, --item-class=[pydantic|scapy|dataclass].
2. code generated by scrapy can have typing hits, for example:

class XxxxSpider(Spider):
    def parse(self,response: Response)->Generator:
        pass

[Gallaecio]

Adding typing hints to Scrapy is an on-going effort.

[felipeboffnunes]

@xdewx may not be the best solution, but I have done a little of work using Pydantic on Scrapy to validate this idea scrapy-pydantic-loader-poc it isn't perfect but I have used this here and there