2.4.0

Hihglights:

• Python 3.5 support has been dropped.

• The file_path method of media pipelines can now access the source item.

  This allows you to set a download file path based on item data.

• The new item_export_kwargs key of the FEEDS setting allows to define keyword parameters to pass to item exporter classes.

• You can now choose whether feed exports overwrite or append to the output file.

  For example, when using the crawl or runspider commands, you can use the -O option instead of -o to overwrite the output file.

• Zstd-compressed responses are now supported if zstandard is installed.

• In settings, where the import path of a class is required, it is now possible to pass a class object instead.

See the full changelog

2.3.0

Hihglights:

• Feed exports now support Google Cloud Storage as a storage backend

• The new FEED_EXPORT_BATCH_ITEM_COUNT setting allows to deliver output items in batches of up to the specified number of items.

  It also serves as a workaround for delayed file delivery, which causes Scrapy to only start item delivery after the crawl has finished when using certain storage backends (S3, FTP, and now GCS).

• The base implementation of item loaders has been moved into a separate library, itemloaders, allowing usage from outside Scrapy and a separate release schedule

See the full changelog

2.2.1

The startproject command no longer makes unintended changes to the permissions of files in the destination folder, such as removing execution permissions.

2.2.0

Highlights:

• Python 3.5.2+ is required now
• dataclass objects and attrs objects are now valid item types
• New TextResponse.json method
• New bytes_received signal that allows canceling response download
• CookiesMiddleware fixes

See the full changelog

2.1.0

Highlights:

• New FEEDS setting to export to multiple feeds
• New Response.ip_address attribute

See the full changelog

2.0.1

• Response.follow_all now supports an empty URL iterable as input (#4408, #4420)
• Removed top-level reactor imports to prevent errors about the wrong Twisted reactor being installed when setting a different Twisted reactor using TWISTED_REACTOR (#4401, #4406)

2.0.0

Highlights:

• Python 2 support has been removed
• Partial coroutine syntax support and experimental asyncio support
• New Response.follow_all method
• FTP support for media pipelines
• New Response.certificate attribute
• IPv6 support through DNS_RESOLVER

See the full changelog

1.7.4

Revert the fix for #3804 (#3819), which has a few undesired side effects (#3897, #3976).

1.7.3

Enforce lxml 4.3.5 or lower for Python 3.4 (#3912, #3918)

1.7.2

Fix Python 2 support (#3889, #3893, #3896)