* Initial BaseSpiderMiddleware.
* Rename the new methods.
* Remove the spider argument from new BaseSpiderMiddleware methods.
* Add docs for BaseSpiderMiddleware.
* Silence pylint.
* Add BaseSpiderMiddleware tests.
* Add a release note.
* Enable AsyncioSelectorReactor by default.
* Improve get_crawler(), switch more tests to it.
* Fix the remaining default-reactor test failures.
* Address documentation feedback.
* Make pinned envs more consistent.
* Remove BaseDupeFilter.log()
It is never called because request_seen() always returns False
* Document the interface of DUPEFILTER_CLASS classes
* Remove unnecessary BaseDupeFilter comments and add a short class description
* Improve the documentation related to the DUPEFILTER_CLASS setting
* Deprecate BaseDupeFilter.log
* Update the docs
* Fix the new code example
* Remove typing to keep the example short
Otherwise, it would have required yet another import line (from __future__ or typing).
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* new 'allow_offsite' parameter in OffsiteMiddleware
* document deprecated dont_filter flag in OffsiteMiddleware
* avoid deprecating dont_filter in OffsiteMiddleware
* Copy the code to the downloader mw.
* Add tests for allow_offsite in the downloader mw.
* Mark allow_offsite with reqmeta.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Cover 2.12 in the release notes up to 9bb973dc54
* Add one more highlight
* Better merge of the news entries.
* Cover 2.12 in the release notes up to 642af40.
* Cover 2.12 in the release notes up to 7a0a34b.
* Cover 2.12 in the release notes up to b4bad97.
* Add not yet merged PRs #6463, #6507, #6511 to the 2.12 release notes.
* Cover 2.12 in the release notes up to d85c39f, small fixes.
* Cover 2.12 in the release notes up to d215669.
* Cover #6527 in the release notes.
* Address PR feedback.
* Cover recent PRs.
* Finalize the 2.12.0 release notes, small additional fixes.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Documentation added for Spider State in extensions.rst
* Made correction in documentation for Spiderstate
* Added appropriate intro for Spider state extension
* Added reference for spiderstate extension
* Added Spiderstate extension hyperlink refrence in jobs.rst
Andrey Rakhmatullin
Keval Sakhiya
Adrián Chaves
Mehraz Hossain Rumman
Laerte Pereira
guillermo-bondonno
anubhav
Kevin Lloyd Bernal
Robert Palmer
Rohit Kumar Singh
mmoriniere