15885a8db4
Generalize the use of build_from_crawler() internally ( #7808 )
...
* Generalize the use of build_from_crawler() internally
* Do not use build_from_crawler() for spiders, since they are guaranteed having from_crawler()
2026-08-10 20:55:10 +05:00
d41bfaec05
Document how to disallow subdomains of allowed domains ( #7903 )
2026-08-09 23:05:43 +05:00
0e324f3d4a
Let spiders change allowed_domains at run time ( #7912 )
2026-08-09 14:17:44 +05:00
2b2e18199b
Type downloader middleware tests ( #7858 )
2026-08-03 15:49:15 +05:00
25e6884e2f
Address recursion and media ignore-request reporting issues ( #7673 )
...
* Address recursion and media ignore-request reporting issues
* Improve test coverage
* Address feedback
2026-07-27 20:34:10 +05:00
dd4549e6f9
Improve test coverage for downloader middlewares ( #7655 )
...
* Improve test coverage for downloader middlewares
* Improve coverage further
2026-06-25 01:48:28 +05:00
f605defefc
Document scrapy-lint, remove start_url check ( #7627 )
...
* Document scrapy-lint, remove start_url check
* Remove the offsite URL check in favor of scrapy-lint
2026-06-22 20:12:16 +05:00
2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. ( #7037 )
...
* Deprecate the spider arg of downloader middleware methods.
* Add a test for deprecated spider args in downloader mws.
2025-09-02 21:51:47 +05:00
a898331d14
Preparations for switching to direct pytest. ( #6678 )
2025-02-16 20:28:58 +01:00
1fc91bb462
new `allow_offsite` parameter in OffsiteMiddleware ( #6151 )
...
* new 'allow_offsite' parameter in OffsiteMiddleware
* document deprecated dont_filter flag in OffsiteMiddleware
* avoid deprecating dont_filter in OffsiteMiddleware
* Copy the code to the downloader mw.
* Add tests for allow_offsite in the downloader mw.
* Mark allow_offsite with reqmeta.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-01-08 21:28:51 +05:00
04d0411bf7
Filter test-time warnings. ( #6501 )
2024-10-21 12:30:49 +02:00
d2f1e00a6a
Merge 2.11.2 changes ( #6363 )
2024-05-14 18:54:11 +02:00
Adrian
Andrey Rakhmatullin
Kevin Lloyd Bernal
Adrián Chaves