2c3ecbff71
fix: missing self argument in ScreenshotPipeline constructor ( #7248 )
...
added missing self argument in __init__ method constructor
2026-02-13 09:42:26 +01:00
fc30c47f38
Add HttpxDownloader. ( #7239 )
2026-02-12 09:30:44 +01:00
66fe5de139
Add Jobdir documentation ( #5260 )
...
* Add Jobdir documentation
* Fix link format
* remove doubtful comments
* some more edits
* Keep the information in jobs.rst an example, and provide details in the reference docs of the corresponding components
* Minor edits
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-06 22:33:02 +04:00
6a42bc6450
Item docs: mention Pydantic support ( #6966 )
...
* Add Pydantic support documentation
* Fix Sphinx references in Pydantic section for non-Sphinx docs
* Minor reformatting
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-05 22:18:01 +04:00
2347138ba4
Unified download handler exceptions ( #7208 )
...
* Add DownloadTimeoutError.
* Don't expect ConnectionAborted in tests.
* Add DownloadCancelledError.
* Add ResponseDataLoss.
* Add more download handler tests.
* Add DownloadConnectionRefusedError.
* Add UnsupportedURLScheme.
* Add CannotResolveHostError.
* Add DownloadFailedError.
* Remove wrapped Twisted exceptions from lists.
* Update a test expectation.
* Extract wrap_twisted_exceptions().
* Wrap TxTimeoutError.
* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
0b9d8da09d
Bump version: 2.14.0 → 2.14.1
2026-01-12 16:25:43 -03:00
c9fbf6c599
Release notes for 2.14.1 ( #7217 )
...
* Release notes for 2.14.1
* Update docs/news.rst
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* Update docs/news.rst
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Update docs/news.rst
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* Update message about github action change
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-01-12 16:23:51 -03:00
3ac1192f35
Bump version: 2.13.4 → 2.14.0
2026-01-05 15:06:13 +05:00
7bef98b4f1
Release notes for 2.14.0 ( #7035 )
...
* Initial release notes for 2.14.0, up to ed63fa9393d715eb49647426aafd1e8de245a7e132
2026-01-05 14:51:05 +05:00
d1bd8eb49f
Async API for download handlers. ( #7164 )
2025-12-31 19:03:16 +01:00
a2463325db
Allow Crawler.crawl_async() without asyncio. ( #7202 )
2025-12-31 18:33:59 +01:00
9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation ( #7198 )
2025-12-22 10:00:39 +01:00
180ca39b23
Deprecate returning Deferreds from pipeline methods ( #7179 )
...
* Add tests for exceptions in pipelines.
* Deprecate returning Deferreds from pipeline process_item().
* Deprecate returning Deferreds from pipeline {open,close}_spider().
* Update the custom pipeline docs.
2025-12-16 13:35:07 +05:00
c49ae2115a
Refactor MediaPipeline.process_item() to async def ( #7177 )
2025-12-15 10:02:04 +01:00
588f3d4f65
Remove docs mentioning deprecated spider attrs. ( #7176 )
2025-12-15 09:51:56 +01:00
d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue ( #6940 )
...
* Change default priority queue to DownloaderAwarePriorityQueue
* Fix documentation building
* Simplify test_start_already_running_exception changes.
* Modernize the test.
* Fix TestEngineCloseSpider.
* Fix typing.
* Remove special slot=None handling.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-11 11:25:17 +01:00
9bfa58e36c
Deprecate send_catch_log_deferred(). ( #7161 )
...
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
2025-12-10 14:42:49 +05:00
5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws ( #7069 )
...
* Restore coverage in MiddlewareManager spider arg handling.
* Move MiddlewareManager._warn_spider_arg() tests.
* Add DownloaderMiddlewareManager.download_async().
* Make download_func async.
* Inline MiddlewareManager._warn_spider_arg().
* Deprecate returning deferreds from downloader middlewares.
* Update the downloader mw doc about async def.
* Use global_object_name() instead of repr().
* Update docs/topics/downloader-middleware.rst
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* Update downloader-middleware.rst
---------
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-12-09 20:07:36 +05:00
0adc561348
Add a list of remaining Deferred APIs. ( #7146 )
2025-11-27 20:13:04 +05:00
4cb0144b39
Merge tag '2.13.4'
...
Bump version: 2.13.3 → 2.13.4
2025-11-17 16:47:54 +05:00
2f62ab532d
Bump version: 2.13.3 → 2.13.4
2025-11-17 16:30:37 +05:00
31a9c03c24
Release notes for 2.13.4. ( #7144 )
...
* Release notes for 2.13.4.
* Formatting.
2025-11-17 16:30:04 +05:00
d091256c58
Remove the deprecated sphinx-hoverxref ( #6922 )
2025-11-14 15:39:21 +05:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
61b4befc60
Remove code deprecated in 2.12.0 ( #7126 )
...
* Remove smaller things.
* Remove from_settings().
* Remove old MediaPipeline init code.
* Cleanup.
* Typo.
2025-10-27 17:14:38 +05:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
eb496470f1
Add missing doc deps for new examples ( #7116 )
2025-10-24 11:58:18 +02:00
a5bbeb2586
Expanded documentation for issue #6676 ( #6957 )
2025-10-24 10:30:33 +02:00
2d073a9c0d
Fix typo in logging documentation ( #7109 )
2025-10-20 15:52:46 +02:00
b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. ( #7079 )
2025-10-16 13:15:07 +02:00
a02abdcf63
Fix typos in documentation ( #7076 )
2025-10-06 11:43:36 +02:00
1c6ba00fd0
Move LogCounterHandler into a new extension ( #7046 )
...
* Move LogCounterHandler into Crawler.crawl().
* Add the LogCount extension.
* Update test_spider_custom_settings_log_level().
* Typo.
* Update docs.
2025-10-03 20:14:11 +05:00
f2531808f3
docs: Add Edit on GitHub ( #7058 )
2025-09-28 13:16:47 +05:00
2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. ( #7037 )
...
* Deprecate the spider arg of downloader middleware methods.
* Add a test for deprecated spider args in downloader mws.
2025-09-02 21:51:47 +05:00
ed63fa94d6
Deprecate spider args of middlewares and pipelines. ( #7006 )
...
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
b68330811b
Deprecate spider arguments of StatsCollector methods ( #7011 )
2025-08-20 09:52:22 +02:00
edba1ad572
Refactor RobotsTxtMiddleware to use async def process_request(). ( #6802 )
2025-08-18 18:40:46 +02:00
89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() ( #6842 )
...
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
d9ef0350d8
Merge remote-tracking branch 'origin/2.13'
2025-07-02 20:46:02 +05:00
155a504f24
Bump version: 2.13.2 → 2.13.3
2025-07-02 20:39:46 +05:00
cf465bf644
Release notes for 2.13.3. ( #6934 )
...
* Release notes for 2.13.3.
* Address feedback.
2025-07-02 20:36:01 +05:00
dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting ( #6921 )
...
* Deprecate CONCURRENT_REQUESTS_PER_IP setting
* Typing fix
* PR review
* tests
* oops
* Update scrapy/settings/default_settings.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* update logic
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-06-30 16:51:05 +05:00
3843091c5f
Mark abstract base classes as such. ( #6930 )
...
* Mark abstract base classes as such.
* Remove an unneeded method.
* Fix exporter test coverage.
2025-06-30 15:14:34 +05:00
03fe7a6424
Add a deprecation notice for the offsite spider middleware.
2025-06-29 02:35:48 +05:00
2c1c10e923
Remove deprecated offsite spider middleware. ( #6926 )
...
* Remove deprecated offsite spider middleware.
* Add a deprecation notice for the offsite spider middleware.
2025-06-29 02:35:20 +05:00
ba1cedee1b
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:13:54 +05:00
3019393686
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:04:40 +05:00
1473f4d347
Remove the deprecated sphinx-hoverxref ( #6922 )
2025-06-28 18:04:06 +05:00
95172659af
Add sphinx-lint. ( #6920 )
2025-06-28 01:37:20 +02:00
Anant Murmu
Andrey Rakhmatullin
Martin Schimandl
Michael
Eugenio Lacuesta
Eugenio Lacuesta
Adham 3llam
Thalison Fernandes
Adrián Chaves
sstegmanbsd
Sebastien Dionne
Laerte Pereira
Felipe Benevolo