0e1526ed30
Fix flaky test_download_with_proxy_https_timeout(). ( #7250 )
2026-02-13 09:45:09 +01:00
2c3ecbff71
fix: missing self argument in ScreenshotPipeline constructor ( #7248 )
...
added missing self argument in __init__ method constructor
2026-02-13 09:42:26 +01:00
fc30c47f38
Add HttpxDownloader. ( #7239 )
2026-02-12 09:30:44 +01:00
0cfc4e4386
Improve dont_filter documentation ( #7245 )
2026-02-10 11:49:10 +03:00
06fb87f7bb
Fix override behavior in getwithbase() issue #6912 ( #6993 )
...
* Fix test for getwithbase() to ensure class-keyed overrides with None are handled correctly
* Test actual import paths from default_settings
* Improvements
* Remove unused logger
* Run pre-commit
* Restore and improve duplicate key handling and warning
* type → object
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-07 00:17:55 +04:00
66fe5de139
Add Jobdir documentation ( #5260 )
...
* Add Jobdir documentation
* Fix link format
* remove doubtful comments
* some more edits
* Keep the information in jobs.rst an example, and provide details in the reference docs of the corresponding components
* Minor edits
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-06 22:33:02 +04:00
16929c0991
Exclude deprecated code from coverage. ( #7241 )
2026-02-06 09:15:07 +01:00
6a42bc6450
Item docs: mention Pydantic support ( #6966 )
...
* Add Pydantic support documentation
* Fix Sphinx references in Pydantic section for non-Sphinx docs
* Minor reformatting
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-05 22:18:01 +04:00
294ee051cc
Reducing memory footprint of `Request` object: __slots__ and lazy evaluation ( #7036 )
...
* reducing memory foorprint of `Request` object
* restore flow
* Test setters
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-05 22:15:53 +04:00
8974580e43
Reword the release note entry to consider the 301 redirect fix a security bug fix
2026-02-03 20:38:11 +01:00
2e53d90e4c
Refactor HTTP download handlers ( #7238 )
2026-02-03 20:13:30 +01:00
11977afba5
Rename async test decorators. ( #7234 )
2026-02-02 20:39:43 +01:00
c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. ( #7233 )
...
* Move no-reactor to pytest-asyncio.
* Unify the style.
* Skip a test that installs a reactor in no-reactor.
* Fix pinned env deps.
* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
3186ccf5d5
Reactorless tests 1: the test env ( #6952 )
...
* Reactorless tests.
* Updates after merging master.
* Update a new import.
* Updates after merging master.
* Update testenv:no-reactor-pinned.
* Foundations for the reactorless mode.
* Fixes after merging master.
* Updates after merging reactorless-base.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Fix a test.
* Fixes.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
* Fix a new import.
* Clarify --reactor marks.
* Cleanup.
* More docs/comments.
* Update test_fallback_workflow().
* Update TestDeferredFFromCoroF.
* Typing improvements.
2026-02-02 19:25:30 +05:00
54d8562fb0
Assorted test improvements ( #7232 )
...
* Await some missed deferreds.
* Simplify test_engine_stop_download_*.
* Fix the header value.
* Extend TestHttpWithCrawlerBase.
* Make test_tls_logging more universal.
* Add more tests for header handling.
* Clarify certificate and ip_address integration tests.
* Make test_download_with_maxsize_very_large_file() more universal.
* Fix test_coroutine_asyncio().
* Typo.
2026-02-02 09:37:39 +01:00
4e1faf883d
Foundations for the reactorless mode. ( #7199 )
...
* Foundations for the reactorless mode.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
2026-01-31 20:48:54 +05:00
49930dfec5
chore: Update outdated GitHub Actions versions ( #7223 )
2026-01-23 12:26:58 +01:00
3ec6ae05c1
Fix omitting repeated dataloss warnings in HTTP11DownloadHandler. ( #7222 )
2026-01-22 00:41:27 +05:00
2347138ba4
Unified download handler exceptions ( #7208 )
...
* Add DownloadTimeoutError.
* Don't expect ConnectionAborted in tests.
* Add DownloadCancelledError.
* Add ResponseDataLoss.
* Add more download handler tests.
* Add DownloadConnectionRefusedError.
* Add UnsupportedURLScheme.
* Add CannotResolveHostError.
* Add DownloadFailedError.
* Remove wrapped Twisted exceptions from lists.
* Update a test expectation.
* Extract wrap_twisted_exceptions().
* Wrap TxTimeoutError.
* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
ba3d7bc7a8
Remove the non-standard 307/308 handling, and align other aspects with the standard
2026-01-21 18:46:05 +01:00
9bae1ee21f
Lazy creation of Downloader._slot_gc_loop. ( #7210 )
2026-01-20 12:58:09 +01:00
04db6a5424
Add a docstring to _load_policy_class()
2026-01-20 12:35:00 +01:00
a39545195e
allow to override → allow overriding
2026-01-20 12:20:37 +01:00
842d0becf0
Rename test function
2026-01-20 12:15:25 +01:00
1b9c8b55da
not → now
2026-01-20 12:13:44 +01:00
2651d48f20
Cover the 301 behavior change in the release notes
2026-01-19 19:25:00 +01:00
99d5d58e20
Add a release notes entry
2026-01-19 19:19:02 +01:00
f7f18123eb
Remove body on 307/308 redirects and switch to GET on 301
2026-01-19 19:15:20 +01:00
6f03f3250b
Add versionadded
2026-01-19 16:46:53 +01:00
b6e5c58ae7
Fix arbitrary callable execution from the Referrer-Policy response header
2026-01-19 16:44:03 +01:00
0b9d8da09d
Bump version: 2.14.0 → 2.14.1
2026-01-12 16:25:43 -03:00
c9fbf6c599
Release notes for 2.14.1 ( #7217 )
...
* Release notes for 2.14.1
* Update docs/news.rst
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* Update docs/news.rst
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Update docs/news.rst
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* Update message about github action change
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-01-12 16:23:51 -03:00
e30ba7d4ca
( #7180 ) Update codecov test results action to v5 on all CI workflows. ( #7215 )
2026-01-12 08:31:53 +01:00
0f07b2e38c
Pass the spider arg to custom stat collectors {open,close}_spider(). ( #7213 )
2026-01-09 16:57:53 +01:00
1af283387f
Deprecate maybeDeferred_coro(). ( #7212 )
2026-01-09 01:09:39 +05:00
3ac1192f35
Bump version: 2.13.4 → 2.14.0
2026-01-05 15:06:13 +05:00
7bef98b4f1
Release notes for 2.14.0 ( #7035 )
...
* Initial release notes for 2.14.0, up to ed63fa9393d715eb49647426aafd1e8de245a7e132
2026-01-05 14:51:05 +05:00
d1bd8eb49f
Async API for download handlers. ( #7164 )
2025-12-31 19:03:16 +01:00
a2463325db
Allow Crawler.crawl_async() without asyncio. ( #7202 )
2025-12-31 18:33:59 +01:00
9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation ( #7198 )
2025-12-22 10:00:39 +01:00
180ca39b23
Deprecate returning Deferreds from pipeline methods ( #7179 )
...
* Add tests for exceptions in pipelines.
* Deprecate returning Deferreds from pipeline process_item().
* Deprecate returning Deferreds from pipeline {open,close}_spider().
* Update the custom pipeline docs.
2025-12-16 13:35:07 +05:00
5a7e132486
Restore and deprecate the spider arg of Downloader._get_slot(). ( #7178 )
2025-12-15 11:42:24 +01:00
c49ae2115a
Refactor MediaPipeline.process_item() to async def ( #7177 )
2025-12-15 10:02:04 +01:00
588f3d4f65
Remove docs mentioning deprecated spider attrs. ( #7176 )
2025-12-15 09:51:56 +01:00
d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue ( #6940 )
...
* Change default priority queue to DownloaderAwarePriorityQueue
* Fix documentation building
* Simplify test_start_already_running_exception changes.
* Modernize the test.
* Fix TestEngineCloseSpider.
* Fix typing.
* Remove special slot=None handling.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-11 11:25:17 +01:00
1a3e343dc4
Update conftest.py to refactor autouse pytest.skip() fixtures into pytest_runtest_setup() ( #7173 )
2025-12-11 09:48:25 +01:00
6ba6b032ad
Fix handling meta["download_slot"] == None. ( #7172 )
2025-12-11 09:46:51 +01:00
9bfa58e36c
Deprecate send_catch_log_deferred(). ( #7161 )
...
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
2025-12-10 14:42:49 +05:00
5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws ( #7069 )
...
* Restore coverage in MiddlewareManager spider arg handling.
* Move MiddlewareManager._warn_spider_arg() tests.
* Add DownloaderMiddlewareManager.download_async().
* Make download_func async.
* Inline MiddlewareManager._warn_spider_arg().
* Deprecate returning deferreds from downloader middlewares.
* Update the downloader mw doc about async def.
* Use global_object_name() instead of repr().
* Update docs/topics/downloader-middleware.rst
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* Update downloader-middleware.rst
---------
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-12-09 20:07:36 +05:00
7ed20ee7f3
Limit the queue size in _parallel_asyncio(). ( #7159 )
2025-12-09 15:24:40 +01:00
Andrey Rakhmatullin
Anant Murmu
Varun Chawla
Ryotaro
Martin Schimandl
Michael
Albert Eduardovich N.
Adrian Chaves
Pádraic Slattery
Adrián Chaves
Eugenio Lacuesta
Eugenio Lacuesta
Santiago Beltrán Ruiz
Adham 3llam
Thalison Fernandes
DenisC