edba1ad572
Refactor RobotsTxtMiddleware to use async def process_request(). ( #6802 )
2025-08-18 18:40:46 +02:00
15655ca834
Improve cleanup in TestEngineCloseSpider. ( #7012 )
2025-08-18 20:00:23 +05:00
3c546bdb82
Refactor chainDeferred() usages ( #7008 )
2025-08-16 09:35:23 +02:00
d3e15a10cf
Add ItemPipelineManager.process_item_async() and ensure_awaitable(). ( #7005 )
...
* Add ItemPipelineManager.process_item_async() and ensure_awaitable().
* Wording.
2025-08-12 19:57:37 +05:00
57f539d7a5
Deprecate Crawler.stop(). ( #6999 )
2025-08-11 08:32:55 +02:00
a0b766f9e1
More *_async APIs ( #6997 )
...
* Add ExecutionEngine.close_spider_async().
* Add Scraper.{open,close}_spider_async().
* Fix double engine stopping via Crawler.stop().
* Add ExecutionEngine.stop_async().
* Add versionadded to new async APIs.
* Add ExecutionEngine.close_async().
2025-08-08 22:15:41 +05:00
d27d0a4ed9
Deprecate setting overrides through spider attributes ( #6994 )
2025-08-07 21:26:10 +02:00
a3daa3612e
Rewrite ExecutionEngine.close_spider() to inlineCallbacks. ( #6986 )
2025-08-06 08:01:55 +02:00
baa579df62
Deprecate various `spider` args in `MiddlewareManager` subclasses ( #6984 )
...
* Add a crawler argument to MiddlewareManager.__init__().
* Deprecate the spider argument in some MiddlewareManager subclasses.
* Remove the spider argument from ExecutionEngine.open_spider_async().
* Remove the spider argument from Scraper.open_spider().
* Fix the Scraper.close_spider() type hint.
* Remove the spider argument from Downloader internals.
* Deprecate the spider arg of download_func.
* Deprecate the spider arg of Downloader.fetch().
* Deprecate the spider arg of ItemPipelineManager.process_item().
* Fix pylint.
* Tests for MiddlewareManager spider code.
* Deprecated spider arg handling for custom pipeline managers.
* Add a test for the deprecated spider arg in a custom downloader.
* Remove TODO.
* Fix pylint.
2025-08-05 12:21:08 +05:00
c47b5d049a
fix for formatting interactive shell ( #6980 )
2025-08-01 11:01:40 +02:00
32f2aede82
Deprecate non-async counterparts of new async APIs. ( #6979 )
...
* Deprecate non-async counterparts of new async APIs.
* Deprecate Scraper.call_spider().
* Add _schedule_coro().
2025-08-01 00:58:48 +05:00
605e7669a5
Improve the assumption in test_queue_push_pop_priorities(). ( #6974 )
2025-07-28 20:37:45 +02:00
89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() ( #6842 )
...
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
6ce0c4c855
Fix small typo in middlewares.py.tmpl ( #6970 )
...
Changed "maching" to "matching"
2025-07-27 01:52:57 +02:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
8c8f4ff033
Extract in-process test HTTP servers into separate processes ( #6960 )
...
* Switch TestEngineBase to mockserver.
* Split mockserver.py.
* Stop using WrappingFactory.
* Remove keyfile/certfile fields from TestHttpBase.
* Move webclient-only resources to test_webclient.py.
* Refactor MockServer.url().
* Migrate TestHttpBase to mockserver.
* Migrate TestSimpleHttpsBase to a separate mockserver.
* Rename TestHttpMockServerBase.
* Migrate TestHttpProxyBase to a separate mockserver.
* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
9e83a58643
Add type check in get_media_requests to enforce image / file urls as a list ( #6949 )
2025-07-19 11:54:35 +02:00
e225d0dea4
Ban more imports that import twisted.internet.reactor. ( #6947 )
2025-07-10 21:19:50 +05:00
6b2997af90
Migrate to pytest-twisted ( #6938 )
...
* Migrate to pytest-twisted (WIP)
* Some typing fixes.
* Make --reactor=asyncio the default again.
* Try installing the correct event loop policy in tests on Windows.
* Make reactor_pytest a normal fixture.
* Fix test warnings.
* Fix FTPDownloadHandler teardown.
* Cleanups, typing.
* More cleanup.
* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
14eace5d8f
Increase the command timeout in test_short_timeout(). ( #6945 )
2025-07-06 01:56:54 +05:00
4279f2837c
Enable the flake8-builtins rules. ( #6942 )
2025-07-05 21:43:18 +05:00
df342eee6e
Bump tool versions. ( #6941 )
2025-07-05 10:30:30 +02:00
16f168b406
Deprecate mustbe_deferred() and defer_*(). ( #6937 )
2025-07-04 16:41:15 +05:00
d9ef0350d8
Merge remote-tracking branch 'origin/2.13'
2025-07-02 20:46:02 +05:00
155a504f24
Bump version: 2.13.2 → 2.13.3
2025-07-02 20:39:46 +05:00
cf465bf644
Release notes for 2.13.3. ( #6934 )
...
* Release notes for 2.13.3.
* Address feedback.
2025-07-02 20:36:01 +05:00
8d82e7bd27
Fix limiting subprocess run time in tests. ( #6933 )
...
* Fix limiting subprocess run time in tests.
* Make test_runspider_dnscache_disabled() faster.
2025-07-01 20:23:35 +05:00
7842180b2f
Merge remote-tracking branch 'origin/2.13'
2025-07-01 17:28:12 +05:00
c9cdf0af3c
Narrow down TestEngine::test_short_timeout() expectations. ( #6911 )
2025-07-01 13:47:43 +02:00
dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting ( #6921 )
...
* Deprecate CONCURRENT_REQUESTS_PER_IP setting
* Typing fix
* PR review
* tests
* oops
* Update scrapy/settings/default_settings.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* update logic
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-06-30 16:51:05 +05:00
3843091c5f
Mark abstract base classes as such. ( #6930 )
...
* Mark abstract base classes as such.
* Remove an unneeded method.
* Fix exporter test coverage.
2025-06-30 15:14:34 +05:00
8b3c3ea4ae
Refactor test functions that were returning a Deferred directly. ( #6889 )
2025-06-30 09:19:23 +02:00
db0be1771c
Test refactoring, part 2. ( #6928 )
2025-06-29 23:10:38 +02:00
03fe7a6424
Add a deprecation notice for the offsite spider middleware.
2025-06-29 02:35:48 +05:00
2c1c10e923
Remove deprecated offsite spider middleware. ( #6926 )
...
* Remove deprecated offsite spider middleware.
* Add a deprecation notice for the offsite spider middleware.
2025-06-29 02:35:20 +05:00
ba1cedee1b
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:13:54 +05:00
076c01104a
Lower the concurrency settings in the default project template. ( #6918 )
...
* Lower the concurrency settings in the default project template.
* Simplify the concurrency settings template.
2025-06-28 18:13:44 +05:00
3019393686
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:04:40 +05:00
1473f4d347
Remove the deprecated sphinx-hoverxref ( #6922 )
2025-06-28 18:04:06 +05:00
95172659af
Add sphinx-lint. ( #6920 )
2025-06-28 01:37:20 +02:00
f454465b14
Lower the concurrency settings in the default project template. ( #6918 )
...
* Lower the concurrency settings in the default project template.
* Simplify the concurrency settings template.
2025-06-27 17:40:31 +05:00
5fbab843bd
Fix invalid XML samples in tests that break with lxml 6. ( #6919 )
2025-06-27 16:24:09 +05:00
eaae59fbef
Fix invalid XML samples in tests that break with lxml 6. ( #6919 )
2025-06-27 13:05:02 +02:00
ef01a953b1
Fix setting ExecutionEngine._slot to None on close. ( #6910 )
...
* Fix setting ExecutionEngine._slot to None on close.
* Check self._slot in needs_backout().
2025-06-25 17:30:57 +05:00
ff7795b159
Fix setting ExecutionEngine._slot to None on close. ( #6910 )
...
* Fix setting ExecutionEngine._slot to None on close.
* Check self._slot in needs_backout().
2025-06-25 17:29:29 +05:00
d70f8a3f14
Refactoring of test_utils_*. ( #6905 )
2025-06-23 23:39:24 +05:00
7fbd56bc9b
Handle exceptions in _start_request_processing(), cancel it on engine stop ( #6900 )
2025-06-23 22:38:38 +05:00
0d75355b41
Handle exceptions in _start_request_processing(), cancel it on engine stop ( #6900 )
2025-06-23 19:36:54 +02:00
b53faacfcd
Add .vscode to .gitignore. ( #6907 )
2025-06-23 22:26:13 +05:00
5e20b46e35
Fix FileFeedStorage handling of Windows paths without file:// scheme ( #6897 )
2025-06-23 22:26:04 +05:00
Andrey Rakhmatullin
NavpreetSSidhu
Neeraj Palliyali
Felipe Benevolo
MarlonDev
M Ikram Ullah Khan
Laerte Pereira
Adrián Chaves
Thalison Fernandes