db37040a09
Merge pull request #7142 from wRAR/pin-twisted
...
Pin Twisted to <= 25.5.0 due to internal API changes.
2025-11-13 22:48:33 +05:00
ba30e8b82c
Pin Twisted to <= 25.5.0 due to internal API changes.
2025-11-10 00:58:27 +05:00
8c5fa6e6ae
Line buffering ( #7094 )
2025-10-27 15:42:43 +01:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
61b4befc60
Remove code deprecated in 2.12.0 ( #7126 )
...
* Remove smaller things.
* Remove from_settings().
* Remove old MediaPipeline init code.
* Cleanup.
* Typo.
2025-10-27 17:14:38 +05:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
e48a1bdde3
Speed up and refactor command tests ( #7118 )
2025-10-27 12:28:16 +01:00
9cce6c30db
Undeprecate download_delay. ( #7117 )
2025-10-24 17:44:43 +05:00
5afc9b0221
Automatically transpose images using EXIF data ( #6975 )
...
* Automatically transpose images using EXIF data
* Added test
---------
Co-authored-by: Andrew Murray <radarhere@users.noreply.github.com>
2025-10-24 16:08:23 +05:00
eb496470f1
Add missing doc deps for new examples ( #7116 )
2025-10-24 11:58:18 +02:00
a5bbeb2586
Expanded documentation for issue #6676 ( #6957 )
2025-10-24 10:30:33 +02:00
2d073a9c0d
Fix typo in logging documentation ( #7109 )
2025-10-20 15:52:46 +02:00
e98c1644ce
Add unit tests that check download handlers behave like basic http clients ( #6946 ) ( #7059 )
2025-10-20 13:08:26 +02:00
b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. ( #7079 )
2025-10-16 13:15:07 +02:00
7b215c6578
UTF-8 BOM at the beginning of the file ignored ( #7095 )
2025-10-15 18:15:07 +04:00
4865a500b6
resolves #6876 ( #6977 )
2025-10-09 22:14:34 +02:00
a02abdcf63
Fix typos in documentation ( #7076 )
2025-10-06 11:43:36 +02:00
c577771838
Drop tests/upper-constraints.txt. ( #7073 )
2025-10-06 11:32:20 +02:00
10850e7d29
Fix extra stdout output in tests ( #7070 )
2025-10-06 11:19:34 +02:00
1c6ba00fd0
Move LogCounterHandler into a new extension ( #7046 )
...
* Move LogCounterHandler into Crawler.crawl().
* Add the LogCount extension.
* Update test_spider_custom_settings_log_level().
* Typo.
* Update docs.
2025-10-03 20:14:11 +05:00
798390c096
Improve performance of deprecated spider arg checks. ( #7045 )
2025-10-03 14:17:46 +02:00
dd0b071bcc
Deprecate sync methods of ItemPipelineManager/ITEM_PROCESSOR. ( #7043 )
2025-10-03 14:15:42 +02:00
f2531808f3
docs: Add Edit on GitHub ( #7058 )
2025-09-28 13:16:47 +05:00
393d715205
Drop PyPy 3.10 support, restore coverage for subprocesses ( #7050 )
2025-09-15 11:24:09 +02:00
d239fcf936
Fix get_meta_refresh to correctly resolve relative URLs using <base> tag ( #7047 )
2025-09-10 15:17:16 +02:00
2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. ( #7037 )
...
* Deprecate the spider arg of downloader middleware methods.
* Add a test for deprecated spider args in downloader mws.
2025-09-02 21:51:47 +05:00
c097921c44
Extract @_warn_spider_arg. ( #7033 )
...
* Extract @_warn_spider_arg.
* Also deprecate passing spider=None.
2025-09-01 20:17:13 +05:00
80beec41b5
Log when AsyncCrawlerProcess or CrawlerProcess is instantiated. ( #7034 )
2025-09-01 13:51:11 +02:00
00b2be0943
Add unit tests for the StatsMailer extension ( #7013 )
2025-09-01 11:43:57 +02:00
3ee4a52aa1
Raise NotConfigured when FILES_STORE or IMAGES_STORE is None or empty ( #6969 )
2025-09-01 11:39:39 +02:00
ed63fa94d6
Deprecate spider args of middlewares and pipelines. ( #7006 )
...
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
b68330811b
Deprecate spider arguments of StatsCollector methods ( #7011 )
2025-08-20 09:52:22 +02:00
edba1ad572
Refactor RobotsTxtMiddleware to use async def process_request(). ( #6802 )
2025-08-18 18:40:46 +02:00
15655ca834
Improve cleanup in TestEngineCloseSpider. ( #7012 )
2025-08-18 20:00:23 +05:00
3c546bdb82
Refactor chainDeferred() usages ( #7008 )
2025-08-16 09:35:23 +02:00
d3e15a10cf
Add ItemPipelineManager.process_item_async() and ensure_awaitable(). ( #7005 )
...
* Add ItemPipelineManager.process_item_async() and ensure_awaitable().
* Wording.
2025-08-12 19:57:37 +05:00
57f539d7a5
Deprecate Crawler.stop(). ( #6999 )
2025-08-11 08:32:55 +02:00
a0b766f9e1
More *_async APIs ( #6997 )
...
* Add ExecutionEngine.close_spider_async().
* Add Scraper.{open,close}_spider_async().
* Fix double engine stopping via Crawler.stop().
* Add ExecutionEngine.stop_async().
* Add versionadded to new async APIs.
* Add ExecutionEngine.close_async().
2025-08-08 22:15:41 +05:00
d27d0a4ed9
Deprecate setting overrides through spider attributes ( #6994 )
2025-08-07 21:26:10 +02:00
a3daa3612e
Rewrite ExecutionEngine.close_spider() to inlineCallbacks. ( #6986 )
2025-08-06 08:01:55 +02:00
baa579df62
Deprecate various `spider` args in `MiddlewareManager` subclasses ( #6984 )
...
* Add a crawler argument to MiddlewareManager.__init__().
* Deprecate the spider argument in some MiddlewareManager subclasses.
* Remove the spider argument from ExecutionEngine.open_spider_async().
* Remove the spider argument from Scraper.open_spider().
* Fix the Scraper.close_spider() type hint.
* Remove the spider argument from Downloader internals.
* Deprecate the spider arg of download_func.
* Deprecate the spider arg of Downloader.fetch().
* Deprecate the spider arg of ItemPipelineManager.process_item().
* Fix pylint.
* Tests for MiddlewareManager spider code.
* Deprecated spider arg handling for custom pipeline managers.
* Add a test for the deprecated spider arg in a custom downloader.
* Remove TODO.
* Fix pylint.
2025-08-05 12:21:08 +05:00
c47b5d049a
fix for formatting interactive shell ( #6980 )
2025-08-01 11:01:40 +02:00
32f2aede82
Deprecate non-async counterparts of new async APIs. ( #6979 )
...
* Deprecate non-async counterparts of new async APIs.
* Deprecate Scraper.call_spider().
* Add _schedule_coro().
2025-08-01 00:58:48 +05:00
605e7669a5
Improve the assumption in test_queue_push_pop_priorities(). ( #6974 )
2025-07-28 20:37:45 +02:00
89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() ( #6842 )
...
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
6ce0c4c855
Fix small typo in middlewares.py.tmpl ( #6970 )
...
Changed "maching" to "matching"
2025-07-27 01:52:57 +02:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
8c8f4ff033
Extract in-process test HTTP servers into separate processes ( #6960 )
...
* Switch TestEngineBase to mockserver.
* Split mockserver.py.
* Stop using WrappingFactory.
* Remove keyfile/certfile fields from TestHttpBase.
* Move webclient-only resources to test_webclient.py.
* Refactor MockServer.url().
* Migrate TestHttpBase to mockserver.
* Migrate TestSimpleHttpsBase to a separate mockserver.
* Rename TestHttpMockServerBase.
* Migrate TestHttpProxyBase to a separate mockserver.
* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
9e83a58643
Add type check in get_media_requests to enforce image / file urls as a list ( #6949 )
2025-07-19 11:54:35 +02:00
e225d0dea4
Ban more imports that import twisted.internet.reactor. ( #6947 )
2025-07-10 21:19:50 +05:00
Mikhail Korobov
Andrey Rakhmatullin
Vasiliy Kiryanov
Andrew Murray
Adrián Chaves
Michael
sstegmanbsd
Wes Galbraith
Vasiliy Kiryanov
adglee
Sebastien Dionne
Laerte Pereira
Ashar Khan
M Ikram Ullah Khan
NavpreetSSidhu
Neeraj Palliyali
Felipe Benevolo
MarlonDev