804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
61b4befc60
Remove code deprecated in 2.12.0 ( #7126 )
...
* Remove smaller things.
* Remove from_settings().
* Remove old MediaPipeline init code.
* Cleanup.
* Typo.
2025-10-27 17:14:38 +05:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
a5bbeb2586
Expanded documentation for issue #6676 ( #6957 )
2025-10-24 10:30:33 +02:00
b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. ( #7079 )
2025-10-16 13:15:07 +02:00
a02abdcf63
Fix typos in documentation ( #7076 )
2025-10-06 11:43:36 +02:00
1c6ba00fd0
Move LogCounterHandler into a new extension ( #7046 )
...
* Move LogCounterHandler into Crawler.crawl().
* Add the LogCount extension.
* Update test_spider_custom_settings_log_level().
* Typo.
* Update docs.
2025-10-03 20:14:11 +05:00
2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. ( #7037 )
...
* Deprecate the spider arg of downloader middleware methods.
* Add a test for deprecated spider args in downloader mws.
2025-09-02 21:51:47 +05:00
ed63fa94d6
Deprecate spider args of middlewares and pipelines. ( #7006 )
...
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
b68330811b
Deprecate spider arguments of StatsCollector methods ( #7011 )
2025-08-20 09:52:22 +02:00
89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() ( #6842 )
...
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting ( #6921 )
...
* Deprecate CONCURRENT_REQUESTS_PER_IP setting
* Typing fix
* PR review
* tests
* oops
* Update scrapy/settings/default_settings.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* update logic
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-06-30 16:51:05 +05:00
3843091c5f
Mark abstract base classes as such. ( #6930 )
...
* Mark abstract base classes as such.
* Remove an unneeded method.
* Fix exporter test coverage.
2025-06-30 15:14:34 +05:00
3019393686
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:04:40 +05:00
95172659af
Add sphinx-lint. ( #6920 )
2025-06-28 01:37:20 +02:00
daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts ( #6888 )
...
* Remove the chdir fixture.
* Re-enable fancy pytest asserts.
* Remove doc files from pytest ignores.
* Restore docs/_ext in test collection ignores.
* Skip a doctest that fails on Windows.
* Fix tests that were writing to the current dir.
2025-06-14 14:01:20 +05:00
3aa5e75787
Use AsyncCrawlerProcess in commands. ( #6845 )
...
* Use AsyncCrawlerProcess in commands.
* Ignore coverage of abstract methods.
* Address feedback.
2025-06-03 17:47:52 +05:00
e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. ( #6824 )
...
* Don't use CrawlerProcess in the commands that don't need it.
* Use a dummy spider loader in runspider.
2025-05-28 19:57:33 +05:00
05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` ( #6817 )
...
* Add a basic Crawler.crawl_async().
* Add custom loop tests for *CrawlerRunner.
* Add AsyncCrawlerProcess.
* Update related docs.
* Update practices.rst.
* Address test failures.
* Add a note about AsyncCrawler* to the docs about switching reactors.
* Address feedback.
* Update for TID253.
* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
2025-05-28 18:55:44 +05:00
dceb85bf3e
Add is_asyncio_available(). ( #6827 )
...
* Add is_asyncio_available().
* Print unexpected warnings in test_install_asyncio_reactor().
* Fix printing warnings.
* Fix printing warnings - 2.
* Skip TestDeferredToFuture on non-asyncio.
* Test the is_asyncio_available() exception.
2025-05-28 14:46:39 +05:00
f2fc177f1f
Fix a wrong versionadded usage. ( #6822 )
2025-05-23 11:06:33 +02:00
ff7d29654a
Fix typo in documentation and code: 'needs_backoff' -> 'needs_backout' ( #6815 )
...
Corrected the typo in the code and documentation where 'needs_backoff' was incorrectly used instead of 'needs_backout'.
2025-05-20 07:51:31 +02:00
3c2cd53abb
Skip the doctest.
2025-05-15 22:17:37 +05:00
bf1bfaaa3e
Slight improvements for the signal docs.
2025-05-15 20:02:38 +05:00
1ddcb568e2
Add send_catch_log_async().
2025-05-15 15:45:55 +05:00
82acef3051
Add AsyncCrawlerRunner. ( #6796 )
2025-05-14 16:21:18 +02:00
0ce693dfa9
Update VERSION strings.
2025-05-08 13:23:19 +05:00
b07e6d4ea8
Merge remote-tracking branch 'origin/master' into relnotes-2.13.0
2025-05-08 13:17:47 +05:00
e86b5051c2
Merge commit '373e501' into relnotes-2.13.0
2025-05-08 10:55:57 +05:00
5f6d1b464b
Cover up to 373e501 in the release notes.
2025-05-08 10:55:40 +05:00
036f3e5627
Support asynchronous start requests ( #6729 )
2025-05-07 19:04:03 +02:00
474f8312ff
Merge pull request #6776 from wRAR/deferred-apis
...
Add a list of Deferred-only APIs.
2025-05-05 21:28:50 +04:00
acb5f895cd
Update docs/topics/coroutines.rst
...
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-05-05 22:28:36 +05:00
523fc25c4d
Document default values set by startproject ( #6775 )
2025-05-05 18:51:15 +02:00
b93290f28a
Add a list of Deferred-only APIs.
2025-05-05 19:38:04 +05:00
095140f134
Cover the current master in the release notes (up to daf9db7).
2025-04-25 19:43:29 +05:00
daf9db72b2
Base class for universal spider middlewares ( #6693 )
...
* Initial BaseSpiderMiddleware.
* Rename the new methods.
* Remove the spider argument from new BaseSpiderMiddleware methods.
* Add docs for BaseSpiderMiddleware.
* Silence pylint.
* Add BaseSpiderMiddleware tests.
* Add a release note.
2025-04-23 19:29:04 +05:00
3ca882fba8
Syntax Error Fixed ( #6738 )
...
* Syntax error fix issue #6731
* test case added
* extra logic removed
* mock spider fixture
* Update scrapy/utils/misc.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* settings.rst updated
* settings.rst updated
* settings.rst updated
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-03-20 17:02:10 +05:00
9057bf4e1e
More docs about Deferred<->Future interoperability. ( #6734 )
2025-03-15 11:47:16 +01:00
fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). ( #6732 )
2025-03-14 20:46:17 +01:00
d0dabbc097
Enable AsyncioSelectorReactor by default. ( #6713 )
...
* Enable AsyncioSelectorReactor by default.
* Improve get_crawler(), switch more tests to it.
* Fix the remaining default-reactor test failures.
* Address documentation feedback.
* Make pinned envs more consistent.
2025-03-12 01:18:30 +05:00
ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) ( #6723 )
2025-03-11 14:53:04 +01:00
5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call ( #6719 )
2025-03-11 14:52:48 +01:00
9b7db1a068
Move some reference docs of Request to the code ( #6721 )
2025-03-11 12:43:50 +01:00
bee74fb753
Remove trailing whitespace
2025-03-11 11:56:44 +01:00
d2e5486d5a
Remove the Splash recommendation.
2025-03-07 13:20:42 +05:00
0c9200094e
Extend BaseSettings with utils for add-ons ( #6614 )
2025-03-05 10:31:59 +01:00
c200458f24
Add more docs for updating sync spider middlewares. ( #6688 )
2025-02-25 00:16:23 +01:00
2eb3c75c69
Remove AjaxCrawlMiddleware mention from built-in downloader middleware
2025-02-05 13:16:51 -03:00
Andrey Rakhmatullin
Michael
Sebastien Dionne
Felipe Benevolo
Laerte Pereira
Adrián Chaves
Keval Sakhiya
Mehraz Hossain Rumman
Laerte Pereira