6a42bc6450
Item docs: mention Pydantic support ( #6966 )
...
* Add Pydantic support documentation
* Fix Sphinx references in Pydantic section for non-Sphinx docs
* Minor reformatting
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-05 22:18:01 +04:00
2347138ba4
Unified download handler exceptions ( #7208 )
...
* Add DownloadTimeoutError.
* Don't expect ConnectionAborted in tests.
* Add DownloadCancelledError.
* Add ResponseDataLoss.
* Add more download handler tests.
* Add DownloadConnectionRefusedError.
* Add UnsupportedURLScheme.
* Add CannotResolveHostError.
* Add DownloadFailedError.
* Remove wrapped Twisted exceptions from lists.
* Update a test expectation.
* Extract wrap_twisted_exceptions().
* Wrap TxTimeoutError.
* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
a39545195e
allow to override → allow overriding
2026-01-20 12:20:37 +01:00
6f03f3250b
Add versionadded
2026-01-19 16:46:53 +01:00
b6e5c58ae7
Fix arbitrary callable execution from the Referrer-Policy response header
2026-01-19 16:44:03 +01:00
7bef98b4f1
Release notes for 2.14.0 ( #7035 )
...
* Initial release notes for 2.14.0, up to ed63fa9393d715eb49647426aafd1e8de245a7e132
2026-01-05 14:51:05 +05:00
d1bd8eb49f
Async API for download handlers. ( #7164 )
2025-12-31 19:03:16 +01:00
a2463325db
Allow Crawler.crawl_async() without asyncio. ( #7202 )
2025-12-31 18:33:59 +01:00
9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation ( #7198 )
2025-12-22 10:00:39 +01:00
180ca39b23
Deprecate returning Deferreds from pipeline methods ( #7179 )
...
* Add tests for exceptions in pipelines.
* Deprecate returning Deferreds from pipeline process_item().
* Deprecate returning Deferreds from pipeline {open,close}_spider().
* Update the custom pipeline docs.
2025-12-16 13:35:07 +05:00
588f3d4f65
Remove docs mentioning deprecated spider attrs. ( #7176 )
2025-12-15 09:51:56 +01:00
d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue ( #6940 )
...
* Change default priority queue to DownloaderAwarePriorityQueue
* Fix documentation building
* Simplify test_start_already_running_exception changes.
* Modernize the test.
* Fix TestEngineCloseSpider.
* Fix typing.
* Remove special slot=None handling.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-11 11:25:17 +01:00
9bfa58e36c
Deprecate send_catch_log_deferred(). ( #7161 )
...
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
2025-12-10 14:42:49 +05:00
5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws ( #7069 )
...
* Restore coverage in MiddlewareManager spider arg handling.
* Move MiddlewareManager._warn_spider_arg() tests.
* Add DownloaderMiddlewareManager.download_async().
* Make download_func async.
* Inline MiddlewareManager._warn_spider_arg().
* Deprecate returning deferreds from downloader middlewares.
* Update the downloader mw doc about async def.
* Use global_object_name() instead of repr().
* Update docs/topics/downloader-middleware.rst
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* Update downloader-middleware.rst
---------
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-12-09 20:07:36 +05:00
0adc561348
Add a list of remaining Deferred APIs. ( #7146 )
2025-11-27 20:13:04 +05:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
61b4befc60
Remove code deprecated in 2.12.0 ( #7126 )
...
* Remove smaller things.
* Remove from_settings().
* Remove old MediaPipeline init code.
* Cleanup.
* Typo.
2025-10-27 17:14:38 +05:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
a5bbeb2586
Expanded documentation for issue #6676 ( #6957 )
2025-10-24 10:30:33 +02:00
b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. ( #7079 )
2025-10-16 13:15:07 +02:00
a02abdcf63
Fix typos in documentation ( #7076 )
2025-10-06 11:43:36 +02:00
1c6ba00fd0
Move LogCounterHandler into a new extension ( #7046 )
...
* Move LogCounterHandler into Crawler.crawl().
* Add the LogCount extension.
* Update test_spider_custom_settings_log_level().
* Typo.
* Update docs.
2025-10-03 20:14:11 +05:00
2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. ( #7037 )
...
* Deprecate the spider arg of downloader middleware methods.
* Add a test for deprecated spider args in downloader mws.
2025-09-02 21:51:47 +05:00
ed63fa94d6
Deprecate spider args of middlewares and pipelines. ( #7006 )
...
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
b68330811b
Deprecate spider arguments of StatsCollector methods ( #7011 )
2025-08-20 09:52:22 +02:00
89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() ( #6842 )
...
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting ( #6921 )
...
* Deprecate CONCURRENT_REQUESTS_PER_IP setting
* Typing fix
* PR review
* tests
* oops
* Update scrapy/settings/default_settings.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* update logic
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-06-30 16:51:05 +05:00
3843091c5f
Mark abstract base classes as such. ( #6930 )
...
* Mark abstract base classes as such.
* Remove an unneeded method.
* Fix exporter test coverage.
2025-06-30 15:14:34 +05:00
3019393686
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:04:40 +05:00
95172659af
Add sphinx-lint. ( #6920 )
2025-06-28 01:37:20 +02:00
daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts ( #6888 )
...
* Remove the chdir fixture.
* Re-enable fancy pytest asserts.
* Remove doc files from pytest ignores.
* Restore docs/_ext in test collection ignores.
* Skip a doctest that fails on Windows.
* Fix tests that were writing to the current dir.
2025-06-14 14:01:20 +05:00
3aa5e75787
Use AsyncCrawlerProcess in commands. ( #6845 )
...
* Use AsyncCrawlerProcess in commands.
* Ignore coverage of abstract methods.
* Address feedback.
2025-06-03 17:47:52 +05:00
e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. ( #6824 )
...
* Don't use CrawlerProcess in the commands that don't need it.
* Use a dummy spider loader in runspider.
2025-05-28 19:57:33 +05:00
05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` ( #6817 )
...
* Add a basic Crawler.crawl_async().
* Add custom loop tests for *CrawlerRunner.
* Add AsyncCrawlerProcess.
* Update related docs.
* Update practices.rst.
* Address test failures.
* Add a note about AsyncCrawler* to the docs about switching reactors.
* Address feedback.
* Update for TID253.
* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
2025-05-28 18:55:44 +05:00
dceb85bf3e
Add is_asyncio_available(). ( #6827 )
...
* Add is_asyncio_available().
* Print unexpected warnings in test_install_asyncio_reactor().
* Fix printing warnings.
* Fix printing warnings - 2.
* Skip TestDeferredToFuture on non-asyncio.
* Test the is_asyncio_available() exception.
2025-05-28 14:46:39 +05:00
f2fc177f1f
Fix a wrong versionadded usage. ( #6822 )
2025-05-23 11:06:33 +02:00
ff7d29654a
Fix typo in documentation and code: 'needs_backoff' -> 'needs_backout' ( #6815 )
...
Corrected the typo in the code and documentation where 'needs_backoff' was incorrectly used instead of 'needs_backout'.
2025-05-20 07:51:31 +02:00
3c2cd53abb
Skip the doctest.
2025-05-15 22:17:37 +05:00
bf1bfaaa3e
Slight improvements for the signal docs.
2025-05-15 20:02:38 +05:00
1ddcb568e2
Add send_catch_log_async().
2025-05-15 15:45:55 +05:00
82acef3051
Add AsyncCrawlerRunner. ( #6796 )
2025-05-14 16:21:18 +02:00
0ce693dfa9
Update VERSION strings.
2025-05-08 13:23:19 +05:00
b07e6d4ea8
Merge remote-tracking branch 'origin/master' into relnotes-2.13.0
2025-05-08 13:17:47 +05:00
e86b5051c2
Merge commit '373e501' into relnotes-2.13.0
2025-05-08 10:55:57 +05:00
5f6d1b464b
Cover up to 373e501 in the release notes.
2025-05-08 10:55:40 +05:00
036f3e5627
Support asynchronous start requests ( #6729 )
2025-05-07 19:04:03 +02:00
474f8312ff
Merge pull request #6776 from wRAR/deferred-apis
...
Add a list of Deferred-only APIs.
2025-05-05 21:28:50 +04:00
acb5f895cd
Update docs/topics/coroutines.rst
...
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-05-05 22:28:36 +05:00
523fc25c4d
Document default values set by startproject ( #6775 )
2025-05-05 18:51:15 +02:00
b93290f28a
Add a list of Deferred-only APIs.
2025-05-05 19:38:04 +05:00
095140f134
Cover the current master in the release notes (up to daf9db7).
2025-04-25 19:43:29 +05:00
daf9db72b2
Base class for universal spider middlewares ( #6693 )
...
* Initial BaseSpiderMiddleware.
* Rename the new methods.
* Remove the spider argument from new BaseSpiderMiddleware methods.
* Add docs for BaseSpiderMiddleware.
* Silence pylint.
* Add BaseSpiderMiddleware tests.
* Add a release note.
2025-04-23 19:29:04 +05:00
3ca882fba8
Syntax Error Fixed ( #6738 )
...
* Syntax error fix issue #6731
* test case added
* extra logic removed
* mock spider fixture
* Update scrapy/utils/misc.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* settings.rst updated
* settings.rst updated
* settings.rst updated
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-03-20 17:02:10 +05:00
9057bf4e1e
More docs about Deferred<->Future interoperability. ( #6734 )
2025-03-15 11:47:16 +01:00
fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). ( #6732 )
2025-03-14 20:46:17 +01:00
d0dabbc097
Enable AsyncioSelectorReactor by default. ( #6713 )
...
* Enable AsyncioSelectorReactor by default.
* Improve get_crawler(), switch more tests to it.
* Fix the remaining default-reactor test failures.
* Address documentation feedback.
* Make pinned envs more consistent.
2025-03-12 01:18:30 +05:00
ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) ( #6723 )
2025-03-11 14:53:04 +01:00
5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call ( #6719 )
2025-03-11 14:52:48 +01:00
9b7db1a068
Move some reference docs of Request to the code ( #6721 )
2025-03-11 12:43:50 +01:00
bee74fb753
Remove trailing whitespace
2025-03-11 11:56:44 +01:00
d2e5486d5a
Remove the Splash recommendation.
2025-03-07 13:20:42 +05:00
0c9200094e
Extend BaseSettings with utils for add-ons ( #6614 )
2025-03-05 10:31:59 +01:00
c200458f24
Add more docs for updating sync spider middlewares. ( #6688 )
2025-02-25 00:16:23 +01:00
2eb3c75c69
Remove AjaxCrawlMiddleware mention from built-in downloader middleware
2025-02-05 13:16:51 -03:00
76a8badd24
Add a deprecation notice to the AjaxCrawlMiddleware docs.
2025-02-03 14:55:10 +05:00
393ff96e45
Deprecate AjaxCrawlMiddleware.
2025-02-03 14:08:05 +05:00
9bc0029d27
Allow updating pre-crawler settings from add-ons ( #6568 )
2025-01-22 12:07:44 +01:00
ca345a3b73
Flexible severity of logging level when items are dropped ( #6608 )
2025-01-15 11:08:18 +01:00
98ba61256d
Deprecate BaseDupeFilter.log() and improve dupefilter docs ( #4151 )
...
* Remove BaseDupeFilter.log()
It is never called because request_seen() always returns False
* Document the interface of DUPEFILTER_CLASS classes
* Remove unnecessary BaseDupeFilter comments and add a short class description
* Improve the documentation related to the DUPEFILTER_CLASS setting
* Deprecate BaseDupeFilter.log
* Update the docs
* Fix the new code example
* Remove typing to keep the example short
Otherwise, it would have required yet another import line (from __future__ or typing).
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-01-14 19:36:56 +05:00
1fc91bb462
new `allow_offsite` parameter in OffsiteMiddleware ( #6151 )
...
* new 'allow_offsite' parameter in OffsiteMiddleware
* document deprecated dont_filter flag in OffsiteMiddleware
* avoid deprecating dont_filter in OffsiteMiddleware
* Copy the code to the downloader mw.
* Add tests for allow_offsite in the downloader mw.
* Mark allow_offsite with reqmeta.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-01-08 21:28:51 +05:00
59fcb9b93c
Improve internal refs to scrapy.Request and scrapy.Selector ( #6526 )
...
* Improve internal refs to scrapy.Selector.
* Improve internal refs to scrapy.Request.
* More scrapy.http fixes.
* Fix FormRequest refs.
* More fixes.
* Simplifications.
* Last fixes.
* Add the parsel intersphinx.
2025-01-07 16:18:18 +05:00
21b9ba717c
Allow customizing logged software versions ( #6582 )
...
Co-authored-by: Grammy Jiang <grammy.jiang@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-12-16 14:46:23 +01:00
10089c6fe2
2.12 release notes ( #6226 )
...
* Cover 2.12 in the release notes up to 9bb973dc54642af407a0a34bb4bad97#6463 , #6507 , #6511 to the 2.12 release notes.
* Cover 2.12 in the release notes up to d85c39fd215669#6527 in the release notes.
* Address PR feedback.
* Cover recent PRs.
* Finalize the 2.12.0 release notes, small additional fixes.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-11-18 13:07:32 +05:00
ab5cb7c7d9
Merge pull request #6540 from wRAR/build_from_settings
...
Remove build_from_settings() and deprecate from_settings() code
2024-11-14 20:40:54 +05:00
261c4b61dc
Enhancement: Update docs to include IgnoreRequest details ( #6506 )
...
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2024-11-13 22:47:39 +05:00
499e7e8aa6
Add from_crawler() to components that only had from_settings().
2024-11-12 21:12:32 +05:00
dc3ebb6cf7
Refactor the docs
2024-11-07 16:38:48 +01:00
2a4b7fe0f8
dont_throttle → autothrottle_dont_adjust_delay
2024-11-07 16:17:16 +01:00
b244ea7ac0
Add the missing import to the docs example
2024-11-05 20:05:58 +01:00
5862216bb1
Fix docs example
2024-11-05 19:55:28 +01:00
f57fc454be
Replace Slot.throttle with Request.meta['dont_throttle']
2024-11-05 19:44:30 +01:00
e7f5ae0b34
Update the outdated Item docstring. ( #6427 )
...
* Update the outdated Item doscstring.
* Fix the reference links in items.html.
2024-11-04 16:17:56 +05:00
ce5a132f12
Run and fix linkcheck. ( #6524 )
2024-11-04 11:40:07 +01:00
7701e590fb
Documentation added for Spider State in extensions.rst ( #6522 )
...
* Documentation added for Spider State in extensions.rst
* Made correction in documentation for Spiderstate
* Added appropriate intro for Spider state extension
* Added reference for spiderstate extension
* Added Spiderstate extension hyperlink refrence in jobs.rst
2024-11-02 10:45:27 +05:00
d85c39f5bc
Deprecation removals. ( #6500 )
...
* Deprecation removals.
* Clean up the default pytest filterwarnings.
* Remove test_get_images_old().
* Redo boto-requiring test filtering.
* Remove an unused function.
* Improve the Crawler.crawl() error message.
* Fix the test.
2024-10-31 18:06:22 +05:00
5759b3f0f2
Drop Reppy.
2024-10-17 23:41:23 +05:00
46cddc6ecf
Ignore SyntaxError as well when SPIDER_LOADER_WARN_ONLY is set to True ( #6484 )
2024-10-02 10:04:03 +02:00
4f9b2343c0
Merge branch 'master' of github.com:Laerte/scrapy
2024-08-27 07:16:10 -03:00
3c2a9fa262
update docs
2024-08-27 07:16:01 -03:00
f68f29dd13
Update docs/topics/contracts.rst
2024-08-26 23:37:57 -03:00
b85e5a66ed
Add support for meta in Spider Contracts
2024-08-26 23:21:09 -03:00
6ce0342beb
Allow yielding items from start_requests ( #6417 )
...
Co-authored-by: Georgiy Zatserklianyi <george.zatseklyany@gmail.com>
Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-08-26 20:53:06 +02:00
af15bd1dad
minor changes to items section of docs ( #6462 )
2024-08-19 13:55:09 +02:00
b9ef1326a5
Proofread the commands documentation ( #6449 )
2024-08-01 09:29:11 +02:00
03a15ced4f
Do not suggest logging dropped items twice ( #6448 )
...
Co-authored-by: Kevin Lloyd Bernal <kevinoxy@gmail.com>
2024-07-31 11:37:19 +02:00
026d606528
clean closespider_pagecount_no_item docs section
...
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-07-12 08:09:03 -03:00
8646d2ec7b
fix docs detail
2024-07-10 11:44:44 -03:00
59782d7308
update docs
2024-07-10 11:08:22 -03:00
365c9e62ad
Removing empty example reference ( #6402 )
...
Co-authored-by: Michael Duane Mooring <mikeumus@gmail.com>
2024-06-17 11:37:11 +02:00
Michael
Andrey Rakhmatullin
Adrián Chaves
Adrian Chaves
Adham 3llam
Thalison Fernandes
Sebastien Dionne
Felipe Benevolo
Laerte Pereira
Keval Sakhiya
Mehraz Hossain Rumman
Laerte Pereira
guillermo-bondonno
anubhav
Kevin Lloyd Bernal
Robert Palmer
Rohit Kumar Singh
mmoriniere
Georgiy Zatserklianyi
mlmsmith
guillermo-bondonno