7e881ce2d7
Release notes for 2.15.0 ( #7373 )
...
* Cover 2.15.0 in the release notes (up to 31bf7c30c6ccf5b68f267
2026-04-09 16:56:30 +05:00
b9be5ce053
Reactorless shell and other small shell improvements ( #7395 )
...
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.
* defer.Deferred -> Deferred.
* Allow Shell with TWISTED_REACTOR_ENABLED=False.
* Shell workflow notes.
* Remove an unused argument.
* Shell.fetch_available.
* Shell._use_reactor.
* Some more comments.
* Simplify _schedule() and fix the spider shell var.
* More async def.
* Reactorless shell support.
* Add pragma: no cover.
* More notes.
* Remove a TODO.
2026-04-08 18:24:51 +05:00
a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER ( #7361 )
2026-04-08 13:58:35 +02:00
830eaeab5d
docs: fix typos in addons.rst ( #7408 )
...
Co-authored-by: Abdul Rahman <abdouls@hotmail.com>
2026-04-08 09:38:09 +02:00
f8d103a65a
Add llms.txt and llms-full.txt generation ( #7380 )
2026-04-06 10:24:21 +02:00
b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. ( #7394 )
2026-04-02 10:20:54 +02:00
0c6ccf50b3
Docs for the reactorless mode ( #7385 )
...
* Reactorless mode docs.
* Unset the ftp handler in the reactorless mode.
* Cleanup reactorless subprocess tests.
* Better handling of incompatible TWISTED_ENABLED cases.
* Improvements.
* Improve early error logging in test scripts.
* Address feedback.
* Address feedback.
2026-04-01 20:40:34 +05:00
fa76ca52e9
Move TLS logging from _ScrapyClientTLSOptions to handlers. ( #7387 )
...
* Move TLS logging from _ScrapyClientTLSOptions to handlers.
* Update the docs.
* Fix typing.
* Update new docs.
* Add the comment back.
2026-04-01 16:02:55 +05:00
eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES ( #7379 )
...
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.
* Deprecate BrowserLikeContextFactory.
* Add DOWNLOAD_VERIFY_CERTIFICATES.
* Cleanup, add acceptableCiphers to the verifying path.
2026-04-01 15:18:43 +05:00
31bf7c3892
Docs for HttpxDownloadHandler, handle httpx not being installed ( #7368 )
...
* Add docs for HttpxDownloadHandler.
* Handle httpx not being installed.
* Revert test_not_configured_without_asyncio().
* Restore the comment.
* Cleanup.
2026-03-27 21:06:55 +05:00
54a4c3af89
Deprecate the mail API ( #7263 )
2026-03-24 21:31:12 +01:00
c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers ( #7283 )
2026-03-19 09:16:40 +01:00
4e3df249f2
Merge branch '2.14'
2026-03-12 16:02:37 +01:00
da1a6b7ebc
Add security warning about pickle files in JOBDIR ( #7274 )
...
* Add security warning about pickle files in JOBDIR
The job directory contains files serialized with pickle (spider.state,
request queues), which can execute arbitrary code when loaded. Add a
warning so users know to treat JOBDIR with the same trust level as
their project code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Make JOBDIR security warning more concise and accurate
Simplify the warning to focus on treating the job directory
as trusted, without mentioning specific serialization details.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 19:36:26 +05:00
2c3ecbff71
fix: missing self argument in ScreenshotPipeline constructor ( #7248 )
...
added missing self argument in __init__ method constructor
2026-02-13 09:42:26 +01:00
66fe5de139
Add Jobdir documentation ( #5260 )
...
* Add Jobdir documentation
* Fix link format
* remove doubtful comments
* some more edits
* Keep the information in jobs.rst an example, and provide details in the reference docs of the corresponding components
* Minor edits
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-06 22:33:02 +04:00
6a42bc6450
Item docs: mention Pydantic support ( #6966 )
...
* Add Pydantic support documentation
* Fix Sphinx references in Pydantic section for non-Sphinx docs
* Minor reformatting
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-05 22:18:01 +04:00
2347138ba4
Unified download handler exceptions ( #7208 )
...
* Add DownloadTimeoutError.
* Don't expect ConnectionAborted in tests.
* Add DownloadCancelledError.
* Add ResponseDataLoss.
* Add more download handler tests.
* Add DownloadConnectionRefusedError.
* Add UnsupportedURLScheme.
* Add CannotResolveHostError.
* Add DownloadFailedError.
* Remove wrapped Twisted exceptions from lists.
* Update a test expectation.
* Extract wrap_twisted_exceptions().
* Wrap TxTimeoutError.
* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
a39545195e
allow to override → allow overriding
2026-01-20 12:20:37 +01:00
6f03f3250b
Add versionadded
2026-01-19 16:46:53 +01:00
b6e5c58ae7
Fix arbitrary callable execution from the Referrer-Policy response header
2026-01-19 16:44:03 +01:00
7bef98b4f1
Release notes for 2.14.0 ( #7035 )
...
* Initial release notes for 2.14.0, up to ed63fa9393d715eb49647426aafd1e8de245a7e132
2026-01-05 14:51:05 +05:00
d1bd8eb49f
Async API for download handlers. ( #7164 )
2025-12-31 19:03:16 +01:00
a2463325db
Allow Crawler.crawl_async() without asyncio. ( #7202 )
2025-12-31 18:33:59 +01:00
9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation ( #7198 )
2025-12-22 10:00:39 +01:00
180ca39b23
Deprecate returning Deferreds from pipeline methods ( #7179 )
...
* Add tests for exceptions in pipelines.
* Deprecate returning Deferreds from pipeline process_item().
* Deprecate returning Deferreds from pipeline {open,close}_spider().
* Update the custom pipeline docs.
2025-12-16 13:35:07 +05:00
588f3d4f65
Remove docs mentioning deprecated spider attrs. ( #7176 )
2025-12-15 09:51:56 +01:00
d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue ( #6940 )
...
* Change default priority queue to DownloaderAwarePriorityQueue
* Fix documentation building
* Simplify test_start_already_running_exception changes.
* Modernize the test.
* Fix TestEngineCloseSpider.
* Fix typing.
* Remove special slot=None handling.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-11 11:25:17 +01:00
9bfa58e36c
Deprecate send_catch_log_deferred(). ( #7161 )
...
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
2025-12-10 14:42:49 +05:00
5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws ( #7069 )
...
* Restore coverage in MiddlewareManager spider arg handling.
* Move MiddlewareManager._warn_spider_arg() tests.
* Add DownloaderMiddlewareManager.download_async().
* Make download_func async.
* Inline MiddlewareManager._warn_spider_arg().
* Deprecate returning deferreds from downloader middlewares.
* Update the downloader mw doc about async def.
* Use global_object_name() instead of repr().
* Update docs/topics/downloader-middleware.rst
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* Update downloader-middleware.rst
---------
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-12-09 20:07:36 +05:00
0adc561348
Add a list of remaining Deferred APIs. ( #7146 )
2025-11-27 20:13:04 +05:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
61b4befc60
Remove code deprecated in 2.12.0 ( #7126 )
...
* Remove smaller things.
* Remove from_settings().
* Remove old MediaPipeline init code.
* Cleanup.
* Typo.
2025-10-27 17:14:38 +05:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
a5bbeb2586
Expanded documentation for issue #6676 ( #6957 )
2025-10-24 10:30:33 +02:00
b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. ( #7079 )
2025-10-16 13:15:07 +02:00
a02abdcf63
Fix typos in documentation ( #7076 )
2025-10-06 11:43:36 +02:00
1c6ba00fd0
Move LogCounterHandler into a new extension ( #7046 )
...
* Move LogCounterHandler into Crawler.crawl().
* Add the LogCount extension.
* Update test_spider_custom_settings_log_level().
* Typo.
* Update docs.
2025-10-03 20:14:11 +05:00
2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. ( #7037 )
...
* Deprecate the spider arg of downloader middleware methods.
* Add a test for deprecated spider args in downloader mws.
2025-09-02 21:51:47 +05:00
ed63fa94d6
Deprecate spider args of middlewares and pipelines. ( #7006 )
...
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
b68330811b
Deprecate spider arguments of StatsCollector methods ( #7011 )
2025-08-20 09:52:22 +02:00
89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() ( #6842 )
...
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting ( #6921 )
...
* Deprecate CONCURRENT_REQUESTS_PER_IP setting
* Typing fix
* PR review
* tests
* oops
* Update scrapy/settings/default_settings.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* update logic
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-06-30 16:51:05 +05:00
3843091c5f
Mark abstract base classes as such. ( #6930 )
...
* Mark abstract base classes as such.
* Remove an unneeded method.
* Fix exporter test coverage.
2025-06-30 15:14:34 +05:00
3019393686
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:04:40 +05:00
95172659af
Add sphinx-lint. ( #6920 )
2025-06-28 01:37:20 +02:00
daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts ( #6888 )
...
* Remove the chdir fixture.
* Re-enable fancy pytest asserts.
* Remove doc files from pytest ignores.
* Restore docs/_ext in test collection ignores.
* Skip a doctest that fails on Windows.
* Fix tests that were writing to the current dir.
2025-06-14 14:01:20 +05:00
3aa5e75787
Use AsyncCrawlerProcess in commands. ( #6845 )
...
* Use AsyncCrawlerProcess in commands.
* Ignore coverage of abstract methods.
* Address feedback.
2025-06-03 17:47:52 +05:00
e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. ( #6824 )
...
* Don't use CrawlerProcess in the commands that don't need it.
* Use a dummy spider loader in runspider.
2025-05-28 19:57:33 +05:00
05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` ( #6817 )
...
* Add a basic Crawler.crawl_async().
* Add custom loop tests for *CrawlerRunner.
* Add AsyncCrawlerProcess.
* Update related docs.
* Update practices.rst.
* Address test failures.
* Add a note about AsyncCrawler* to the docs about switching reactors.
* Address feedback.
* Update for TID253.
* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
2025-05-28 18:55:44 +05:00
dceb85bf3e
Add is_asyncio_available(). ( #6827 )
...
* Add is_asyncio_available().
* Print unexpected warnings in test_install_asyncio_reactor().
* Fix printing warnings.
* Fix printing warnings - 2.
* Skip TestDeferredToFuture on non-asyncio.
* Test the is_asyncio_available() exception.
2025-05-28 14:46:39 +05:00
f2fc177f1f
Fix a wrong versionadded usage. ( #6822 )
2025-05-23 11:06:33 +02:00
ff7d29654a
Fix typo in documentation and code: 'needs_backoff' -> 'needs_backout' ( #6815 )
...
Corrected the typo in the code and documentation where 'needs_backoff' was incorrectly used instead of 'needs_backout'.
2025-05-20 07:51:31 +02:00
3c2cd53abb
Skip the doctest.
2025-05-15 22:17:37 +05:00
bf1bfaaa3e
Slight improvements for the signal docs.
2025-05-15 20:02:38 +05:00
1ddcb568e2
Add send_catch_log_async().
2025-05-15 15:45:55 +05:00
82acef3051
Add AsyncCrawlerRunner. ( #6796 )
2025-05-14 16:21:18 +02:00
0ce693dfa9
Update VERSION strings.
2025-05-08 13:23:19 +05:00
b07e6d4ea8
Merge remote-tracking branch 'origin/master' into relnotes-2.13.0
2025-05-08 13:17:47 +05:00
e86b5051c2
Merge commit '373e501' into relnotes-2.13.0
2025-05-08 10:55:57 +05:00
5f6d1b464b
Cover up to 373e501 in the release notes.
2025-05-08 10:55:40 +05:00
036f3e5627
Support asynchronous start requests ( #6729 )
2025-05-07 19:04:03 +02:00
474f8312ff
Merge pull request #6776 from wRAR/deferred-apis
...
Add a list of Deferred-only APIs.
2025-05-05 21:28:50 +04:00
acb5f895cd
Update docs/topics/coroutines.rst
...
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-05-05 22:28:36 +05:00
523fc25c4d
Document default values set by startproject ( #6775 )
2025-05-05 18:51:15 +02:00
b93290f28a
Add a list of Deferred-only APIs.
2025-05-05 19:38:04 +05:00
095140f134
Cover the current master in the release notes (up to daf9db7).
2025-04-25 19:43:29 +05:00
daf9db72b2
Base class for universal spider middlewares ( #6693 )
...
* Initial BaseSpiderMiddleware.
* Rename the new methods.
* Remove the spider argument from new BaseSpiderMiddleware methods.
* Add docs for BaseSpiderMiddleware.
* Silence pylint.
* Add BaseSpiderMiddleware tests.
* Add a release note.
2025-04-23 19:29:04 +05:00
3ca882fba8
Syntax Error Fixed ( #6738 )
...
* Syntax error fix issue #6731
* test case added
* extra logic removed
* mock spider fixture
* Update scrapy/utils/misc.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* settings.rst updated
* settings.rst updated
* settings.rst updated
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-03-20 17:02:10 +05:00
9057bf4e1e
More docs about Deferred<->Future interoperability. ( #6734 )
2025-03-15 11:47:16 +01:00
fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). ( #6732 )
2025-03-14 20:46:17 +01:00
d0dabbc097
Enable AsyncioSelectorReactor by default. ( #6713 )
...
* Enable AsyncioSelectorReactor by default.
* Improve get_crawler(), switch more tests to it.
* Fix the remaining default-reactor test failures.
* Address documentation feedback.
* Make pinned envs more consistent.
2025-03-12 01:18:30 +05:00
ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) ( #6723 )
2025-03-11 14:53:04 +01:00
5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call ( #6719 )
2025-03-11 14:52:48 +01:00
9b7db1a068
Move some reference docs of Request to the code ( #6721 )
2025-03-11 12:43:50 +01:00
bee74fb753
Remove trailing whitespace
2025-03-11 11:56:44 +01:00
d2e5486d5a
Remove the Splash recommendation.
2025-03-07 13:20:42 +05:00
0c9200094e
Extend BaseSettings with utils for add-ons ( #6614 )
2025-03-05 10:31:59 +01:00
c200458f24
Add more docs for updating sync spider middlewares. ( #6688 )
2025-02-25 00:16:23 +01:00
2eb3c75c69
Remove AjaxCrawlMiddleware mention from built-in downloader middleware
2025-02-05 13:16:51 -03:00
76a8badd24
Add a deprecation notice to the AjaxCrawlMiddleware docs.
2025-02-03 14:55:10 +05:00
393ff96e45
Deprecate AjaxCrawlMiddleware.
2025-02-03 14:08:05 +05:00
9bc0029d27
Allow updating pre-crawler settings from add-ons ( #6568 )
2025-01-22 12:07:44 +01:00
ca345a3b73
Flexible severity of logging level when items are dropped ( #6608 )
2025-01-15 11:08:18 +01:00
98ba61256d
Deprecate BaseDupeFilter.log() and improve dupefilter docs ( #4151 )
...
* Remove BaseDupeFilter.log()
It is never called because request_seen() always returns False
* Document the interface of DUPEFILTER_CLASS classes
* Remove unnecessary BaseDupeFilter comments and add a short class description
* Improve the documentation related to the DUPEFILTER_CLASS setting
* Deprecate BaseDupeFilter.log
* Update the docs
* Fix the new code example
* Remove typing to keep the example short
Otherwise, it would have required yet another import line (from __future__ or typing).
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-01-14 19:36:56 +05:00
1fc91bb462
new `allow_offsite` parameter in OffsiteMiddleware ( #6151 )
...
* new 'allow_offsite' parameter in OffsiteMiddleware
* document deprecated dont_filter flag in OffsiteMiddleware
* avoid deprecating dont_filter in OffsiteMiddleware
* Copy the code to the downloader mw.
* Add tests for allow_offsite in the downloader mw.
* Mark allow_offsite with reqmeta.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-01-08 21:28:51 +05:00
59fcb9b93c
Improve internal refs to scrapy.Request and scrapy.Selector ( #6526 )
...
* Improve internal refs to scrapy.Selector.
* Improve internal refs to scrapy.Request.
* More scrapy.http fixes.
* Fix FormRequest refs.
* More fixes.
* Simplifications.
* Last fixes.
* Add the parsel intersphinx.
2025-01-07 16:18:18 +05:00
21b9ba717c
Allow customizing logged software versions ( #6582 )
...
Co-authored-by: Grammy Jiang <grammy.jiang@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-12-16 14:46:23 +01:00
10089c6fe2
2.12 release notes ( #6226 )
...
* Cover 2.12 in the release notes up to 9bb973dc54642af407a0a34bb4bad97#6463 , #6507 , #6511 to the 2.12 release notes.
* Cover 2.12 in the release notes up to d85c39fd215669#6527 in the release notes.
* Address PR feedback.
* Cover recent PRs.
* Finalize the 2.12.0 release notes, small additional fixes.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-11-18 13:07:32 +05:00
ab5cb7c7d9
Merge pull request #6540 from wRAR/build_from_settings
...
Remove build_from_settings() and deprecate from_settings() code
2024-11-14 20:40:54 +05:00
261c4b61dc
Enhancement: Update docs to include IgnoreRequest details ( #6506 )
...
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2024-11-13 22:47:39 +05:00
499e7e8aa6
Add from_crawler() to components that only had from_settings().
2024-11-12 21:12:32 +05:00
dc3ebb6cf7
Refactor the docs
2024-11-07 16:38:48 +01:00
2a4b7fe0f8
dont_throttle → autothrottle_dont_adjust_delay
2024-11-07 16:17:16 +01:00
b244ea7ac0
Add the missing import to the docs example
2024-11-05 20:05:58 +01:00
5862216bb1
Fix docs example
2024-11-05 19:55:28 +01:00
f57fc454be
Replace Slot.throttle with Request.meta['dont_throttle']
2024-11-05 19:44:30 +01:00
e7f5ae0b34
Update the outdated Item docstring. ( #6427 )
...
* Update the outdated Item doscstring.
* Fix the reference links in items.html.
2024-11-04 16:17:56 +05:00
ce5a132f12
Run and fix linkcheck. ( #6524 )
2024-11-04 11:40:07 +01:00
7701e590fb
Documentation added for Spider State in extensions.rst ( #6522 )
...
* Documentation added for Spider State in extensions.rst
* Made correction in documentation for Spiderstate
* Added appropriate intro for Spider state extension
* Added reference for spiderstate extension
* Added Spiderstate extension hyperlink refrence in jobs.rst
2024-11-02 10:45:27 +05:00
d85c39f5bc
Deprecation removals. ( #6500 )
...
* Deprecation removals.
* Clean up the default pytest filterwarnings.
* Remove test_get_images_old().
* Redo boto-requiring test filtering.
* Remove an unused function.
* Improve the Crawler.crawl() error message.
* Fix the test.
2024-10-31 18:06:22 +05:00
5759b3f0f2
Drop Reppy.
2024-10-17 23:41:23 +05:00
46cddc6ecf
Ignore SyntaxError as well when SPIDER_LOADER_WARN_ONLY is set to True ( #6484 )
2024-10-02 10:04:03 +02:00
4f9b2343c0
Merge branch 'master' of github.com:Laerte/scrapy
2024-08-27 07:16:10 -03:00
3c2a9fa262
update docs
2024-08-27 07:16:01 -03:00
f68f29dd13
Update docs/topics/contracts.rst
2024-08-26 23:37:57 -03:00
b85e5a66ed
Add support for meta in Spider Contracts
2024-08-26 23:21:09 -03:00
6ce0342beb
Allow yielding items from start_requests ( #6417 )
...
Co-authored-by: Georgiy Zatserklianyi <george.zatseklyany@gmail.com>
Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-08-26 20:53:06 +02:00
af15bd1dad
minor changes to items section of docs ( #6462 )
2024-08-19 13:55:09 +02:00
b9ef1326a5
Proofread the commands documentation ( #6449 )
2024-08-01 09:29:11 +02:00
03a15ced4f
Do not suggest logging dropped items twice ( #6448 )
...
Co-authored-by: Kevin Lloyd Bernal <kevinoxy@gmail.com>
2024-07-31 11:37:19 +02:00
026d606528
clean closespider_pagecount_no_item docs section
...
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-07-12 08:09:03 -03:00
8646d2ec7b
fix docs detail
2024-07-10 11:44:44 -03:00
59782d7308
update docs
2024-07-10 11:08:22 -03:00
365c9e62ad
Removing empty example reference ( #6402 )
...
Co-authored-by: Michael Duane Mooring <mikeumus@gmail.com>
2024-06-17 11:37:11 +02:00
144ff6c756
Document missing parts of response.json method
2024-06-05 21:09:10 -03:00
62c89aaf05
Update docs/topics/practices.rst
...
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-05-23 07:00:45 -03:00
8ec67ca230
Update docs/topics/practices.rst
...
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-05-23 07:00:35 -03:00
17e623cf0c
Update docs/topics/practices.rst
...
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-05-23 07:00:24 -03:00
3f66b66e3f
fix: checks
2024-05-22 22:01:55 -03:00
dc6a495fee
Update docs/topics/practices.rst
...
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-05-22 18:51:02 -03:00
8210fae25a
Update docs/topics/practices.rst
...
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-05-22 18:50:50 -03:00
e676cd3ce0
docs: Remove top-level reactor imports from CrawlerProces/CrawlerRunner examples
2024-05-22 07:55:53 -03:00
812fd2368f
Allow user-defined secure cookies ( #6357 )
2024-05-15 11:48:43 +02:00
d2f1e00a6a
Merge 2.11.2 changes ( #6363 )
2024-05-14 18:54:11 +02:00
fe163d98ea
Fix typo
2024-05-08 20:41:20 +02:00
2e13a9b8e1
Update METAREFRESH_IGNORE_TAGS default value in documentation
2024-05-08 23:33:13 +05:30
3590a1f66b
Closes #6342 . Update documentation on METAREFRESH_IGNORE_TAGS default value change
2024-05-08 23:23:17 +05:30
180bc9bad7
Closes #6342 . Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default
2024-05-08 22:06:46 +05:30
5f67c01d1d
chore: fix some typos in comments ( #6317 )
...
Signed-off-by: TechVest <techdashen@qq.com>
2024-04-17 10:56:26 +02:00
1d11ea3a54
Update practices.rst
2024-04-11 12:19:32 +02:00
d7581c6b41
Allow disabling the AutoThrottle extension for a given slot ( #6246 )
2024-03-12 09:44:29 +01:00
642af40704
Full typing for scrapy/extensions, part 2. ( #6279 )
2024-03-11 10:09:09 +01:00
282767f23b
Bump black.
2024-02-28 14:30:38 +05:00
2d46b4acf5
Complete coverage for the AutoThrottle extension ( #6245 )
2024-02-27 09:28:02 +01:00
6fc7827042
Do not close the underlying file from compression plugins ( #6239 )
2024-02-21 14:27:42 +01:00
ee1189512f
Replace urlparse with urlparse_cached where possible ( #6229 )
2024-02-20 12:47:29 +01:00
36f72877ba
update: docs/topics/selectors.rst
2024-02-16 10:39:16 -06:00
3e7b704c08
update: docs/topics/selectors.rst
2024-02-15 16:59:32 -06:00
660e3b1953
update: docs/topics/items.rst
2024-02-15 16:59:31 -06:00
6bd45bb6d9
Merge remote-tracking branch 'scrapy/2.11' into 2.11-port
2024-02-15 06:39:50 +01:00
479619b340
Merge branch '2.11-redos' into 2.11
2024-02-14 17:18:26 +01:00
809bfac489
Merge branch '2.11-compression-bomb' into 2.11
2024-02-14 17:16:25 +01:00
24634f1bb2
Attend PR comments
2024-01-31 12:29:43 -03:00
bacaf0db7a
Update documentation
2024-01-31 10:14:48 -03:00
d25cfe5315
Add JsonResponse ( #6174 )
2023-12-21 10:36:21 +01:00
48a9a58ff2
Link to Zyte’s export guides
2023-12-20 21:43:21 +04:00
369712ee50
SPM → Zyte API ( #6163 )
2023-12-20 21:43:13 +04:00
b095dd218f
Extend Request.meta documentation ( #5565 )
2023-12-20 21:43:03 +04:00
Andrey Rakhmatullin
Kailesh
Ahmed Arshad
Adrian
Julian Hamze
Yuval Elbar
Anant Murmu
Martin Schimandl
Michael
Adrián Chaves
Adham 3llam
Thalison Fernandes
Sebastien Dionne
Felipe Benevolo
Laerte Pereira
Keval Sakhiya
Mehraz Hossain Rumman
Laerte Pereira
guillermo-bondonno
anubhav
Kevin Lloyd Bernal
Robert Palmer
Rohit Kumar Singh
mmoriniere
Georgiy Zatserklianyi
mlmsmith
guillermo-bondonno
aisha-partha
TechVest
Jalil SA