Commit Graph

219 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 2798c03bb0
Add Python 3.14 to CI. (#6604)
* Add Python 3.14 (alpha3) to CI.

* Disable mitmproxy on 3.14 for now.

* 3.14.0-alpha.4.

* 3.14.0-alpha.5

* 3.14.0-beta2.

* 3.14 release.

* Fix test_non_pickable_object.

* Fix handling of file:/path feed URIs.

* Better mocking of streams for TextTestResult.

* Do not use .php in test_file_path() as it's now a known extension.

* Fix the URL in TestFeedExporterSignals.

* Fix typing.

* Bump more envs to 3.14.

* Silence pylint.

* Fix another test for .php handling change.

* Remove test_install_asyncio_reactor.

* More bumps to 3.14.

* Revert docs-tests to use 3.13.

* Debug options for Windows.

* Re-enable xdist.

* Revert Windows PYTEST_ADDOPTS.

* Silence loop policy deprecation warnings.

* Restore a lost pylint suppression.

* Update asyncio_enabled_reactor_same_loop.py to new Twisted.

* Fix RobotFileParser tests for Python 3.14.5.
2026-05-12 23:49:37 +05:00
Andrey Rakhmatullin 7fc84d372a
Cleanup/clarifications of proxy support and improvements for proxy tests (#7496)
* Run mitmproxy-based tests for every handler and improve them.

* Fixes for H2DownloadHandler.

* Revert "Fixes for H2DownloadHandler."

This reverts commit bcdbd097cd.

* Add TestHttpsProxy for HTTP11DownloadHandler.

* Use the configured context factory in ScrapyProxyAgent.

* Reword.

* Reword.

* pragma: no cover for H2DownloadHandler proxy code.

* Raise an exception for HTTPS proxies instead of using HTTP.

* Remove non-working proxy support and explicitly forbid HTTP.

* pragma: no cover

* Add a docs note about HTTPS proxies.

* Rename ScrapyProxyAgent.
2026-05-12 17:45:54 +05:00
Andrey Rakhmatullin 7e881ce2d7
Release notes for 2.15.0 (#7373)
* Cover 2.15.0 in the release notes (up to 31bf7c3).

* Replace VERSION.

* Document unified download handler exceptions.

* Updates.

* Cover 2.15.0 in the release notes (up to 0c6ccf5).

* Cover 2.15.0 in the release notes (up to b68f267).

* Run linkcheck and fix issues.

* Cleanup unused link targets.

* Cleanup.
2026-04-09 16:56:30 +05:00
Andrey Rakhmatullin 31bf7c3892
Docs for HttpxDownloadHandler, handle httpx not being installed (#7368)
* Add docs for HttpxDownloadHandler.

* Handle httpx not being installed.

* Revert test_not_configured_without_asyncio().

* Restore the comment.

* Cleanup.
2026-03-27 21:06:55 +05:00
Andrey Rakhmatullin 2347138ba4
Unified download handler exceptions (#7208)
* Add DownloadTimeoutError.

* Don't expect ConnectionAborted in tests.

* Add DownloadCancelledError.

* Add ResponseDataLoss.

* Add more download handler tests.

* Add DownloadConnectionRefusedError.

* Add UnsupportedURLScheme.

* Add CannotResolveHostError.

* Add DownloadFailedError.

* Remove wrapped Twisted exceptions from lists.

* Update a test expectation.

* Extract wrap_twisted_exceptions().

* Wrap TxTimeoutError.

* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
Andrey Rakhmatullin d1bd8eb49f
Async API for download handlers. (#7164) 2025-12-31 19:03:16 +01:00
Adham 3llam 9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation (#7198) 2025-12-22 10:00:39 +01:00
Andrey Rakhmatullin 588f3d4f65
Remove docs mentioning deprecated spider attrs. (#7176) 2025-12-15 09:51:56 +01:00
Andrey Rakhmatullin 5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws (#7069)
* Restore coverage in MiddlewareManager spider arg handling.

* Move MiddlewareManager._warn_spider_arg() tests.

* Add DownloaderMiddlewareManager.download_async().

* Make download_func async.

* Inline MiddlewareManager._warn_spider_arg().

* Deprecate returning deferreds from downloader middlewares.

* Update the downloader mw doc about async def.

* Use global_object_name() instead of repr().

* Update docs/topics/downloader-middleware.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update downloader-middleware.rst

---------

Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-12-09 20:07:36 +05:00
Andrey Rakhmatullin 2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. (#7037)
* Deprecate the spider arg of downloader middleware methods.

* Add a test for deprecated spider args in downloader mws.
2025-09-02 21:51:47 +05:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Laerte Pereira 2eb3c75c69 Remove AjaxCrawlMiddleware mention from built-in downloader middleware 2025-02-05 13:16:51 -03:00
Andrey Rakhmatullin 76a8badd24 Add a deprecation notice to the AjaxCrawlMiddleware docs. 2025-02-03 14:55:10 +05:00
Andrey Rakhmatullin 393ff96e45 Deprecate AjaxCrawlMiddleware. 2025-02-03 14:08:05 +05:00
Kevin Lloyd Bernal 1fc91bb462
new `allow_offsite` parameter in OffsiteMiddleware (#6151)
* new 'allow_offsite' parameter in OffsiteMiddleware

* document deprecated dont_filter flag in OffsiteMiddleware

* avoid deprecating dont_filter in OffsiteMiddleware

* Copy the code to the downloader mw.

* Add tests for allow_offsite in the downloader mw.

* Mark allow_offsite with reqmeta.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-01-08 21:28:51 +05:00
Andrey Rakhmatullin 59fcb9b93c
Improve internal refs to scrapy.Request and scrapy.Selector (#6526)
* Improve internal refs to scrapy.Selector.

* Improve internal refs to scrapy.Request.

* More scrapy.http fixes.

* Fix FormRequest refs.

* More fixes.

* Simplifications.

* Last fixes.

* Add the parsel intersphinx.
2025-01-07 16:18:18 +05:00
Adrián Chaves 10089c6fe2
2.12 release notes (#6226)
* Cover 2.12 in the release notes up to 9bb973dc54

* Add one more highlight

* Better merge of the news entries.

* Cover 2.12 in the release notes up to 642af40.

* Cover 2.12 in the release notes up to 7a0a34b.

* Cover 2.12 in the release notes up to b4bad97.

* Add not yet merged PRs #6463, #6507, #6511 to the 2.12 release notes.

* Cover 2.12 in the release notes up to d85c39f, small fixes.

* Cover 2.12 in the release notes up to d215669.

* Cover #6527 in the release notes.

* Address PR feedback.

* Cover recent PRs.

* Finalize the 2.12.0 release notes, small additional fixes.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-11-18 13:07:32 +05:00
Robert Palmer 261c4b61dc
Enhancement: Update docs to include IgnoreRequest details (#6506)
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2024-11-13 22:47:39 +05:00
Andrey Rakhmatullin ce5a132f12
Run and fix linkcheck. (#6524) 2024-11-04 11:40:07 +01:00
Andrey Rakhmatullin 5759b3f0f2 Drop Reppy. 2024-10-17 23:41:23 +05:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Adrián Chaves fe163d98ea
Fix typo 2024-05-08 20:41:20 +02:00
aisha-partha 2e13a9b8e1 Update METAREFRESH_IGNORE_TAGS default value in documentation 2024-05-08 23:33:13 +05:30
aisha-partha 3590a1f66b Closes #6342. Update documentation on METAREFRESH_IGNORE_TAGS default value change 2024-05-08 23:23:17 +05:30
aisha-partha 180bc9bad7 Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default 2024-05-08 22:06:46 +05:30
Andrey Rakhmatullin 39ee8d1ee2
Deprecate ReppyRobotParser (#6099) 2023-10-18 09:10:21 +02:00
Serhii A 777a6ea412
Make the retry middleware exception list configurable (#5929) 2023-06-16 15:46:06 +02:00
Andrey Rakhmatullin 76eba9977b
Skip or fix failing code block tests (#5826) 2023-02-14 09:40:38 +01:00
pankaj1707k 03f32c018f
Update black version and fix minor typos and punctuations 2023-02-03 11:52:20 +05:30
pankaj1707k c1bbb299d7
Add and run pre-commit hook 'blacken-docs'
Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
2023-02-01 16:30:57 +05:30
Mikhail Korobov f60c7ae768
Fixed heading levels in downloader middleware docs (#5567) 2022-07-20 11:01:22 +02:00
Víctor Ruiz 2227be7af6
Fix a typo in the HTTP cache documentation (#5455) 2022-03-22 15:21:16 +01:00
Eugene 9a28eb0bad
Suggest installing the brotli package instead of brotlipy (#4267) 2022-03-17 05:39:54 +01:00
Andrey Rahmatullin 0b0eea3601
Merge pull request #5419 from PendalF89/patch-2
Update downloader-middleware.rst
2022-02-15 10:57:35 +05:00
Abhishek K M 187b5c8876
Update the documentation link for robots.txt (#5415) 2022-02-14 19:16:53 +01:00
Boris Zabolotskikh bbb693d046
Update downloader-middleware.rst
Added a link to the method
2022-02-14 12:07:45 +03:00
Andrey Rakhmatullin 7ec5f299c4 Small documentation fixes. 2021-10-06 12:33:32 +02:00
Andrey Rakhmatullin b081f18a2f Add http_auth_domain to HttpAuthMiddleware. 2021-10-06 12:33:18 +02:00
databender 572d347b3b warning view updated 2021-08-18 16:17:52 +05:30
databender d623ed15d1 indentation updated 2021-08-18 14:51:03 +05:30
databender bcf38a6719 added upstream issue for not supported python version 2021-08-18 14:48:47 +05:30
databender ebddb77a33 updated suggested changes after review 2021-08-16 18:08:26 +05:30
databender cc1cb2de0c updated suggested changes 2021-08-16 17:21:47 +05:30
databender 1a8b98843a updated documentation for python version for reppy 2021-08-16 17:00:05 +05:30
databender 8bbaea9892 updated documentation for python version for reppy 2021-08-16 16:57:43 +05:30
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Adrián Chaves 6ab990181c Document get_retry_requests 2021-02-22 14:48:03 +01:00
Andrey Rahmatullin 9f8c3938cc
Merge pull request #4823 from elacuesta/cookies-revert-header
Do not process cookies from headers
2020-10-08 20:20:32 +05:00
Eugenio Lacuesta 137c8ba6ee
Docs: mention limitation about Cookie header 2020-10-06 10:50:17 -03:00
P. Chen da3171d4f7
Using the `zstandard` package than `zstd` for supporting frames both with and without the content size info
See also: https://github.com/sergey-dryabzhinsky/python-zstd/issues/53
2020-10-05 23:18:58 +01:00