Commit Graph

118 Commits

Author SHA1 Message Date
Janit Rajkarnikar 3fc7148c5e
Don't evaluate annotations when inspecting signatures (#7818)
Since Python 3.14 (PEP 649) annotations are evaluated lazily, so
inspect.signature() raises NameError for callables whose annotations
reference names imported only under TYPE_CHECKING. This broke
middleware registration (via argument_is_required()) and custom stats
collectors (via _warn_spider_arg) for user code with such annotations.

Use annotation_format=Format.FORWARDREF on 3.14+: parameter names,
kinds and defaults are unchanged, and unresolvable annotations become
ForwardRef proxies instead of raising.

Resolves #7796.
2026-07-30 20:18:45 +05:00
Andrey Rakhmatullin 628a3afbbd
Remove deprecated xmliter(), deprecate re_rsearch() (#7765) 2026-07-23 10:38:56 +02:00
Andrey Rakhmatullin 4e956bd2de
Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs (#7575)
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.

* Update the docs.

* Trim the tables.

* Restore lost wording.

* Handlers docs improvements and fixes.
2026-06-09 01:10:44 +05:00
Andrey Rakhmatullin a8a8f20d9c
Remove support for sync process_spider_output(). (#7504)
* Remove support for sync process_spider_output().

* Slight check fix.

* Don't expect exceptions from calling process_spider_output().

* Remove dead code.

* Fix test_deprecated_mw_spider_arg() to run all methods.

* Fix typing.

* Typos.

* Remove references to the removed section, move the universal section to the spider middleware docs and write it from a different perspective

* Fix indentation issue

* Minor rewordings

* older versions → lower versions

* Update news.rst

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-05-15 16:36:05 +05:00
Albert Eduardovich N. 9776a72a6a
chore: simplify some code, get rid of nested fns where it's makes sence + pylint (#7401)
* chore: simplify some code, get rid of nested fns where it's makes sence

reasoning: https://wemake-python-styleguide.readthedocs.io/en/latest/pages/usage/violations/best_practices.html#wemake_python_styleguide.violations.best_practices.NestedFunctionViolation

* chore: address some pylint rules

* remove `pylint: disable=deprecated-class` from `reactor.py`

* fix typo

* fix another typo :)

* revert backward incompatible arg renames

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 23:32:46 +05:00
Adrian Chaves b6e5c58ae7 Fix arbitrary callable execution from the Referrer-Policy response header 2026-01-19 16:44:03 +01:00
Andrey Rakhmatullin 7bef98b4f1
Release notes for 2.14.0 (#7035)
* Initial release notes for 2.14.0, up to ed63fa9.

* Cover 2.14 in the release notes up to 393d715.

* Cover 2.14 in the release notes up to eb49647.

* Cover 2.14 in the release notes up to 426aafd.

* Cover 2.14 in the release notes up to 1e8de24.

* Cover 2.14 in the release notes up to 5a7e132.

* Bump sphinx-lint.

* Finalize the 2.14.0 release notes.

* Drop more of the old versionadded directives.

* Address feedback.
2026-01-05 14:51:05 +05:00
Andrey Rakhmatullin 61b4befc60
Remove code deprecated in 2.12.0 (#7126)
* Remove smaller things.

* Remove from_settings().

* Remove old MediaPipeline init code.

* Cleanup.

* Typo.
2025-10-27 17:14:38 +05:00
Andrey Rakhmatullin d414d393d4
Remove Python 3.9 support (#7121) 2025-10-27 12:37:49 +01:00
Andrey Rakhmatullin baa579df62
Deprecate various `spider` args in `MiddlewareManager` subclasses (#6984)
* Add a crawler argument to MiddlewareManager.__init__().

* Deprecate the spider argument in some MiddlewareManager subclasses.

* Remove the spider argument from ExecutionEngine.open_spider_async().

* Remove the spider argument from Scraper.open_spider().

* Fix the Scraper.close_spider() type hint.

* Remove the spider argument from Downloader internals.

* Deprecate the spider arg of download_func.

* Deprecate the spider arg of Downloader.fetch().

* Deprecate the spider arg of ItemPipelineManager.process_item().

* Fix pylint.

* Tests for MiddlewareManager spider code.

* Deprecated spider arg handling for custom pipeline managers.

* Add a test for the deprecated spider arg in a custom downloader.

* Remove TODO.

* Fix pylint.
2025-08-05 12:21:08 +05:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin ede9e9c3c3 Use full method names in all spidermw log messages. 2025-02-11 23:07:25 +05:00
Andrey Rakhmatullin cec0aeca58
Bump ruff, switch from black to ruff-format (#6631) 2025-01-27 11:07:09 +01:00
Andrey Rakhmatullin 1e4c81e9dc Add Perflint rules to ruff. 2024-12-12 21:03:34 +05:00
Andrey Rakhmatullin cde0845ab2
Ruff: migrate pyupgrade and bandit, enable some other rules (#6577) 2024-12-10 19:53:27 +01:00
Adrián Chaves 10089c6fe2
2.12 release notes (#6226)
* Cover 2.12 in the release notes up to 9bb973dc54

* Add one more highlight

* Better merge of the news entries.

* Cover 2.12 in the release notes up to 642af40.

* Cover 2.12 in the release notes up to 7a0a34b.

* Cover 2.12 in the release notes up to b4bad97.

* Add not yet merged PRs #6463, #6507, #6511 to the 2.12 release notes.

* Cover 2.12 in the release notes up to d85c39f, small fixes.

* Cover 2.12 in the release notes up to d215669.

* Cover #6527 in the release notes.

* Address PR feedback.

* Cover recent PRs.

* Finalize the 2.12.0 release notes, small additional fixes.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-11-18 13:07:32 +05:00
Andrey Rakhmatullin b042ad255d
Address some previously ignored pylint messages. (#6531) 2024-11-11 11:49:52 +01:00
Rohit Kumar Singh d2156696c4
Deprecate unused scrapy utils (#6519)
* Added deprecation warnings for unused Scrapy.utils

* Grammatical corrections

* Exceptions class connected

* Deprecation of ScrapyJSONDecoder

* request_authenticate function deprecation

* Making all warning similar

* Added ignore statements for deprecation warning in tests

* Missing stacklevel attr. added

* Added Deprecation message
2024-11-04 21:09:45 +05:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin 3d8dbd5648 flake8-bugbear 2024-06-20 00:27:38 +05:00
Andrey Rakhmatullin 2e214210f6 Add parameters to iterable generics, replace generators with iterables. 2024-06-05 21:44:04 +05:00
Andrey Rakhmatullin 019f23e3b7 Add parameters to some of typing.Callable. 2024-06-02 18:42:01 +05:00
Andrey Rakhmatullin 751c91e614 Add parameters to misc generics. 2024-05-31 21:57:14 +05:00
Andrey Rakhmatullin 98c755e5fb Add parameters to typing.List. 2024-05-31 21:20:22 +05:00
Andrey Rakhmatullin da42e8f124 Add parameters to typing.Dict. 2024-05-31 21:11:50 +05:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Andrey Rakhmatullin 4ed5c5ae91
Use ParamSpec for callables. (#6353) 2024-05-13 12:01:52 +02:00
Laerte Pereira 63acd07209 Fix and re-enable unnecessary-comprehension and use-dict-literal pylint tags 2024-02-28 16:14:08 -03:00
Andrey Rakhmatullin 282767f23b Bump black. 2024-02-28 14:30:38 +05:00
Rotzbua 0d445a3224
refactor(yield): use `yield from` syntax 2024-01-05 12:30:10 +01:00
Andrey Rakhmatullin d400f1ac06 Add more typing to scrapy/utils/python.py. 2023-05-07 18:35:24 +04:00
Andrey Rakhmatullin ea299dfd7c Add typing to scrapy/utils/misc.py. 2023-05-07 01:48:09 +04:00
Andrey Rakhmatullin 02f3e8d413
Typing for scrapy/core/downloader (#5896) 2023-04-17 08:37:52 +02:00
Serhii A c2a31974ff
Improve get_func_args (#5885) 2023-04-13 11:44:20 +02:00
Andrey Rakhmatullin a1717aa48c Fix most of FormRequest typing issues. 2023-02-12 00:42:39 +04:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677)
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
2022-11-27 11:00:13 +01:00
Laerte Pereira d5b6c236a9
Remove deprecated code (#5719) 2022-11-21 09:32:26 +01:00
Andrey Rakhmatullin fa58ab21e4 Replace _getargspec_py23() with inspect.getfullargspec(). 2022-10-06 20:59:06 +06:00
Andrey Rakhmatullin f789547551 Implement spider middleware iterable upgrade/downgrade. 2022-01-11 18:28:32 +05:00
Andrey Rakhmatullin 61197d3dba Add/update typing, cleanup iterator/iterable inconsistencies. 2021-04-14 18:21:58 +05:00
Andrey Rakhmatullin ffc6f525ce Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-04-14 17:34:58 +05:00
Andrey Rakhmatullin a9e96f9907 Add typing for middleware and coroutine related code. 2021-04-03 17:40:45 +05:00
Andrey Rakhmatullin 7e9f498e00 Add MutableAsyncChain. 2021-02-05 16:21:56 +05:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512)
Co-authored-by: Yuval Hager <yhager@yhager.com>
2020-08-17 15:10:08 +02:00
Kshitij Sharma 983b7ddf2e aesthetic fixes 2020-08-05 16:13:52 +05:30
Kshitij Sharma 9d84289109 deprecated weakkeycache by specifying in __init__ 2020-08-05 09:11:59 +05:30
Kshitij Sharma 49337bd2ae Code cleanup scrapy.utils.python.WeakKeyCache #4684 and fixing ci alerts 2020-07-30 12:25:21 +05:30