RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

58 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 61f99f2df1
More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Andrey Rakhmatullin 55c17a8985
Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
Andrey Rakhmatullin a8a8f20d9c
Remove support for sync process_spider_output(). (#7504) 
* Remove support for sync process_spider_output().

* Slight check fix.

* Don't expect exceptions from calling process_spider_output().

* Remove dead code.

* Fix test_deprecated_mw_spider_arg() to run all methods.

* Fix typing.

* Typos.

* Remove references to the removed section, move the universal section to the spider middleware docs and write it from a different perspective

* Fix indentation issue

* Minor rewordings

* older versions → lower versions

* Update news.rst

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
 2026-05-15 16:36:05 +05:00
Andrey Rakhmatullin fc14a0ce59
Enable mypy warn_return_any. (#7492) 2026-05-04 22:24:47 +02:00
Andrey Rakhmatullin 11977afba5
Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
Andrey Rakhmatullin 3186ccf5d5
Reactorless tests 1: the test env (#6952) 
* Reactorless tests.

* Updates after merging master.

* Update a new import.

* Updates after merging master.

* Update testenv:no-reactor-pinned.

* Foundations for the reactorless mode.

* Fixes after merging master.

* Updates after merging reactorless-base.

* Add simple subprocess tests for reactorless AsyncCrawler*.

* More reactorless tests.

* Refactor AsyncCrawlerProcess.start().

* More checks.

* Fix test_reactorless_import_hook.

* More tests.

* Call install_reactor() before asyncio.run().

* Cleanup.

* Rephrase.

* Fix a test.

* Fixes.

* Rephrasing.

* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.

* Fix a new import.

* Clarify --reactor marks.

* Cleanup.

* More docs/comments.

* Update test_fallback_workflow().

* Update TestDeferredFFromCoroF.

* Typing improvements.
 2026-02-02 19:25:30 +05:00
Andrey Rakhmatullin 804ae167df
Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
Andrey Rakhmatullin dd0b071bcc
Deprecate sync methods of ItemPipelineManager/ITEM_PROCESSOR. (#7043) 2025-10-03 14:15:42 +02:00
Andrey Rakhmatullin ed63fa94d6
Deprecate spider args of middlewares and pipelines. (#7006) 
* Deprecate spider args of spider middleware methods.

* Deprecate the spider arg of pipeline process_item().

* Fix a typing issue.

* Deprecate the spider arg of pipeline {open,close}_spider().

* Cleanup.

* Update docs.

* Add pragma: no cover to some deprecated code.

* More tests.

* More tests.

* More tests.

* Simplify _process_parallel().
 2025-08-21 13:23:10 +05:00
Andrey Rakhmatullin baa579df62
Deprecate various `spider` args in `MiddlewareManager` subclasses (#6984) 
* Add a crawler argument to MiddlewareManager.__init__().

* Deprecate the spider argument in some MiddlewareManager subclasses.

* Remove the spider argument from ExecutionEngine.open_spider_async().

* Remove the spider argument from Scraper.open_spider().

* Fix the Scraper.close_spider() type hint.

* Remove the spider argument from Downloader internals.

* Deprecate the spider arg of download_func.

* Deprecate the spider arg of Downloader.fetch().

* Deprecate the spider arg of ItemPipelineManager.process_item().

* Fix pylint.

* Tests for MiddlewareManager spider code.

* Deprecated spider arg handling for custom pipeline managers.

* Add a test for the deprecated spider arg in a custom downloader.

* Remove TODO.

* Fix pylint.
 2025-08-05 12:21:08 +05:00
Andrey Rakhmatullin 32f2aede82
Deprecate non-async counterparts of new async APIs. (#6979) 
* Deprecate non-async counterparts of new async APIs.

* Deprecate Scraper.call_spider().

* Add _schedule_coro().
 2025-08-01 00:58:48 +05:00
Andrey Rakhmatullin 552f2fb91e
Generic improvements related to reactorless tests (#6968) 
* Skip doctests that import the reactor.

* Extract FTPDownloadHandler tests.

* Skip TestHttps2ClientProtocol via pytestmark.

* Replace some explicit reactor.callLater() in tests.

* Simplify TestRequestSendOrder._test_request_order().
 2025-07-25 13:35:00 +05:00
Andrey Rakhmatullin 6b2997af90
Migrate to pytest-twisted (#6938) 
* Migrate to pytest-twisted (WIP)

* Some typing fixes.

* Make --reactor=asyncio the default again.

* Try installing the correct event loop policy in tests on Windows.

* Make reactor_pytest a normal fixture.

* Fix test warnings.

* Fix FTPDownloadHandler teardown.

* Cleanups, typing.

* More cleanup.

* Update only_asyncio/only_not_asyncio mark messages.
 2025-07-06 21:27:17 +05:00
Andrey Rakhmatullin 16f168b406
Deprecate mustbe_deferred() and defer_*(). (#6937) 2025-07-04 16:41:15 +05:00
Andrey Rakhmatullin db0be1771c
Test refactoring, part 2. (#6928) 2025-06-29 23:10:38 +02:00
Andrey Rakhmatullin ac956f8595
Replace most of the @inlineCallbacks test helpers. (#6883) 2025-06-10 10:02:27 +02:00
Andrey Rakhmatullin d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873) 2025-06-06 22:59:09 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin 2a1edbd473
Remove usages of TestCase._wait(). (#6773) 2025-05-05 09:44:17 +02:00
Andrey Rakhmatullin 3ded1dfe31
Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680) 
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
 2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin d8978d405c Improve diagnostics for sync-only spider middlewares. 2025-02-09 01:33:27 +05:00
Andrey Rakhmatullin 7dd92e6e43
Add pylint rules to ruff, refresh the ignore list of pylint itself (#6584) 2024-12-16 09:44:36 +01:00
Andrey Rakhmatullin 1e4c81e9dc Add Perflint rules to ruff. 2024-12-12 21:03:34 +05:00
Andrey Rakhmatullin e7595837a6 Add flake8-raise rules to ruff. 2024-12-12 20:33:35 +05:00
Andrey Rakhmatullin 897e124a27 Add flake8-return rules to ruff. 2024-12-12 20:32:25 +05:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Georgiy Zatserklianyi 6ce0342beb
Allow yielding items from start_requests (#6417) 
Co-authored-by: Georgiy Zatserklianyi <george.zatseklyany@gmail.com>
Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-08-26 20:53:06 +02:00
Rotzbua 0d445a3224
refactor(yield): use `yield from` syntax 2024-01-05 12:30:10 +01:00
Andrey Rakhmatullin 824641e349 Merge remote-tracking branch 'origin/master' into mypy-imports 2023-02-11 22:02:19 +04:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Andrey Rakhmatullin afd5d85320 Fix remaining issues in tests. 2023-01-25 22:15:10 +04:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Andrey Rakhmatullin b21c16099e Fix flake8 issues. 2022-07-22 19:18:33 +05:00
Adrián Chaves c961438d5d tests: cover scenarios of bad results from process_spider_output 2022-03-16 18:45:56 +01:00
Andrey Rakhmatullin f789547551 Implement spider middleware iterable upgrade/downgrade. 2022-01-11 18:28:32 +05:00
Andrey Rakhmatullin 7306a81188 Disable builtin middlewares in spider middleware tests. 2021-07-19 20:09:11 +05:00
Andrey Rakhmatullin 30ed7fa349 Some cleanup, make sync middlewares fail earlier. 2021-04-02 22:20:56 +05:00
Andrey Rakhmatullin f422861ef4 Add more tests for spider middlewares. 2021-03-25 20:49:12 +05:00
Marc Hernandez Cabot 6fb85951ce fix E22X flake8 2020-02-21 08:39:14 +01:00
Andrey Rakhmatullin 92ffd2f249 Simplify some more imports. 2019-11-14 13:45:39 +05:00
Eugenio Lacuesta 40449fa0eb Update docs, add tests, remove FIXME comment 2018-08-03 18:20:25 -03:00
Eugenio Lacuesta 8c55f5eb15 Simplify check for re-raised exception. Add tests. 2018-08-03 15:16:26 -03:00
Eugenio Lacuesta d6d3e87e3a Rename test file 2018-07-27 14:47:52 -03:00
Eugenio Lacuesta 71a1406c99 Logging changes 2018-07-18 17:43:05 -03:00
Eugenio Lacuesta 6329441c82 ModuleNotFoundError was added in py3.6 2018-07-18 17:31:31 -03:00
Eugenio Lacuesta a3af0bfd56 More tests 2018-07-18 15:15:55 -03:00
Eugenio Lacuesta 56e92d90fd Update tests 2018-07-17 15:15:38 -03:00