Commit Graph

37 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 61f99f2df1
More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
Andrey Rakhmatullin 90deebe75e
Convert tests that fail with testfixtures 12.0.0 (#7545) 2026-06-03 09:16:24 +02:00
Albert Eduardovich N. 58d85282cf
chore: more ruff rules and overall minor improvements (#7386)
* overall prettyfication

- don't create empty mutable containers (lists, dicts) where it is appropriate
- removed from ignore section and applied some rules from ruff (but keep them ignored in tests)
- use `deque` in `_AsyncCooperatorAdapter` instead of `list.pop(0)`
- remove `f` prefix from strings without any formatting

* return `SIM300` to ignore

* apply ruff rules to all files

* another one

* remove extra space in pyproject.toml

* simplify `__getattr__` in `scrapy.utils.url`

* lazy `url_is_from_any_domain` and `url_is_from_spider`

* improve typing

* specify `arg_to_iter` with None as arg
2026-04-03 19:00:13 +05:00
Andrey Rakhmatullin ccfa052fa1
Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
Andrey Rakhmatullin 11977afba5
Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
Andrey Rakhmatullin c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. (#7233)
* Move no-reactor to pytest-asyncio.

* Unify the style.

* Skip a test that installs a reactor in no-reactor.

* Fix pinned env deps.

* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
Andrey Rakhmatullin 3186ccf5d5
Reactorless tests 1: the test env (#6952)
* Reactorless tests.

* Updates after merging master.

* Update a new import.

* Updates after merging master.

* Update testenv:no-reactor-pinned.

* Foundations for the reactorless mode.

* Fixes after merging master.

* Updates after merging reactorless-base.

* Add simple subprocess tests for reactorless AsyncCrawler*.

* More reactorless tests.

* Refactor AsyncCrawlerProcess.start().

* More checks.

* Fix test_reactorless_import_hook.

* More tests.

* Call install_reactor() before asyncio.run().

* Cleanup.

* Rephrase.

* Fix a test.

* Fixes.

* Rephrasing.

* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.

* Fix a new import.

* Clarify --reactor marks.

* Cleanup.

* More docs/comments.

* Update test_fallback_workflow().

* Update TestDeferredFFromCoroF.

* Typing improvements.
2026-02-02 19:25:30 +05:00
Andrey Rakhmatullin ed63fa94d6
Deprecate spider args of middlewares and pipelines. (#7006)
* Deprecate spider args of spider middleware methods.

* Deprecate the spider arg of pipeline process_item().

* Fix a typing issue.

* Deprecate the spider arg of pipeline {open,close}_spider().

* Cleanup.

* Update docs.

* Add pragma: no cover to some deprecated code.

* More tests.

* More tests.

* More tests.

* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
Andrey Rakhmatullin 8c8f4ff033
Extract in-process test HTTP servers into separate processes (#6960)
* Switch TestEngineBase to mockserver.

* Split mockserver.py.

* Stop using WrappingFactory.

* Remove keyfile/certfile fields from TestHttpBase.

* Move webclient-only resources to test_webclient.py.

* Refactor MockServer.url().

* Migrate TestHttpBase to mockserver.

* Migrate TestSimpleHttpsBase to a separate mockserver.

* Rename TestHttpMockServerBase.

* Migrate TestHttpProxyBase to a separate mockserver.

* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
Andrey Rakhmatullin 6b2997af90
Migrate to pytest-twisted (#6938)
* Migrate to pytest-twisted (WIP)

* Some typing fixes.

* Make --reactor=asyncio the default again.

* Try installing the correct event loop policy in tests on Windows.

* Make reactor_pytest a normal fixture.

* Fix test warnings.

* Fix FTPDownloadHandler teardown.

* Cleanups, typing.

* More cleanup.

* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
Andrey Rakhmatullin db0be1771c
Test refactoring, part 2. (#6928) 2025-06-29 23:10:38 +02:00
Andrey Rakhmatullin d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873) 2025-06-06 22:59:09 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin 3ded1dfe31
Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680)
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin 783b98deda Make mockserver instances per-class. 2025-02-02 14:10:09 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin b4acf5c827
Fix and remove most of the entries from the mypy ignore list (#6137) 2023-11-07 09:34:35 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677)
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
2022-11-27 11:00:13 +01:00
Mikhail Korobov f0c8d31193
Merge pull request #4694 from Jgaldos/improve-httpstatus-all-meta
Improve http status all on http error middleware
2021-04-01 22:43:45 +05:00
Jose Galdos 13181ba788 Improve http status all on http error middleware 2020-08-06 20:43:16 -05:00
Marc Hernández 336f19f5cc
Change super syntax (#4707) 2020-08-04 20:42:01 +02:00
Eugenio Lacuesta be39f274b8
Merge branch 'master' into flake8-remove-e128 2020-05-08 18:33:29 -03:00
Eugenio Lacuesta c2c3054ac1
Flake8: remove E121 2020-05-08 16:43:47 -03:00
Eugenio Lacuesta bf56517abf
Flake8: Remove E128 (tests, part 2) 2020-05-06 19:18:29 -03:00
Eugenio Lacuesta fe0c582ee0
Flake8: remove E127 in tests (continuation line over-indented for visual indent) 2020-05-06 09:49:10 -03:00
Vostretsov Nikita b364d27247 [MRG+1] Automatic port selection for servicies in unit tests (#3210)
* ability to pass port as a parameter

* try to find free ports

* use environment variables to pass mock server address

* get mock server address from environment variables

* ability to select ports for proxy in runtime

* use common method for URLs from mock server

* https support

* get mock server address

* get mock address

* replace hand-written mechanism by kernel-based one

* use ephemeral ports in mockserver

* strip EOL from addresses

* use ephemeral port in proxy

* no need to restore environment as it is restored in tearDown

* decode bytes

* use mockserver address as a variable

* ability to pass address as variable

* per test-case mockserver

* use base class

* remove obsolete environment manipulation

* return usage of proxy for http cases

* common method for broking proxy auth credentials

* python version-independent url methods
2018-05-23 21:25:50 +03:00
Chomba Ng'ang'a 4ca61a2051 Update deprecated test aliases
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``

https://docs.python.org/2/library/unittest.html#deprecated-aliases
2017-08-09 12:52:10 +02:00
Mikhail Korobov 26c1256f95 HttpErrorMiddleware stats 2017-02-22 17:36:25 +05:00
Paul Tremberth 81a0e3cd93 Raise log level for HttpErrorMiddleware to INFO (from DEBUG)
Fixes GH-910
2016-09-20 13:44:21 +02:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina c2d716807a Use LogCapture in testfixtures package for tests
This allows to remove `get_testlog` helper, `flushLoggedErrors` from
twisted.trial.unittest.TestCase and Twisted log observers created for
each test on conftest.py.
2015-04-22 17:24:40 -03:00
Julia Medina 870438e5f4 Update tests utils, fixing get_crawler and removing docrawl 2014-08-12 14:02:56 -03:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00