fc14a0ce59
Enable mypy warn_return_any. ( #7492 )
2026-05-04 22:24:47 +02:00
6fe27ba33e
Add more no-reactor tests ( #7259 )
...
* Generic changes and scrapy bench.
* scrapy check.
* scrapy crawl.
* scrapy fetch.
* scrapy parse.
* scrapy runspider.
* scrapy shell.
* Skip httpx tests on default-reactor.
* Review requires_reactor marks.
* Make test functions that require an event loop async def.
* Improve test_pending_asyncio_tasks().
* Add Mac OS DNS error.
* Refactor most of test_scheduler.py.
* Finish refactoring DownloaderAwareSchedulerTestMixin.
* Refactor test_engine_loop.py.
* Add the no-reactor-extra-deps tox env, run no-reactor on macos.
* Skip unhandled CancelledError when shutting down the engine.
* Fix typing and pre-commit checks.
* Fix typing problems in master.
---------
Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
939db88b04
Disable mypy allow_untyped_calls. ( #7331 )
2026-03-24 10:31:17 +01:00
e02ad08672
Move to mypy --strict with exceptions. ( #7300 )
...
* Move to mypy --strict with exceptions.
* Fix typing-tests.
2026-03-02 15:47:23 +05:00
2e53d90e4c
Refactor HTTP download handlers ( #7238 )
2026-02-03 20:13:30 +01:00
3186ccf5d5
Reactorless tests 1: the test env ( #6952 )
...
* Reactorless tests.
* Updates after merging master.
* Update a new import.
* Updates after merging master.
* Update testenv:no-reactor-pinned.
* Foundations for the reactorless mode.
* Fixes after merging master.
* Updates after merging reactorless-base.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Fix a test.
* Fixes.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
* Fix a new import.
* Clarify --reactor marks.
* Cleanup.
* More docs/comments.
* Update test_fallback_workflow().
* Update TestDeferredFFromCoroF.
* Typing improvements.
2026-02-02 19:25:30 +05:00
2347138ba4
Unified download handler exceptions ( #7208 )
...
* Add DownloadTimeoutError.
* Don't expect ConnectionAborted in tests.
* Add DownloadCancelledError.
* Add ResponseDataLoss.
* Add more download handler tests.
* Add DownloadConnectionRefusedError.
* Add UnsupportedURLScheme.
* Add CannotResolveHostError.
* Add DownloadFailedError.
* Remove wrapped Twisted exceptions from lists.
* Update a test expectation.
* Extract wrap_twisted_exceptions().
* Wrap TxTimeoutError.
* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
8c8f4ff033
Extract in-process test HTTP servers into separate processes ( #6960 )
...
* Switch TestEngineBase to mockserver.
* Split mockserver.py.
* Stop using WrappingFactory.
* Remove keyfile/certfile fields from TestHttpBase.
* Move webclient-only resources to test_webclient.py.
* Refactor MockServer.url().
* Migrate TestHttpBase to mockserver.
* Migrate TestSimpleHttpsBase to a separate mockserver.
* Rename TestHttpMockServerBase.
* Migrate TestHttpProxyBase to a separate mockserver.
* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
6b2997af90
Migrate to pytest-twisted ( #6938 )
...
* Migrate to pytest-twisted (WIP)
* Some typing fixes.
* Make --reactor=asyncio the default again.
* Try installing the correct event loop policy in tests on Windows.
* Make reactor_pytest a normal fixture.
* Fix test warnings.
* Fix FTPDownloadHandler teardown.
* Cleanups, typing.
* More cleanup.
* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
4279f2837c
Enable the flake8-builtins rules. ( #6942 )
2025-07-05 21:43:18 +05:00
df342eee6e
Bump tool versions. ( #6941 )
2025-07-05 10:30:30 +02:00
b4d11b8b25
Further reduce deps on unittest. ( #6884 )
2025-06-11 01:28:09 +02:00
d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. ( #6873 )
2025-06-06 22:59:09 +02:00
916fe50974
Fix and ban the top-level twisted.internet.reactor imports. ( #6835 )
2025-05-28 15:53:52 +05:00
380c2279b9
Converting tests to plain asserts, part 7. ( #6710 )
2025-03-09 20:23:51 +01:00
8d92c28a16
Switch to pytest.raises(). ( #6680 )
...
* Switch to pytest.raises().
* Add matches= to broad pytest.raises().
* Adjust the test_nonserializable_object() regex for Python <= 3.11.
* Adjust the test_nonserializable_object() regex for PyPy.
* Adjust other test exception regexes for PyPy.
* Cleanup.
2025-02-27 23:13:04 +05:00
a898331d14
Preparations for switching to direct pytest. ( #6678 )
2025-02-16 20:28:58 +01:00
5391663072
Drop Python 3.8 Support ( #6472 )
2024-10-16 10:03:16 +02:00
d08f559600
Add flake8-type-checking. ( #6413 )
2024-06-25 10:20:59 +02:00
b0ef9a89a1
Remove usage of deprecated mktemp ( #5285 )
2024-02-26 19:53:06 +01:00
01d9d28324
Full typing for scrapy/http/response/__init__.py.
2023-11-01 01:10:32 +04:00
5807970a22
Add typing hints to CaselessDict and Headers. ( #6097 )
2023-10-17 19:07:13 +02:00
824641e349
Merge remote-tracking branch 'origin/master' into mypy-imports
2023-02-11 22:02:19 +04:00
a5c1ef8276
sort imports with isort
2023-01-25 14:25:15 -06:00
afd5d85320
Fix remaining issues in tests.
2023-01-25 22:15:10 +04:00
e211ec0aa2
adding black formatter to all the code
2022-11-29 11:30:46 -03:00
f9a29f03d9
Address some issues reported by Pylint ( #5677 )
...
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
2022-11-27 11:00:13 +01:00
40d9ca3bdd
use pathlib
2022-10-17 17:40:10 -04:00
8284de5e76
Fix/silence the Pylint messages added in 2.10 ( #5235 )
2021-08-24 12:15:29 +02:00
c5b1ee8101
Make Twisted[http2] installation optional ( #5113 )
...
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
2021-05-11 09:04:53 +02:00
6326178bc5
http2: acceptable protocol update, tests ( #4994 )
2021-02-22 16:50:51 +01:00
6e8d20a07a
HTTP/2: add some type hints ( #4785 )
2020-09-16 09:57:07 +02:00
92bec38591
feat: MethodNotAllowed405, Content-Length header
...
- add tests to check for Content-Length header
- raise MethodNotAllowed405 when remote send 'HTTP/2.0 405 Method Not
Allowed'
2020-07-29 13:49:14 +05:30
3685e99cca
test: http2 connection timeout
2020-07-22 14:47:20 +05:30
316620b517
chore: pass spider as argument for request method
...
- download_maxsize and download_warnsize can now be extracted from the
spider directly and passed to the stream
- remove `partial` flag from the response as per RFC 7540 - Section
8.1.2.6
2020-07-22 13:57:49 +05:30
1dd27a92fa
feat: Idle Timeout for H2Connection (240s)
2020-07-14 22:46:32 +05:30
aeaeb7385b
feat: assert negotiated protocol as h2
...
- implement IHandshakeListener in H2ClientProtocol to know when
handshake is completed
- implement IProtocolNegotiationFactory in H2ClientFactory to provide
information about the acceptableProtols (h2) during NPN or ALPN protocol
2020-07-14 03:55:14 +05:30
64c6af10e1
refactor: use str instead of to_unicode
2020-07-13 00:57:49 +05:30
2ea7d82534
feat: H2ClientFactory
2020-07-08 18:57:13 +05:30
1c40dfa740
fix: handle CONNECTION_LOST & RESET separately
2020-07-07 15:36:41 +05:30
a94b30342a
test: reduce test data size to 1MB
2020-07-06 12:49:12 +05:30
4acdc2e5d6
refactor: use __qualname__, () for large strings
2020-07-01 20:15:33 +05:30
c361fe0d3b
feat: check for invalid hostname
...
- Initiating requests having hostname or (ip_address, port) different
from the peer to which HTTP/2 connection is made can lead to closing the
whole connection and close out all the pending streams.
- This change aims to fix that problem
- Add required tests
- Save hostname & port in H2ConnectionMetadataDict
2020-07-01 18:14:44 +05:30
7b1ad995a4
test: query params, certificate & ip_address
...
- refactor from str.format() to f-strings
2020-07-01 10:45:36 +05:30
50dd9271b4
fix: disable redundant logs
...
- while testing the job exceeded the maximum log length
and was terminated
- reduce the number of requests from 20 to 10
2020-06-30 07:36:56 +05:30
26ab3e4137
feat: FIFO policy to handle large no. of requests
...
- add required test -- test by sending 1000 requests
- increase test timeout to 180 seconds to account for tests taking long
time
2020-06-30 06:44:20 +05:30
90a7007f88
test: warnsize logs, no content header, dataloss
2020-06-29 18:29:31 +05:30
690dd7f38b
test: GET & POST request test for h2 client
...
- Remove repeated dependency Twisted from setup.py
- Test for both GET & POST when
- Only 1 request
- Large number (=20) of requests
and
- Small Data (10 KB) per request
- Large Data (10 MB) per request
- Test when request is cancelled by the client'
BREAKING CHANGES
Tests raises OpenSSL.SSL.Error when run using tox. However, all tests
passes when ran using `python -m unittest`.
2020-06-28 16:38:49 +05:30
Andrey Rakhmatullin
Vsevolod Breus
Jon
Jalil SA
Emmanuel Rondan
Mark Mayo
Matthew Donoughe
Adrián Chaves
Eugenio Lacuesta
Aditya