RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,899 Commits 49 Branches 132 Tags 134 MiB
Commit Graph

78 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 32f2aede82
Deprecate non-async counterparts of new async APIs. (#6979) 
* Deprecate non-async counterparts of new async APIs.

* Deprecate Scraper.call_spider().

* Add _schedule_coro().
 2025-08-01 00:58:48 +05:00
Felipe Benevolo 89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() (#6842) 
* added async_download()

* add case tests for ExecutionEngine download() and async_download()

* change async_download() to download_async() in Engine class and subsequent tests

* fixed reactor import in test_engine.py

* Simplify TestEngineDownloadAsync.

* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks

* Simplify ExecutionEngine.download().

* Remove an unneeded import.

* Refactor ExecutionEngine.download{,_async}() tests.

* Deprecate ExecutionEngine.download().

* Update doc examples to use download_async().

* Replace uses of download() with download_async().

* Fix test_downloadermiddleware_robotstxt.py.

* Remove a stray file.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-07-27 23:17:51 +05:00
Andrey Rakhmatullin 8c8f4ff033
Extract in-process test HTTP servers into separate processes (#6960) 
* Switch TestEngineBase to mockserver.

* Split mockserver.py.

* Stop using WrappingFactory.

* Remove keyfile/certfile fields from TestHttpBase.

* Move webclient-only resources to test_webclient.py.

* Refactor MockServer.url().

* Migrate TestHttpBase to mockserver.

* Migrate TestSimpleHttpsBase to a separate mockserver.

* Rename TestHttpMockServerBase.

* Migrate TestHttpProxyBase to a separate mockserver.

* Refactor HTTP mockservers.
 2025-07-25 12:42:27 +05:00
Andrey Rakhmatullin 6b2997af90
Migrate to pytest-twisted (#6938) 
* Migrate to pytest-twisted (WIP)

* Some typing fixes.

* Make --reactor=asyncio the default again.

* Try installing the correct event loop policy in tests on Windows.

* Make reactor_pytest a normal fixture.

* Fix test warnings.

* Fix FTPDownloadHandler teardown.

* Cleanups, typing.

* More cleanup.

* Update only_asyncio/only_not_asyncio mark messages.
 2025-07-06 21:27:17 +05:00
Andrey Rakhmatullin 14eace5d8f
Increase the command timeout in test_short_timeout(). (#6945) 2025-07-06 01:56:54 +05:00
Andrey Rakhmatullin 8d82e7bd27
Fix limiting subprocess run time in tests. (#6933) 
* Fix limiting subprocess run time in tests.

* Make test_runspider_dnscache_disabled() faster.
 2025-07-01 20:23:35 +05:00
Andrey Rakhmatullin 7842180b2f Merge remote-tracking branch 'origin/2.13' 2025-07-01 17:28:12 +05:00
Andrey Rakhmatullin c9cdf0af3c
Narrow down TestEngine::test_short_timeout() expectations. (#6911) 2025-07-01 13:47:43 +02:00
Andrey Rakhmatullin 7fbd56bc9b Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 22:38:38 +05:00
Andrey Rakhmatullin 0d75355b41
Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 19:36:54 +02:00
Andrey Rakhmatullin d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873) 2025-06-06 22:59:09 +02:00
Andrey Rakhmatullin d329eedfef Don't try to close ExecutionEngine.downloader when it doesn't exist. (#6867) 2025-06-06 16:26:38 +05:00
Andrey Rakhmatullin 657e6cb2b5
Don't try to close ExecutionEngine.downloader when it doesn't exist. (#6867) 2025-06-06 13:02:15 +02:00
Andrey Rakhmatullin 916fe50974
Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin eced5ca2d3 Remove the unnecessary spider argument from Scraper and related code. 2025-04-30 00:57:44 +05:00
Andrey Rakhmatullin eb654aa1a8
Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
Andrey Rakhmatullin 5a605969bd
Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
Andrey Rakhmatullin a898331d14
Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
Andrey Rakhmatullin 200d76afa9 Refactor EngineTest tests. 2025-02-01 16:07:55 +05:00
Andrey Rakhmatullin cec0aeca58
Bump ruff, switch from black to ruff-format (#6631) 2025-01-27 11:07:09 +01:00
Andrey Rakhmatullin cde0845ab2
Ruff: migrate pyupgrade and bandit, enable some other rules (#6577) 2024-12-10 19:53:27 +01:00
LucasSD f260f819e0 Remove debug log message from _schedule_request method 2024-09-09 20:26:02 +01:00
Andrey Rakhmatullin 3d8dbd5648 flake8-bugbear 2024-06-20 00:27:38 +05:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Laerte Pereira 09c63a178b
Remove the deprecated spider parameter from the engine (#5998) 2023-08-04 07:53:04 +02:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Adrián Chaves 00ccb02f1f Merge remote-tracking branch 'upstream/2.6' 2022-07-25 14:13:22 +02:00
Mikhail Korobov b103664bf4
Address 2/3 of warnings from tests (#5561) 2022-07-19 17:39:26 +02:00
Adrián Chaves b59a69be17 Test that a low CLOSEPIDER_TIMEOUT does not raise an exception 2022-03-15 12:10:41 +01:00
Andrey Rahmatullin 8284de5e76
Fix/silence the Pylint messages added in 2.10 (#5235) 2021-08-24 12:15:29 +02:00
Iván de Prado eca641aa3d
Update tests/test_engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:40:20 +01:00
Iván de Prado Alonso f35970778b Test case for raising CloseSpider on spider idle signal handler 2021-06-29 13:21:38 +01:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Eugenio Lacuesta d8d1dc5b50
Ignore typing warning in test 2021-04-12 10:43:02 -03:00
Eugenio Lacuesta b6f77806b0
Engine tests: fix item class spider, add minimal type hints 2021-04-09 13:09:47 -03:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Marc Hernández fffb0a5b6a
replace set constructor with set literals (#4573) 2020-05-15 15:23:09 +02:00
Eugenio Lacuesta c4a5e3f0da
Simplify bytes_received signal 
Remove "source" parameter
 2020-04-06 09:26:13 -03:00
Eugenio Lacuesta e3342669ae
Merge branch 'master' into bytes_received_signal 2020-04-06 09:17:35 -03:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
Eugenio Lacuesta 122ce6d6fb
Check bytes are received in order (bytes_received signal) 2020-02-10 10:20:26 -03:00
Eugenio Lacuesta 6f02a8dccb
Add source parameter to bytes_received signal 2020-01-29 15:51:25 -03:00