RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,894 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

10894 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 552f2fb91e
Generic improvements related to reactorless tests (#6968) 
* Skip doctests that import the reactor.

* Extract FTPDownloadHandler tests.

* Skip TestHttps2ClientProtocol via pytestmark.

* Replace some explicit reactor.callLater() in tests.

* Simplify TestRequestSendOrder._test_request_order().
 2025-07-25 13:35:00 +05:00
Andrey Rakhmatullin 8c8f4ff033
Extract in-process test HTTP servers into separate processes (#6960) 
* Switch TestEngineBase to mockserver.

* Split mockserver.py.

* Stop using WrappingFactory.

* Remove keyfile/certfile fields from TestHttpBase.

* Move webclient-only resources to test_webclient.py.

* Refactor MockServer.url().

* Migrate TestHttpBase to mockserver.

* Migrate TestSimpleHttpsBase to a separate mockserver.

* Rename TestHttpMockServerBase.

* Migrate TestHttpProxyBase to a separate mockserver.

* Refactor HTTP mockservers.
 2025-07-25 12:42:27 +05:00
M Ikram Ullah Khan 9e83a58643
Add type check in get_media_requests to enforce image / file urls as a list (#6949) 2025-07-19 11:54:35 +02:00
Andrey Rakhmatullin e225d0dea4
Ban more imports that import twisted.internet.reactor. (#6947) 2025-07-10 21:19:50 +05:00
Andrey Rakhmatullin 6b2997af90
Migrate to pytest-twisted (#6938) 
* Migrate to pytest-twisted (WIP)

* Some typing fixes.

* Make --reactor=asyncio the default again.

* Try installing the correct event loop policy in tests on Windows.

* Make reactor_pytest a normal fixture.

* Fix test warnings.

* Fix FTPDownloadHandler teardown.

* Cleanups, typing.

* More cleanup.

* Update only_asyncio/only_not_asyncio mark messages.
 2025-07-06 21:27:17 +05:00
Andrey Rakhmatullin 14eace5d8f
Increase the command timeout in test_short_timeout(). (#6945) 2025-07-06 01:56:54 +05:00
Andrey Rakhmatullin 4279f2837c
Enable the flake8-builtins rules. (#6942) 2025-07-05 21:43:18 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin 16f168b406
Deprecate mustbe_deferred() and defer_*(). (#6937) 2025-07-04 16:41:15 +05:00
Andrey Rakhmatullin d9ef0350d8 Merge remote-tracking branch 'origin/2.13' 2025-07-02 20:46:02 +05:00
Andrey Rakhmatullin 155a504f24 Bump version: 2.13.2 → 2.13.3 2025-07-02 20:39:46 +05:00
Andrey Rakhmatullin cf465bf644
Release notes for 2.13.3. (#6934) 
* Release notes for 2.13.3.

* Address feedback.
 2025-07-02 20:36:01 +05:00
Andrey Rakhmatullin 8d82e7bd27
Fix limiting subprocess run time in tests. (#6933) 
* Fix limiting subprocess run time in tests.

* Make test_runspider_dnscache_disabled() faster.
 2025-07-01 20:23:35 +05:00
Andrey Rakhmatullin 7842180b2f Merge remote-tracking branch 'origin/2.13' 2025-07-01 17:28:12 +05:00
Andrey Rakhmatullin c9cdf0af3c
Narrow down TestEngine::test_short_timeout() expectations. (#6911) 2025-07-01 13:47:43 +02:00
Laerte Pereira dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting (#6921) 
* Deprecate CONCURRENT_REQUESTS_PER_IP setting

* Typing fix

* PR review

* tests

* oops

* Update scrapy/settings/default_settings.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* update logic

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-06-30 16:51:05 +05:00
Andrey Rakhmatullin 3843091c5f
Mark abstract base classes as such. (#6930) 
* Mark abstract base classes as such.

* Remove an unneeded method.

* Fix exporter test coverage.
 2025-06-30 15:14:34 +05:00
Andrey Rakhmatullin 8b3c3ea4ae
Refactor test functions that were returning a Deferred directly. (#6889) 2025-06-30 09:19:23 +02:00
Andrey Rakhmatullin db0be1771c
Test refactoring, part 2. (#6928) 2025-06-29 23:10:38 +02:00
Andrey Rakhmatullin 03fe7a6424 Add a deprecation notice for the offsite spider middleware. 2025-06-29 02:35:48 +05:00
Andrey Rakhmatullin 2c1c10e923
Remove deprecated offsite spider middleware. (#6926) 
* Remove deprecated offsite spider middleware.

* Add a deprecation notice for the offsite spider middleware.
 2025-06-29 02:35:20 +05:00
Adrián Chaves ba1cedee1b Document the new defaults of throttling settings (#6923) 2025-06-28 18:13:54 +05:00
Andrey Rakhmatullin 076c01104a Lower the concurrency settings in the default project template. (#6918) 
* Lower the concurrency settings in the default project template.

* Simplify the concurrency settings template.
 2025-06-28 18:13:44 +05:00
Adrián Chaves 3019393686
Document the new defaults of throttling settings (#6923) 2025-06-28 18:04:40 +05:00
Adrián Chaves 1473f4d347
Remove the deprecated sphinx-hoverxref (#6922) 2025-06-28 18:04:06 +05:00
Andrey Rakhmatullin 95172659af
Add sphinx-lint. (#6920) 2025-06-28 01:37:20 +02:00
Andrey Rakhmatullin f454465b14
Lower the concurrency settings in the default project template. (#6918) 
* Lower the concurrency settings in the default project template.

* Simplify the concurrency settings template.
 2025-06-27 17:40:31 +05:00
Andrey Rakhmatullin 5fbab843bd Fix invalid XML samples in tests that break with lxml 6. (#6919) 2025-06-27 16:24:09 +05:00
Andrey Rakhmatullin eaae59fbef
Fix invalid XML samples in tests that break with lxml 6. (#6919) 2025-06-27 13:05:02 +02:00
Andrey Rakhmatullin ef01a953b1 Fix setting ExecutionEngine._slot to None on close. (#6910) 
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
 2025-06-25 17:30:57 +05:00
Andrey Rakhmatullin ff7795b159
Fix setting ExecutionEngine._slot to None on close. (#6910) 
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
 2025-06-25 17:29:29 +05:00
Andrey Rakhmatullin d70f8a3f14
Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
Andrey Rakhmatullin 7fbd56bc9b Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 22:38:38 +05:00
Andrey Rakhmatullin 0d75355b41
Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 19:36:54 +02:00
Andrey Rakhmatullin b53faacfcd Add .vscode to .gitignore. (#6907) 2025-06-23 22:26:13 +05:00
Thalison Fernandes 5e20b46e35 Fix FileFeedStorage handling of Windows paths without file:// scheme (#6897) 2025-06-23 22:26:04 +05:00
Andrey Rakhmatullin 843ad1afb1 Replace Black with Ruff in contributing.rst. (#6903) 2025-06-23 22:25:56 +05:00
Thalison Fernandes 020bfa7e5f Add .venv folder to .gitignore (#6901) 2025-06-23 22:25:49 +05:00
Andrey Rakhmatullin 9149b6e7fc Use new pytest for new Twisted. (#6893) 2025-06-23 22:25:34 +05:00
Andrey Rakhmatullin 9d324ebd13
Add .vscode to .gitignore. (#6907) 2025-06-23 18:58:54 +02:00
Thalison Fernandes 0d86fb69dc
Fix FileFeedStorage handling of Windows paths without file:// scheme (#6897) 2025-06-23 18:56:29 +02:00
Andrey Rakhmatullin 712e965dbd
Replace Black with Ruff in contributing.rst. (#6903) 2025-06-23 17:56:48 +02:00
Thalison Fernandes d1575220ef
Add .venv folder to .gitignore (#6901) 2025-06-23 17:49:57 +02:00
Andrey Rakhmatullin 91b186cf18
Use new pytest for new Twisted. (#6893) 2025-06-23 17:42:11 +02:00
Adrián Chaves 85aeda365d
Clean up setting getter defaults (#6892) 2025-06-16 09:28:06 +02:00
Andrey Rakhmatullin daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts (#6888) 
* Remove the chdir fixture.

* Re-enable fancy pytest asserts.

* Remove doc files from pytest ignores.

* Restore docs/_ext in test collection ignores.

* Skip a doctest that fails on Windows.

* Fix tests that were writing to the current dir.
 2025-06-14 14:01:20 +05:00
Andrey Rakhmatullin 92c18d15b4
Remove ProcessTest and SiteTest. (#6885) 
* Remove ProcessTest and SiteTest.

* Restore the support for Windows line endings in TestParseCommand.

* Add a test for running a scrapy command in a project subdir.

* Remove pywin32 from test deps.
 2025-06-13 22:18:27 +05:00
Andrey Rakhmatullin b4d11b8b25
Further reduce deps on unittest. (#6884) 2025-06-11 01:28:09 +02:00
Andrey Rakhmatullin ac956f8595
Replace most of the @inlineCallbacks test helpers. (#6883) 2025-06-10 10:02:27 +02:00
Andrey Rakhmatullin 0390176ecd Merge branch '2.13' 2025-06-09 15:54:35 +05:00