RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,888 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

10888 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 4279f2837c
Enable the flake8-builtins rules. (#6942) 2025-07-05 21:43:18 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin 16f168b406
Deprecate mustbe_deferred() and defer_*(). (#6937) 2025-07-04 16:41:15 +05:00
Andrey Rakhmatullin d9ef0350d8 Merge remote-tracking branch 'origin/2.13' 2025-07-02 20:46:02 +05:00
Andrey Rakhmatullin 155a504f24 Bump version: 2.13.2 → 2.13.3 2025-07-02 20:39:46 +05:00
Andrey Rakhmatullin cf465bf644
Release notes for 2.13.3. (#6934) 
* Release notes for 2.13.3.

* Address feedback.
 2025-07-02 20:36:01 +05:00
Andrey Rakhmatullin 8d82e7bd27
Fix limiting subprocess run time in tests. (#6933) 
* Fix limiting subprocess run time in tests.

* Make test_runspider_dnscache_disabled() faster.
 2025-07-01 20:23:35 +05:00
Andrey Rakhmatullin 7842180b2f Merge remote-tracking branch 'origin/2.13' 2025-07-01 17:28:12 +05:00
Andrey Rakhmatullin c9cdf0af3c
Narrow down TestEngine::test_short_timeout() expectations. (#6911) 2025-07-01 13:47:43 +02:00
Laerte Pereira dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting (#6921) 
* Deprecate CONCURRENT_REQUESTS_PER_IP setting

* Typing fix

* PR review

* tests

* oops

* Update scrapy/settings/default_settings.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* update logic

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-06-30 16:51:05 +05:00
Andrey Rakhmatullin 3843091c5f
Mark abstract base classes as such. (#6930) 
* Mark abstract base classes as such.

* Remove an unneeded method.

* Fix exporter test coverage.
 2025-06-30 15:14:34 +05:00
Andrey Rakhmatullin 8b3c3ea4ae
Refactor test functions that were returning a Deferred directly. (#6889) 2025-06-30 09:19:23 +02:00
Andrey Rakhmatullin db0be1771c
Test refactoring, part 2. (#6928) 2025-06-29 23:10:38 +02:00
Andrey Rakhmatullin 03fe7a6424 Add a deprecation notice for the offsite spider middleware. 2025-06-29 02:35:48 +05:00
Andrey Rakhmatullin 2c1c10e923
Remove deprecated offsite spider middleware. (#6926) 
* Remove deprecated offsite spider middleware.

* Add a deprecation notice for the offsite spider middleware.
 2025-06-29 02:35:20 +05:00
Adrián Chaves ba1cedee1b Document the new defaults of throttling settings (#6923) 2025-06-28 18:13:54 +05:00
Andrey Rakhmatullin 076c01104a Lower the concurrency settings in the default project template. (#6918) 
* Lower the concurrency settings in the default project template.

* Simplify the concurrency settings template.
 2025-06-28 18:13:44 +05:00
Adrián Chaves 3019393686
Document the new defaults of throttling settings (#6923) 2025-06-28 18:04:40 +05:00
Adrián Chaves 1473f4d347
Remove the deprecated sphinx-hoverxref (#6922) 2025-06-28 18:04:06 +05:00
Andrey Rakhmatullin 95172659af
Add sphinx-lint. (#6920) 2025-06-28 01:37:20 +02:00
Andrey Rakhmatullin f454465b14
Lower the concurrency settings in the default project template. (#6918) 
* Lower the concurrency settings in the default project template.

* Simplify the concurrency settings template.
 2025-06-27 17:40:31 +05:00
Andrey Rakhmatullin 5fbab843bd Fix invalid XML samples in tests that break with lxml 6. (#6919) 2025-06-27 16:24:09 +05:00
Andrey Rakhmatullin eaae59fbef
Fix invalid XML samples in tests that break with lxml 6. (#6919) 2025-06-27 13:05:02 +02:00
Andrey Rakhmatullin ef01a953b1 Fix setting ExecutionEngine._slot to None on close. (#6910) 
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
 2025-06-25 17:30:57 +05:00
Andrey Rakhmatullin ff7795b159
Fix setting ExecutionEngine._slot to None on close. (#6910) 
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
 2025-06-25 17:29:29 +05:00
Andrey Rakhmatullin d70f8a3f14
Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
Andrey Rakhmatullin 7fbd56bc9b Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 22:38:38 +05:00
Andrey Rakhmatullin 0d75355b41
Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 19:36:54 +02:00
Andrey Rakhmatullin b53faacfcd Add .vscode to .gitignore. (#6907) 2025-06-23 22:26:13 +05:00
Thalison Fernandes 5e20b46e35 Fix FileFeedStorage handling of Windows paths without file:// scheme (#6897) 2025-06-23 22:26:04 +05:00
Andrey Rakhmatullin 843ad1afb1 Replace Black with Ruff in contributing.rst. (#6903) 2025-06-23 22:25:56 +05:00
Thalison Fernandes 020bfa7e5f Add .venv folder to .gitignore (#6901) 2025-06-23 22:25:49 +05:00
Andrey Rakhmatullin 9149b6e7fc Use new pytest for new Twisted. (#6893) 2025-06-23 22:25:34 +05:00
Andrey Rakhmatullin 9d324ebd13
Add .vscode to .gitignore. (#6907) 2025-06-23 18:58:54 +02:00
Thalison Fernandes 0d86fb69dc
Fix FileFeedStorage handling of Windows paths without file:// scheme (#6897) 2025-06-23 18:56:29 +02:00
Andrey Rakhmatullin 712e965dbd
Replace Black with Ruff in contributing.rst. (#6903) 2025-06-23 17:56:48 +02:00
Thalison Fernandes d1575220ef
Add .venv folder to .gitignore (#6901) 2025-06-23 17:49:57 +02:00
Andrey Rakhmatullin 91b186cf18
Use new pytest for new Twisted. (#6893) 2025-06-23 17:42:11 +02:00
Adrián Chaves 85aeda365d
Clean up setting getter defaults (#6892) 2025-06-16 09:28:06 +02:00
Andrey Rakhmatullin daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts (#6888) 
* Remove the chdir fixture.

* Re-enable fancy pytest asserts.

* Remove doc files from pytest ignores.

* Restore docs/_ext in test collection ignores.

* Skip a doctest that fails on Windows.

* Fix tests that were writing to the current dir.
 2025-06-14 14:01:20 +05:00
Andrey Rakhmatullin 92c18d15b4
Remove ProcessTest and SiteTest. (#6885) 
* Remove ProcessTest and SiteTest.

* Restore the support for Windows line endings in TestParseCommand.

* Add a test for running a scrapy command in a project subdir.

* Remove pywin32 from test deps.
 2025-06-13 22:18:27 +05:00
Andrey Rakhmatullin b4d11b8b25
Further reduce deps on unittest. (#6884) 2025-06-11 01:28:09 +02:00
Andrey Rakhmatullin ac956f8595
Replace most of the @inlineCallbacks test helpers. (#6883) 2025-06-10 10:02:27 +02:00
Andrey Rakhmatullin 0390176ecd Merge branch '2.13' 2025-06-09 15:54:35 +05:00
Andrey Rakhmatullin c6740604a4 Bump version: 2.13.1 → 2.13.2 2025-06-09 15:21:06 +05:00
Andrey Rakhmatullin 7400868ad5
Release notes for 2.13.2. (#6868) 
* Release notes for 2.13.2.

* Update release notes.
 2025-06-09 15:19:02 +05:00
nakanoh 6b5a4a6417
Minor improvement in cmdline.py (#6875) 2025-06-09 12:07:01 +02:00
Andrey Rakhmatullin 24a827c72e Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 13:09:44 +05:00
Andrey Rakhmatullin ba10dcfd1a
Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 09:53:42 +02:00
Adrián Chaves bb1c81ba6a Remove the old artwork (#6874) 2025-06-09 12:43:15 +05:00