RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,068 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

11068 Commits

Author SHA1 Message Date
Andrey Rakhmatullin f454465b14
Lower the concurrency settings in the default project template. (#6918) 
* Lower the concurrency settings in the default project template.

* Simplify the concurrency settings template.
 2025-06-27 17:40:31 +05:00
Andrey Rakhmatullin 5fbab843bd Fix invalid XML samples in tests that break with lxml 6. (#6919) 2025-06-27 16:24:09 +05:00
Andrey Rakhmatullin eaae59fbef
Fix invalid XML samples in tests that break with lxml 6. (#6919) 2025-06-27 13:05:02 +02:00
Andrey Rakhmatullin ef01a953b1 Fix setting ExecutionEngine._slot to None on close. (#6910) 
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
 2025-06-25 17:30:57 +05:00
Andrey Rakhmatullin ff7795b159
Fix setting ExecutionEngine._slot to None on close. (#6910) 
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
 2025-06-25 17:29:29 +05:00
Andrey Rakhmatullin d70f8a3f14
Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
Andrey Rakhmatullin 7fbd56bc9b Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 22:38:38 +05:00
Andrey Rakhmatullin 0d75355b41
Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 19:36:54 +02:00
Andrey Rakhmatullin b53faacfcd Add .vscode to .gitignore. (#6907) 2025-06-23 22:26:13 +05:00
Thalison Fernandes 5e20b46e35 Fix FileFeedStorage handling of Windows paths without file:// scheme (#6897) 2025-06-23 22:26:04 +05:00
Andrey Rakhmatullin 843ad1afb1 Replace Black with Ruff in contributing.rst. (#6903) 2025-06-23 22:25:56 +05:00
Thalison Fernandes 020bfa7e5f Add .venv folder to .gitignore (#6901) 2025-06-23 22:25:49 +05:00
Andrey Rakhmatullin 9149b6e7fc Use new pytest for new Twisted. (#6893) 2025-06-23 22:25:34 +05:00
Andrey Rakhmatullin 9d324ebd13
Add .vscode to .gitignore. (#6907) 2025-06-23 18:58:54 +02:00
Thalison Fernandes 0d86fb69dc
Fix FileFeedStorage handling of Windows paths without file:// scheme (#6897) 2025-06-23 18:56:29 +02:00
Andrey Rakhmatullin 712e965dbd
Replace Black with Ruff in contributing.rst. (#6903) 2025-06-23 17:56:48 +02:00
Thalison Fernandes d1575220ef
Add .venv folder to .gitignore (#6901) 2025-06-23 17:49:57 +02:00
Andrey Rakhmatullin 91b186cf18
Use new pytest for new Twisted. (#6893) 2025-06-23 17:42:11 +02:00
Adrián Chaves 85aeda365d
Clean up setting getter defaults (#6892) 2025-06-16 09:28:06 +02:00
Andrey Rakhmatullin daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts (#6888) 
* Remove the chdir fixture.

* Re-enable fancy pytest asserts.

* Remove doc files from pytest ignores.

* Restore docs/_ext in test collection ignores.

* Skip a doctest that fails on Windows.

* Fix tests that were writing to the current dir.
 2025-06-14 14:01:20 +05:00
Andrey Rakhmatullin 92c18d15b4
Remove ProcessTest and SiteTest. (#6885) 
* Remove ProcessTest and SiteTest.

* Restore the support for Windows line endings in TestParseCommand.

* Add a test for running a scrapy command in a project subdir.

* Remove pywin32 from test deps.
 2025-06-13 22:18:27 +05:00
Andrey Rakhmatullin b4d11b8b25
Further reduce deps on unittest. (#6884) 2025-06-11 01:28:09 +02:00
Andrey Rakhmatullin ac956f8595
Replace most of the @inlineCallbacks test helpers. (#6883) 2025-06-10 10:02:27 +02:00
Andrey Rakhmatullin 0390176ecd Merge branch '2.13' 2025-06-09 15:54:35 +05:00
Andrey Rakhmatullin c6740604a4 Bump version: 2.13.1 → 2.13.2 2025-06-09 15:21:06 +05:00
Andrey Rakhmatullin 7400868ad5
Release notes for 2.13.2. (#6868) 
* Release notes for 2.13.2.

* Update release notes.
 2025-06-09 15:19:02 +05:00
nakanoh 6b5a4a6417
Minor improvement in cmdline.py (#6875) 2025-06-09 12:07:01 +02:00
Andrey Rakhmatullin 24a827c72e Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 13:09:44 +05:00
Andrey Rakhmatullin ba10dcfd1a
Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 09:53:42 +02:00
Adrián Chaves bb1c81ba6a Remove the old artwork (#6874) 2025-06-09 12:43:15 +05:00
Adrián Chaves 0ae27b8fa1
Remove the old artwork (#6874) 2025-06-07 09:17:36 +02:00
Andrey Rakhmatullin d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873) 2025-06-06 22:59:09 +02:00
Adrián Chaves 744edb9ba9
Make scrapy fetch work with scrapy-poet (#6872) 2025-06-06 16:09:51 +02:00
Andrey Rakhmatullin d329eedfef Don't try to close ExecutionEngine.downloader when it doesn't exist. (#6867) 2025-06-06 16:26:38 +05:00
Andrey Rakhmatullin 657e6cb2b5
Don't try to close ExecutionEngine.downloader when it doesn't exist. (#6867) 2025-06-06 13:02:15 +02:00
Andrey Rakhmatullin 405d9bc8a2
More docs for the is_asyncio_reactor_installed() behavior change. (#6866) 2025-06-06 12:59:49 +02:00
Andrey Rakhmatullin d99234a33f
Install the reactor explicitly in CrawlerRunner examples. (#6865) 2025-06-06 11:54:21 +02:00
Andrey Rakhmatullin b20995c9d8 Silence a typing error. 2025-06-06 13:16:48 +05:00
Andrey Rakhmatullin 54474ceb0d Fix a regression in errback result handling. (#6863) 2025-06-06 12:47:38 +05:00
Adrián Chaves 3d382aa650 Avoid raw HTML in the README (#6839) 2025-06-06 12:44:06 +05:00
Adrián Chaves b8cd079014 Shorten the README and remove broken links to scrapy.org (#6833) 2025-06-06 12:44:02 +05:00
Adrián Chaves 105c0afb6e Feature the new logo in the README (#6831) 2025-06-06 12:43:56 +05:00
Andrey Rakhmatullin d602f13e8c
Fix a regression in errback result handling. (#6863) 2025-06-05 18:02:31 +05:00
Andrey Rakhmatullin 5902aab25c
Add the call_later() wrapper. (#6858) 2025-06-04 20:37:36 +05:00
Mehraz Hossain Rumman c6698b9fe8
fixing settings order (#6849) 
* fixing issue #6838

* Reorder some more settings.

* Clarify the header.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-06-04 16:02:29 +05:00
Andrey Rakhmatullin 8fb8d2c6b8
Add AsyncioLoopingCall. (#6855) 2025-06-03 20:21:33 +02:00
Andrey Rakhmatullin 3aa5e75787
Use AsyncCrawlerProcess in commands. (#6845) 
* Use AsyncCrawlerProcess in commands.

* Ignore coverage of abstract methods.

* Address feedback.
 2025-06-03 17:47:52 +05:00
Andrey Rakhmatullin d400aa3e2d
Add _parallel_asyncio(). (#6852) 2025-06-03 11:19:15 +02:00
Rodrigosnrocha 9cc23641cc
Deprecate _parse_response and implement parse_with_rules (#6804) 2025-06-02 17:00:17 +02:00
Andrey Rakhmatullin 8ae418df44
Rewrite download handler tests to coroutines. (#6846) 2025-06-02 16:02:08 +02:00