RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,296 Commits 49 Branches 132 Tags 148 MiB
Commit Graph

11296 Commits

Author SHA1 Message Date
Adrian 0b2d220197
Improve docs for multi-spider runs (#7907) 2026-08-09 14:19:50 +05:00
Adrian 0e324f3d4a
Let spiders change allowed_domains at run time (#7912) 2026-08-09 14:17:44 +05:00
Adrian 94dff69468
Add VCS CI job, fix support for upcoming parsel version (#7924) 2026-08-09 14:15:58 +05:00
Adrian b4279e243b
Document that process_value runs before allow and deny (#7940) 2026-08-09 14:13:29 +05:00
Adrian fc9c505e79
Add URL benchmarks (#7914) 2026-08-09 14:12:44 +05:00
Adrian 4a69e48f0f
Log the first depth-limited link only (#7916) 2026-08-09 14:08:49 +05:00
Adrian 81d12c6eb8
Document the Referer caveat of DEFAULT_REQUEST_HEADERS (#7917) 2026-08-09 14:08:06 +05:00
Adrian 5b4888a0b1
Document that signal handler order is undefined (#7941) 2026-08-09 14:07:23 +05:00
Adrian 38f6e3cfd1
Remove the leftover KEEP_ALIVE setting from scrapy shell (#7928) 2026-08-09 14:06:39 +05:00
Adrian 56f4afd84e
Fix the telnet console shutdown error after a failed start (#7910) 2026-08-09 14:03:57 +05:00
Adrian 1e92635a18
Benchmark item processing and item concurrency (#7954) 2026-08-09 14:01:06 +05:00
Adrian 63485522b6
Remove the job directory of a download slot once it drains (#7955) 2026-08-09 13:58:59 +05:00
Adrian 9d2dea7a8d
Stop skipping the IPv6 resolver tests (#7935) 2026-08-09 13:57:44 +05:00
Adrian 1bd839b57d
llm-check: allow-list authors by org or track record (#7906) 2026-08-06 08:39:46 +02:00
Adrian 0c89e87b18
Switch tactics (#7895) 2026-08-05 20:17:25 +02:00
Adrian e0128c20c3
Extend benchmarks (#7887) 2026-08-05 11:51:22 +05:00
Adrian 91b70e4db4
Improve the docs about scrapy parse --pipelines (#7876) 2026-08-04 17:30:51 +02:00
Adrian 639fac78b3
Cover the signature change of scrape_func (#7875) 2026-08-04 20:02:52 +05:00
Adrian 0c542afa65
Upgrade the minimum queuelib to 1.6.1 (#7874) 2026-08-04 18:47:00 +05:00
Adrian 6e2081ca41
Ask custom download handlers not to use engine.download_async() (#7871) 2026-08-04 14:41:25 +05:00
Andrey Rakhmatullin 4cdde1c79c
Bump CodSpeed to 5. (#7870) 2026-08-04 09:25:52 +02:00
Adrian 388d4fcf04
Add a _load_objects() helper for object-or-path setting lists (#7857) 
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-04 11:56:46 +05:00
Adrian 54da6c88aa
Deprecate the download_delay spider attribute, and fix the suggested replacement for max_concurrent_requests (#7833) 
* Deprecate the download_delay and max_concurrent_requests spider attributes

* Fix the deprecation entry of max_concurrent_requests
 2026-08-03 23:46:41 +05:00
Adrian a7385d6e51
Type tests.mockserver and other resource-defining files (#7865) 
* Type tests.mockserver and other resource-defining files

* Address pylint issues
 2026-08-03 23:44:28 +05:00
Adrian 298c9e610e
Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
Adrian cde7af87fa
Cover passing spider to a deprecated Downloader.fetch() (#7863) 2026-08-03 13:55:39 +02:00
Adrian a9f1770306
Docs: sort and compact the component-settings list (#7862) 2026-08-03 13:03:28 +02:00
Adrian 2b2e18199b
Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
Adrian e83c709574
Docs: a job directory belongs to one Scrapy version (#7861) 2026-08-03 12:44:48 +02:00
Adrian a499dc9511
CI: Reduce the test job matrix while maintaining coverage (#7844) 2026-08-01 20:30:50 +05:00
Andrey Rakhmatullin 11d1712a2c
Force CI job names to not include "true" for "coverage". (#7848) 2026-08-01 09:51:45 +02:00
Adrian 24de06bcd3
CI: Install dependencies with uv (#7838) 
* CI: Install dependencies with uv

* Setup GitHub Actions hardening

* CI: Use uv for mitmproxy, benchmarks and cache keys

* Fix mitmproxy install on PyPy
 2026-07-31 23:02:52 +05:00
Laerte Pereira 8caaac6ecb
fix(shell): Run IPython prompt in a thread under a running loop (#7816) 2026-07-31 18:05:22 +02:00
Adrian 14478e3f24
Support CONCURRENT_REQUESTS = 0 for unlimited concurrency (#7840) 2026-07-31 20:19:07 +05:00
Adrian 1f03fbc17e
Add scrapy.utils.asyncio.sleep() (#7843) 2026-07-31 18:59:38 +05:00
Adrian 6f87d3f863
Rename the current benchmark for the future (#7839) 2026-07-31 16:46:33 +05:00
Adrian 6cefaa5434
Add a stats reference (#7814) 2026-07-31 15:43:40 +05:00
Adrian aa5ded2539
Set up CodSpeed (#7831) 
* Set up CodSpeed

* CodSpeed: update permissions
 2026-07-31 13:27:14 +05:00
Adrian 3180116cd0
Improve test coverage for scrapy.commands (#7829) 2026-07-31 12:19:49 +05:00
Adrian ea7c0af2f9
Use a single badge for all tests (#7836) 2026-07-31 12:18:06 +05:00
Adrian 746bc7548d
Clarify crawl vs runspider in help and docs (#7832) 2026-07-31 12:17:34 +05:00
Adrian 37661508db
Add RobotParser.crawl_delay() and a robots_parsed signal (#7830) 
* Add RobotParser.crawl_delay() and a robots_parsed signal

* Improve test coverage
 2026-07-31 11:59:52 +05:00
Adrian 434fd1154a
Improve pre-crawler setting docs (#7835) 2026-07-31 11:38:33 +05:00
Adrian 259be5d2dd
CI: Run coverage in as few jobs as necessary (#7834) 
* CI: Run coverage in as few jobs as necessary

* Complete test coverage
 2026-07-31 11:20:21 +05:00
Adrian 7436afc95f
Improve test coverage for scrapy.extensions (#7809) 2026-07-30 20:27:27 +05:00
Janit Rajkarnikar 3fc7148c5e
Don't evaluate annotations when inspecting signatures (#7818) 
Since Python 3.14 (PEP 649) annotations are evaluated lazily, so
inspect.signature() raises NameError for callables whose annotations
reference names imported only under TYPE_CHECKING. This broke
middleware registration (via argument_is_required()) and custom stats
collectors (via _warn_spider_arg) for user code with such annotations.

Use annotation_format=Format.FORWARDREF on 3.14+: parameter names,
kinds and defaults are unchanged, and unresolvable annotations become
ForwardRef proxies instead of raising.

Resolves #7796.
 2026-07-30 20:18:45 +05:00
Adrian f02a99fe71
Add doc sections for callbacks and errbacks (#7821) 2026-07-30 20:15:15 +05:00
Adrian 433603e6ca
Add AWS_MAX_POOL_CONNECTIONS (#7794) 2026-07-30 15:51:45 +02:00
Adrian 98696efa80
Export item fields in declaration order (#7824) 2026-07-30 15:46:45 +05:00
Laerte Pereira 5b4828a012
docs(practices): Remove scrapoxy mention (#7817) 2026-07-30 08:02:54 +02:00