RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,294 Commits 49 Branches 132 Tags 149 MiB
Commit Graph

11294 Commits

Author SHA1 Message Date
Adrian 94dff69468
Add VCS CI job, fix support for upcoming parsel version (#7924) 2026-08-09 14:15:58 +05:00
Adrian b4279e243b
Document that process_value runs before allow and deny (#7940) 2026-08-09 14:13:29 +05:00
Adrian fc9c505e79
Add URL benchmarks (#7914) 2026-08-09 14:12:44 +05:00
Adrian 4a69e48f0f
Log the first depth-limited link only (#7916) 2026-08-09 14:08:49 +05:00
Adrian 81d12c6eb8
Document the Referer caveat of DEFAULT_REQUEST_HEADERS (#7917) 2026-08-09 14:08:06 +05:00
Adrian 5b4888a0b1
Document that signal handler order is undefined (#7941) 2026-08-09 14:07:23 +05:00
Adrian 38f6e3cfd1
Remove the leftover KEEP_ALIVE setting from scrapy shell (#7928) 2026-08-09 14:06:39 +05:00
Adrian 56f4afd84e
Fix the telnet console shutdown error after a failed start (#7910) 2026-08-09 14:03:57 +05:00
Adrian 1e92635a18
Benchmark item processing and item concurrency (#7954) 2026-08-09 14:01:06 +05:00
Adrian 63485522b6
Remove the job directory of a download slot once it drains (#7955) 2026-08-09 13:58:59 +05:00
Adrian 9d2dea7a8d
Stop skipping the IPv6 resolver tests (#7935) 2026-08-09 13:57:44 +05:00
Adrian 1bd839b57d
llm-check: allow-list authors by org or track record (#7906) 2026-08-06 08:39:46 +02:00
Adrian 0c89e87b18
Switch tactics (#7895) 2026-08-05 20:17:25 +02:00
Adrian e0128c20c3
Extend benchmarks (#7887) 2026-08-05 11:51:22 +05:00
Adrian 91b70e4db4
Improve the docs about scrapy parse --pipelines (#7876) 2026-08-04 17:30:51 +02:00
Adrian 639fac78b3
Cover the signature change of scrape_func (#7875) 2026-08-04 20:02:52 +05:00
Adrian 0c542afa65
Upgrade the minimum queuelib to 1.6.1 (#7874) 2026-08-04 18:47:00 +05:00
Adrian 6e2081ca41
Ask custom download handlers not to use engine.download_async() (#7871) 2026-08-04 14:41:25 +05:00
Andrey Rakhmatullin 4cdde1c79c
Bump CodSpeed to 5. (#7870) 2026-08-04 09:25:52 +02:00
Adrian 388d4fcf04
Add a _load_objects() helper for object-or-path setting lists (#7857) 
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-04 11:56:46 +05:00
Adrian 54da6c88aa
Deprecate the download_delay spider attribute, and fix the suggested replacement for max_concurrent_requests (#7833) 
* Deprecate the download_delay and max_concurrent_requests spider attributes

* Fix the deprecation entry of max_concurrent_requests
 2026-08-03 23:46:41 +05:00
Adrian a7385d6e51
Type tests.mockserver and other resource-defining files (#7865) 
* Type tests.mockserver and other resource-defining files

* Address pylint issues
 2026-08-03 23:44:28 +05:00
Adrian 298c9e610e
Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
Adrian cde7af87fa
Cover passing spider to a deprecated Downloader.fetch() (#7863) 2026-08-03 13:55:39 +02:00
Adrian a9f1770306
Docs: sort and compact the component-settings list (#7862) 2026-08-03 13:03:28 +02:00
Adrian 2b2e18199b
Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
Adrian e83c709574
Docs: a job directory belongs to one Scrapy version (#7861) 2026-08-03 12:44:48 +02:00
Adrian a499dc9511
CI: Reduce the test job matrix while maintaining coverage (#7844) 2026-08-01 20:30:50 +05:00
Andrey Rakhmatullin 11d1712a2c
Force CI job names to not include "true" for "coverage". (#7848) 2026-08-01 09:51:45 +02:00
Adrian 24de06bcd3
CI: Install dependencies with uv (#7838) 
* CI: Install dependencies with uv

* Setup GitHub Actions hardening

* CI: Use uv for mitmproxy, benchmarks and cache keys

* Fix mitmproxy install on PyPy
 2026-07-31 23:02:52 +05:00
Laerte Pereira 8caaac6ecb
fix(shell): Run IPython prompt in a thread under a running loop (#7816) 2026-07-31 18:05:22 +02:00
Adrian 14478e3f24
Support CONCURRENT_REQUESTS = 0 for unlimited concurrency (#7840) 2026-07-31 20:19:07 +05:00
Adrian 1f03fbc17e
Add scrapy.utils.asyncio.sleep() (#7843) 2026-07-31 18:59:38 +05:00
Adrian 6f87d3f863
Rename the current benchmark for the future (#7839) 2026-07-31 16:46:33 +05:00
Adrian 6cefaa5434
Add a stats reference (#7814) 2026-07-31 15:43:40 +05:00
Adrian aa5ded2539
Set up CodSpeed (#7831) 
* Set up CodSpeed

* CodSpeed: update permissions
 2026-07-31 13:27:14 +05:00
Adrian 3180116cd0
Improve test coverage for scrapy.commands (#7829) 2026-07-31 12:19:49 +05:00
Adrian ea7c0af2f9
Use a single badge for all tests (#7836) 2026-07-31 12:18:06 +05:00
Adrian 746bc7548d
Clarify crawl vs runspider in help and docs (#7832) 2026-07-31 12:17:34 +05:00
Adrian 37661508db
Add RobotParser.crawl_delay() and a robots_parsed signal (#7830) 
* Add RobotParser.crawl_delay() and a robots_parsed signal

* Improve test coverage
 2026-07-31 11:59:52 +05:00
Adrian 434fd1154a
Improve pre-crawler setting docs (#7835) 2026-07-31 11:38:33 +05:00
Adrian 259be5d2dd
CI: Run coverage in as few jobs as necessary (#7834) 
* CI: Run coverage in as few jobs as necessary

* Complete test coverage
 2026-07-31 11:20:21 +05:00
Adrian 7436afc95f
Improve test coverage for scrapy.extensions (#7809) 2026-07-30 20:27:27 +05:00
Janit Rajkarnikar 3fc7148c5e
Don't evaluate annotations when inspecting signatures (#7818) 
Since Python 3.14 (PEP 649) annotations are evaluated lazily, so
inspect.signature() raises NameError for callables whose annotations
reference names imported only under TYPE_CHECKING. This broke
middleware registration (via argument_is_required()) and custom stats
collectors (via _warn_spider_arg) for user code with such annotations.

Use annotation_format=Format.FORWARDREF on 3.14+: parameter names,
kinds and defaults are unchanged, and unresolvable annotations become
ForwardRef proxies instead of raising.

Resolves #7796.
 2026-07-30 20:18:45 +05:00
Adrian f02a99fe71
Add doc sections for callbacks and errbacks (#7821) 2026-07-30 20:15:15 +05:00
Adrian 433603e6ca
Add AWS_MAX_POOL_CONNECTIONS (#7794) 2026-07-30 15:51:45 +02:00
Adrian 98696efa80
Export item fields in declaration order (#7824) 2026-07-30 15:46:45 +05:00
Laerte Pereira 5b4828a012
docs(practices): Remove scrapoxy mention (#7817) 2026-07-30 08:02:54 +02:00
Fandu b2d4eedea8
Fix immediate delivery of full feed export batches (#7730) (#7733) 
* Store full feed batches before spider closes (#7730)

Start closing and storing each batch as soon as it reaches the configured item count. Track unfinished close tasks so spider shutdown still waits for all deliveries before emitting the exporter-closed signal.

Add an end-to-end regression test that verifies the first batch is stored while the crawl is still running.

* Remove the issue reference

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-07-29 19:30:03 +05:00
Laerte Pereira 01447f9965
fix(commands/parse): Restore request callback before invoking spider (#7803) 
* fix(commands/parse): restore request callback before invoking spider

* Add other test
 2026-07-29 16:00:24 +05:00