RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,256 Commits 49 Branches 132 Tags 149 MiB
Commit Graph

11256 Commits

Author SHA1 Message Date
Adrian Chaves bce275c1dc Human review 2026-07-03 12:31:54 +02:00
Adrian Chaves 1fb8b09334 Remove rampup 2026-07-03 12:07:34 +02:00
Adrian Chaves f933d73323 WIP 2026-07-03 11:15:26 +02:00
Adrian Chaves 9fe2a229b6 Human review 2026-07-03 10:24:00 +02:00
Adrian Chaves 9da2b1cbc4 Undo addon switch to a package 2026-07-03 07:54:54 +02:00
Adrian Chaves b26befe4c6 Add missing files 2026-07-03 07:49:27 +02:00
Adrian Chaves 555e8736f2 KISS 2026-07-03 07:39:00 +02:00
Adrian Chaves c8e9fbd7a1 KISS 2026-07-03 07:13:46 +02:00
Adrian Chaves 1f9fbee5a4 KISS 2026-07-03 06:44:28 +02:00
Adrian Chaves 6dfbe1e30b KISS 2026-07-03 06:16:55 +02:00
Adrian Chaves 5b276a3da1 KISS 2026-07-03 05:32:31 +02:00
Adrian Chaves 584462815d Improve the default warning about concurrency settings 2026-07-02 16:20:51 +02:00
Adrian Chaves 370ff74dc9 WIP 2026-07-02 15:02:13 +02:00
Adrian Chaves 3f8841efa3 WIP 2026-07-02 10:37:22 +02:00
Adrian Chaves 27f239074c WIP 2026-07-02 10:05:47 +02:00
Adrian Chaves 8f1f66e1eb Address backward compatibility issues, remove CONCURRENT_REQUESTS_PER_IP 2026-07-02 09:28:01 +02:00
Adrian Chaves 7784594f7a Rederepcate PER_IP, deprecate PER_DOMAIN 2026-07-02 08:27:20 +02:00
Adrian Chaves 8e488d837c Clarify backoff per-scope overrides 2026-07-02 06:59:52 +02:00
Adrian Chaves e203c326c2 WIP 2026-07-02 06:48:46 +02:00
Adrian Chaves c21f8dcfc5 WIP 2026-07-01 14:51:10 +02:00
Adrian Chaves 61c9cc33ff WIP 2026-07-01 13:23:00 +02:00
Adrian Chaves 7cae0e53fb Merge remote-tracking branch 'origin/master' into throttling 2026-07-01 12:58:53 +02:00
Adrian Chaves fefa6b651e WIP 2026-07-01 12:43:16 +02:00
Adrian Chaves 7ae19cc2a2 WIP 2026-07-01 12:26:53 +02:00
Adrian Chaves 074f9e567d WIP 2026-07-01 10:19:27 +02:00
Fat-Coder-CN 870803b7fb
fix-utf16-response-test-on-big-endian-systems (#7508) 2026-07-01 12:16:01 +05:00
Adrian Chaves 46e4ffc650 Merge remote-tracking branch 'origin/master' into throttling 2026-07-01 09:14:34 +02:00
Adrian Chaves ce35abbce7 WIP 2026-07-01 09:14:10 +02:00
Adrian 361f689df7
Improve test coverage for crawler.py (#7682) 
* Improve test coverage for crawler.py

* Silence mypy warnings

* Improve test coverage for crawler.py
 2026-07-01 11:53:07 +05:00
Andrey Rakhmatullin fc5216f156
Clarify/cleanup Selector.type (#7704) 2026-07-01 08:50:32 +02:00
Adrian Chaves f87717e05a WIP 2026-06-30 16:45:58 +02:00
Adrian Chaves c30fcc2c16 Merge remote-tracking branch 'origin/master' into throttling 2026-06-30 16:30:24 +02:00
Adrian Chaves b3ea60354f WIP 2026-06-30 16:28:21 +02:00
Adrian a6d6a48aa6
Keep Item fields in definition order (#7694) 2026-06-30 16:26:22 +02:00
Adrian Chaves 0e7c89a6a2 WIP 2026-06-30 15:33:10 +02:00
Andrey Rakhmatullin 00098cb596
Assorted docstring fixes (#7698) 
* Smaller fixes.

* Add more code blocks in docstrings.

* Queue stuff.

* Response stuff.

* Round 2.

* Address feedback.
 2026-06-30 18:27:32 +05:00
Gaurav Yadav deb7e2861e
Fix _get_tag_name() crash for non-string elem.tag (#7686) (#7687) 
* Fix _get_tag_name() crash for non-string elem.tag (#7686)

* test: improve non-string tag test accuracy and add direct unit test

* test: use minimal payload for non-string tag test

* test: address Adrian+syncrain PR feedback

- remove first docstring line (Adrian: unnecessary)
- replace weak isinstance assert with no-op call
- keep Cython function mention (Adrian: wording is great)

* test: replace silent call with assert results == [] per Adrian

* chore: drop accidental pyproject.toml and uv.lock changes
 2026-06-30 17:49:47 +05:00
Adrian Chaves 3ae5ab40c1 WIP 2026-06-30 13:32:57 +02:00
Adrian Chaves ba8bb9e4e9 WIP 2026-06-30 12:53:18 +02:00
Adrian 6ad8a043ca
Fix genspider --editor (#7683) 2026-06-30 12:05:42 +02:00
Adrian Chaves e0cc7ed7db WIP 2026-06-30 11:33:37 +02:00
Adrian Chaves 80f161ef3e Merge remote-tracking branch 'origin/master' into throttling 2026-06-30 08:23:04 +02:00
Andrey Rakhmatullin 52147017b4
Cleanup cookie handling in request_to_curl() (#7684) 
* Adjust CookiesT.

* Drop list of plain cookie dicts support from request_to_curl().

* Extract _decode_cookie().

* Unify logging.

* Extract _to_verbose_cookies().

* Sync and type hint _cookie_to_set_cookie_value() in tests.

* Add tests for bytes in Request.cookies.

* Type hint assertCookieValEqual().
 2026-06-29 21:49:09 +05:00
tanishqtayade 6591cb756c
Fix to LocalCache with limit=0 still stores items rather than disabling the cache (#7663) 
* Fix cell-var-from-loop in _send_catch_log_deferred

Replace lambda capturing receiver by reference with a default
argument to capture it by value, fixing a potential bug where
all deferred callbacks could reference the last receiver in the
loop instead of their respective receivers.

Removes the pylint disable comment and TODO that were suppressing
this issue.

* chore: trigger CI rerun for mypy network error

* Fix mypy error: pass receiver via addBoth args instead of lambda default

* style: apply pre-commit ruff formatting

* fix: LocalCache with limit=0 incorrectly stores items

When limit=0 is passed to LocalCache (e.g. when DNSCACHE_ENABLED=False),
the condition 'if self.limit' evaluates to False due to Python's truthiness
rules, causing items to be stored despite the cache being disabled.

This leads to an unbounded memory leak during long crawls when DNS caching
is explicitly disabled.

Fix changes the condition to 'if self.limit is not None' and adds an early
return when limit=0 to correctly handle the disabled cache case.

* test: add resolver-level regression tests for DNSCACHE_ENABLED=False

Add two regression tests that verify DNS results are not stored in
dnscache when DNSCACHE_ENABLED=False (cache_size=0):

- test_caching_hostname_resolver_dnscache_disabled_rejects_storage:
  verifies _CachingResolutionReceiver does not write to dnscache
  when CachingHostnameResolver is initialized with cache_size=0

- test_caching_threaded_resolver_dnscache_disabled_rejects_storage:
  verifies dnscache rejects storage at the LocalCache level
  when limit=0

* test: drop misleading threaded resolver test

it was writing directly to dnscache, not actually going through
CachingThreadedResolver at all. the threaded resolver already has
its own if dnscache.limit: guard so the bug doesnt affect it anyway.

keeping only the hostname resolver test which covers the actual
bug path through _CachingResolutionReceiver

* test: clean up comments in resolver test

* test: remove unnecessary comment
 2026-06-29 21:37:06 +05:00
Adrian Chaves 8dfb548415 WIP 2026-06-29 16:29:31 +02:00
Javier 4b2b56f384
Update on Request objects (#7286) 2026-06-29 15:34:12 +02:00
Adrian 9559cbee1e
Solve timing issues with HTTP cache tests? (#7692) 2026-06-29 17:21:20 +05:00
Adrian Chaves 184166d629 Use get_* for getters 2026-06-29 10:58:18 +02:00
Adrian Chaves 7a211960c9 Merge remote-tracking branch 'origin/master' into throttling 2026-06-29 10:47:28 +02:00
Adrian Chaves 8834250059 WIP 2026-06-26 22:53:11 +02:00