RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,238 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

11238 Commits

Author SHA1 Message Date
Shadow_Lu 59ebb26e60
Fix CaseInsensitiveDict.copy() sharing state with the original (#7783) 
* Fix CaseInsensitiveDict.copy() sharing state with the original

* Address review: don't re-normalise in __copy__, keep _keys in sync in __ior__

UserDict.__ior__ writes self.data directly, bypassing __setitem__, so
_keys never learned about the new keys.
 2026-07-28 15:21:55 +05:00
Andrey Rakhmatullin 7e8b58a2b2
Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
Andrey Rakhmatullin a5bc43e34c
Don't generate test keys concurrently with xdist. (#7792) 2026-07-28 09:51:52 +02:00
Adrian f3693aa8ba
Fix port issue with cached DNS (#7772) 
* Fix port issue with cached DNS

* Keep ports off the cache

* Complete coverage
 2026-07-27 21:23:56 +05:00
Adrian 25e6884e2f
Address recursion and media ignore-request reporting issues (#7673) 
* Address recursion and media ignore-request reporting issues

* Improve test coverage

* Address feedback
 2026-07-27 20:34:10 +05:00
Andrey Rakhmatullin cec86f216e
Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
SpiliosDmk 13be37e4b1
Add type hints to test_link.py and test_downloadermiddleware_stats.py (#7785) 2026-07-27 10:07:32 +02:00
SpiliosDmk e710b9c18e
core: fix verb tense in http2 stream comment (#7778) 
'needs to be send' -> 'needs to be sent'. Comment-only, no functional change.
 2026-07-25 10:20:30 +05:00
SpiliosDmk 96195e4a61
docs: fix a/an article before XML in faq.rst (#7779) 
'a XML document' -> 'an XML document' (XML is pronounced with a vowel sound).
 2026-07-25 10:20:18 +05:00
SpiliosDmk e4ae4aad52
docs: fix missing 'of' in faq.rst (#7780) 
'instead joining the strings' -> 'instead of joining the strings'.
 2026-07-25 10:19:53 +05:00
Adrian 58ed9fdccc
Document urlparse_cached (#7777) 2026-07-24 20:08:58 +05:00
Adrian 41bb09741a
Use autodoc for contracts (#7775) 2026-07-24 15:35:54 +02:00
Adrian 0b578c1cbf
trackref: use autodoc (#7771) 2026-07-24 16:24:54 +05:00
Adrian abbc024bbe
Use autodoc for stats.rst (#7769) 2026-07-23 19:48:59 +05:00
Adrian 67e5282684
Use autodoc for exceptions and improve their docs (#7767) 2026-07-23 16:02:41 +05:00
Mridankan Mandal 8489b3dad8
Support iterable values in BaseSettings.update() (#7763) 2026-07-23 12:37:21 +02:00
Andrey Rakhmatullin 628a3afbbd
Remove deprecated xmliter(), deprecate re_rsearch() (#7765) 2026-07-23 10:38:56 +02:00
Andrey Rakhmatullin 56dee203e9
Use httpx2 as a preferred alternative to httpx. (#7762) 2026-07-23 10:29:22 +02:00
Adrian 1157b3e235
Add a middleware to detect response.meta copies (#7588) 2026-07-22 13:27:31 +02:00
Adrian a54c438da1
Provide extras for optional dependencies (#7596) 2026-07-22 10:44:26 +02:00
Andrey Rakhmatullin 64358f547b
Replace import tests with find_spec() or importorskip(). (#7755) 2026-07-22 08:58:05 +02:00
Andrey Rakhmatullin ca21306df7
Reactorless import hook improvements. (#7747) 
* Reactorless import hook improvements.

* Replace prints with logging.

* More tests.

* Uninstall the import hook earlier.

* Add pragma: no branch.
 2026-07-21 19:58:40 +05:00
Mridankan Mandal 1ddf024b89
Log ignored responses exceeding DOWNLOAD_MAXSIZE (#7742) 
* Log ignored responses exceeding DOWNLOAD_MAXSIZE

Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>

* Use caplog in compression warning test

Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>

* Make compression warning test robust

Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>

---------

Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
 2026-07-21 18:15:30 +05:00
Andrey Rakhmatullin 394c2797f3
Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
Andrey Rakhmatullin e322905255
Assorted test fixes, part 3. (#7741) 
* Assorted test fixes, part 3.

* Fix asyncio_enabled_reactor_different_loop.py on 3.14.
 2026-07-21 15:54:43 +05:00
Andrey Rakhmatullin 4ee3676464
Test restructuring (#7736) 2026-07-21 12:44:56 +02:00
Andrey Rakhmatullin 36bf1185e5
Modernize code examples, drop docs for nonexistent `MEMDEBUG_NOTIFY` (#7737) 2026-07-21 12:22:04 +02:00
Andrey Rakhmatullin b1b9efb473
Rewrite GCSFilesStore tests to use mocking. (#7727) 2026-07-21 11:43:36 +02:00
Andrey Rakhmatullin 1c5404dce5
Add docs about running Scrapy from apps and notebooks. (#7751) 2026-07-21 11:23:52 +02:00
Andrey Rakhmatullin 4d4a04f318
Improve a flaky test_stacktracedump_dump_stacktrace(). (#7753) 2026-07-21 13:17:56 +05:00
Andrey Rakhmatullin e80f94fe8a
Tests for extensions.debug and extensions.memdebug. (#7749) 
* Tests for extensions.debug and extensions.memdebug.

* Use a unique class in test_crawl_sets_stats.

* Don't assume a single log message.
 2026-07-20 15:21:28 +02:00
Madan kumar 7faf20c6b5
Merge repeated curl -d/--data options in Request.from_curl() (#7728) 
curl merges repeated -d/--data/--data-raw options into a single request
body joined with "&" (e.g. `curl -d a=1 -d b=2` sends `a=1&b=2`). scrapy's
DataAction kept only the last value, silently dropping the earlier ones, so
`Request.from_curl("curl -d a=1 -d b=2 ...")` produced a body of just `b=2`.
Accumulate the values with "&" to match curl. -H/--header and -b/--cookie
already accumulate via action="append"; -d was the only repeatable data flag
that didn't.
 2026-07-20 15:18:55 +02:00
Andrey Rakhmatullin a591d15c04
Deprecate Spider.log(). (#7739) 2026-07-20 14:59:55 +02:00
Andrey Rakhmatullin 11d7a05a6f
Deprecate is_secure=False for s3://. (#7738) 2026-07-20 14:59:11 +02:00
Andrey Rakhmatullin d8ba1571e7
Assorted docs fixes, part 2. (#7725) 
* Assorted docs fixes, part 2.

* Second pass.

* Address feedback.
 2026-07-14 00:49:44 +05:00
Adrian b3670369b8
Solve the feed Path issue (#7674) 
* Solve the feed Path issue

* Address additional, related issues
 2026-07-13 22:46:41 +05:00
Andrey Rakhmatullin c9446931a8
Use external mitmproxy. (#7720) 2026-07-09 17:02:05 +02:00
Shaquon Kelley 9d9950df69
Fix silent data loss in CsvItemExporter (closes #4002) (#7651) 
Adds state tracking to CsvItemExporter to detect when auto-detected fields drop data on items. Injects a warning instructing users to configure FEED_EXPORT_FIELDS. Includes unit tests. Corrects the false positive logic in stalled PR #7613.
 2026-07-08 17:49:29 +05:00
Andrey Rakhmatullin 61f99f2df1
More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
Andrey Rakhmatullin bdf3067935
Assorted docs fixes, part 1. (#7710) 2026-07-07 18:39:34 +05:00
Adrian c5ec881f1d
Update tools (#7724) 2026-07-07 17:27:26 +05:00
Adrian Chaves feb692f552 Bump version: 2.16.0 → 2.17.0 2026-07-07 12:26:52 +02:00
Adrian 9523e1ec8c
Merge commit from fork 2026-07-07 12:25:36 +02:00
Andrey Rakhmatullin 5ccc8dbe8a
Release notes for 2.17.0 (#7723) 
* Cover 2.17.0 in the release notes up to dd10cb8.

* Fix a copypasted issue number.

* Address minor issues

* Minor fixes

* Cover additional deprecations introduced while improving test coverage

* versionadded/versionchanged improvements

* Add a versionadded to the give_up_log_level docstring

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
 2026-07-07 14:57:42 +05:00
Adrian dd10cb8e9a
LxmlLinkExtractor: add deny_attrs and deny_tags (#7679) 2026-07-05 16:46:22 +05:00
Fat-Coder-CN 870803b7fb
fix-utf16-response-test-on-big-endian-systems (#7508) 2026-07-01 12:16:01 +05:00
Adrian 361f689df7
Improve test coverage for crawler.py (#7682) 
* Improve test coverage for crawler.py

* Silence mypy warnings

* Improve test coverage for crawler.py
 2026-07-01 11:53:07 +05:00
Andrey Rakhmatullin fc5216f156
Clarify/cleanup Selector.type (#7704) 2026-07-01 08:50:32 +02:00
Adrian a6d6a48aa6
Keep Item fields in definition order (#7694) 2026-06-30 16:26:22 +02:00
Andrey Rakhmatullin 00098cb596
Assorted docstring fixes (#7698) 
* Smaller fixes.

* Add more code blocks in docstrings.

* Queue stuff.

* Response stuff.

* Round 2.

* Address feedback.
 2026-06-30 18:27:32 +05:00