2b2e18199b
Type downloader middleware tests ( #7858 )
2026-08-03 15:49:15 +05:00
8caaac6ecb
fix(shell): Run IPython prompt in a thread under a running loop ( #7816 )
2026-07-31 18:05:22 +02:00
14478e3f24
Support CONCURRENT_REQUESTS = 0 for unlimited concurrency ( #7840 )
2026-07-31 20:19:07 +05:00
1f03fbc17e
Add scrapy.utils.asyncio.sleep() ( #7843 )
2026-07-31 18:59:38 +05:00
6f87d3f863
Rename the current benchmark for the future ( #7839 )
2026-07-31 16:46:33 +05:00
aa5ded2539
Set up CodSpeed ( #7831 )
...
* Set up CodSpeed
* CodSpeed: update permissions
2026-07-31 13:27:14 +05:00
3180116cd0
Improve test coverage for scrapy.commands ( #7829 )
2026-07-31 12:19:49 +05:00
37661508db
Add RobotParser.crawl_delay() and a robots_parsed signal ( #7830 )
...
* Add RobotParser.crawl_delay() and a robots_parsed signal
* Improve test coverage
2026-07-31 11:59:52 +05:00
259be5d2dd
CI: Run coverage in as few jobs as necessary ( #7834 )
...
* CI: Run coverage in as few jobs as necessary
* Complete test coverage
2026-07-31 11:20:21 +05:00
7436afc95f
Improve test coverage for scrapy.extensions ( #7809 )
2026-07-30 20:27:27 +05:00
3fc7148c5e
Don't evaluate annotations when inspecting signatures ( #7818 )
...
Since Python 3.14 (PEP 649) annotations are evaluated lazily, so
inspect.signature() raises NameError for callables whose annotations
reference names imported only under TYPE_CHECKING. This broke
middleware registration (via argument_is_required()) and custom stats
collectors (via _warn_spider_arg) for user code with such annotations.
Use annotation_format=Format.FORWARDREF on 3.14+: parameter names,
kinds and defaults are unchanged, and unresolvable annotations become
ForwardRef proxies instead of raising.
Resolves #7796 .
2026-07-30 20:18:45 +05:00
433603e6ca
Add AWS_MAX_POOL_CONNECTIONS ( #7794 )
2026-07-30 15:51:45 +02:00
98696efa80
Export item fields in declaration order ( #7824 )
2026-07-30 15:46:45 +05:00
b2d4eedea8
Fix immediate delivery of full feed export batches ( #7730 ) ( #7733 )
...
* Store full feed batches before spider closes (#7730 )
Start closing and storing each batch as soon as it reaches the configured item count. Track unfinished close tasks so spider shutdown still waits for all deliveries before emitting the exporter-closed signal.
Add an end-to-end regression test that verifies the first batch is stored while the crawl is still running.
* Remove the issue reference
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-07-29 19:30:03 +05:00
01447f9965
fix(commands/parse): Restore request callback before invoking spider ( #7803 )
...
* fix(commands/parse): restore request callback before invoking spider
* Add other test
2026-07-29 16:00:24 +05:00
0cbb20e8e8
Treat broken cache records as cache misses ( #7805 )
2026-07-29 14:39:27 +05:00
8b5147ae2e
Improve test coverage for scrapy.pipelines ( #7798 )
...
* Improve test coverage for scrapy.pipelines
* Restore old Pillow support
2026-07-29 14:32:18 +05:00
bc5b5fb1f6
Add Request.to_curl() ( #7743 ) ( #7802 )
2026-07-29 13:47:15 +05:00
e7d8b34e73
Next refactoring pass of test_utils_*. ( #7797 )
2026-07-28 14:24:34 +02:00
ad816d2b3a
Improve test coverage for scrapy.cmdline ( #7795 )
2026-07-28 17:04:54 +05:00
5a65bdcc18
Improve spiders coverage ( #7768 )
...
* Skip the deprecated scrapy.mail in coverage data
* Improve CrawlSpider coverage
* Improve XMLFeedSpider coverage
* Improve SitemapSpider coverage
* Solve mypy issues
* Align new spider tests with the shared test helper structure
2026-07-28 15:24:04 +05:00
59ebb26e60
Fix CaseInsensitiveDict.copy() sharing state with the original ( #7783 )
...
* Fix CaseInsensitiveDict.copy() sharing state with the original
* Address review: don't re-normalise in __copy__, keep _keys in sync in __ior__
UserDict.__ior__ writes self.data directly, bypassing __setitem__, so
_keys never learned about the new keys.
2026-07-28 15:21:55 +05:00
7e8b58a2b2
Migrate away from testfixtures. ( #7793 )
2026-07-28 09:52:52 +02:00
f3693aa8ba
Fix port issue with cached DNS ( #7772 )
...
* Fix port issue with cached DNS
* Keep ports off the cache
* Complete coverage
2026-07-27 21:23:56 +05:00
25e6884e2f
Address recursion and media ignore-request reporting issues ( #7673 )
...
* Address recursion and media ignore-request reporting issues
* Improve test coverage
* Address feedback
2026-07-27 20:34:10 +05:00
cec86f216e
Remove remaining cross-imports in test modules. ( #7782 )
2026-07-27 12:19:12 +02:00
13be37e4b1
Add type hints to test_link.py and test_downloadermiddleware_stats.py ( #7785 )
2026-07-27 10:07:32 +02:00
8489b3dad8
Support iterable values in BaseSettings.update() ( #7763 )
2026-07-23 12:37:21 +02:00
628a3afbbd
Remove deprecated xmliter(), deprecate re_rsearch() ( #7765 )
2026-07-23 10:38:56 +02:00
56dee203e9
Use httpx2 as a preferred alternative to httpx. ( #7762 )
2026-07-23 10:29:22 +02:00
1157b3e235
Add a middleware to detect response.meta copies ( #7588 )
2026-07-22 13:27:31 +02:00
a54c438da1
Provide extras for optional dependencies ( #7596 )
2026-07-22 10:44:26 +02:00
64358f547b
Replace import tests with find_spec() or importorskip(). ( #7755 )
2026-07-22 08:58:05 +02:00
ca21306df7
Reactorless import hook improvements. ( #7747 )
...
* Reactorless import hook improvements.
* Replace prints with logging.
* More tests.
* Uninstall the import hook earlier.
* Add pragma: no branch.
2026-07-21 19:58:40 +05:00
1ddf024b89
Log ignored responses exceeding DOWNLOAD_MAXSIZE ( #7742 )
...
* Log ignored responses exceeding DOWNLOAD_MAXSIZE
Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
* Use caplog in compression warning test
Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
* Make compression warning test robust
Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
---------
Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
2026-07-21 18:15:30 +05:00
394c2797f3
Remove non-Twisted uses of zope.interface ( #7731 )
2026-07-21 13:30:21 +02:00
e322905255
Assorted test fixes, part 3. ( #7741 )
...
* Assorted test fixes, part 3.
* Fix asyncio_enabled_reactor_different_loop.py on 3.14.
2026-07-21 15:54:43 +05:00
4ee3676464
Test restructuring ( #7736 )
2026-07-21 12:44:56 +02:00
b1b9efb473
Rewrite GCSFilesStore tests to use mocking. ( #7727 )
2026-07-21 11:43:36 +02:00
4d4a04f318
Improve a flaky test_stacktracedump_dump_stacktrace(). ( #7753 )
2026-07-21 13:17:56 +05:00
e80f94fe8a
Tests for extensions.debug and extensions.memdebug. ( #7749 )
...
* Tests for extensions.debug and extensions.memdebug.
* Use a unique class in test_crawl_sets_stats.
* Don't assume a single log message.
2026-07-20 15:21:28 +02:00
7faf20c6b5
Merge repeated curl -d/--data options in Request.from_curl() ( #7728 )
...
curl merges repeated -d/--data/--data-raw options into a single request
body joined with "&" (e.g. `curl -d a=1 -d b=2` sends `a=1&b=2`). scrapy's
DataAction kept only the last value, silently dropping the earlier ones, so
`Request.from_curl("curl -d a=1 -d b=2 ...")` produced a body of just `b=2`.
Accumulate the values with "&" to match curl. -H/--header and -b/--cookie
already accumulate via action="append"; -d was the only repeatable data flag
that didn't.
2026-07-20 15:18:55 +02:00
a591d15c04
Deprecate Spider.log(). ( #7739 )
2026-07-20 14:59:55 +02:00
11d7a05a6f
Deprecate is_secure=False for s3://. ( #7738 )
2026-07-20 14:59:11 +02:00
b3670369b8
Solve the feed Path issue ( #7674 )
...
* Solve the feed Path issue
* Address additional, related issues
2026-07-13 22:46:41 +05:00
c9446931a8
Use external mitmproxy. ( #7720 )
2026-07-09 17:02:05 +02:00
9d9950df69
Fix silent data loss in CsvItemExporter ( closes #4002 ) ( #7651 )
...
Adds state tracking to CsvItemExporter to detect when auto-detected fields drop data on items. Injects a warning instructing users to configure FEED_EXPORT_FIELDS. Includes unit tests. Corrects the false positive logic in stalled PR #7613 .
2026-07-08 17:49:29 +05:00
61f99f2df1
More granular untyped defs checking in tests. ( #7712 )
2026-07-08 10:05:45 +02:00
9523e1ec8c
Merge commit from fork
2026-07-07 12:25:36 +02:00
dd10cb8e9a
LxmlLinkExtractor: add deny_attrs and deny_tags ( #7679 )
2026-07-05 16:46:22 +05:00
Adrian
Laerte Pereira
Janit Rajkarnikar
Fandu
Youssef Mohamed
Andrey Rakhmatullin
Shadow_Lu
SpiliosDmk
Mridankan Mandal
Madan kumar
Shaquon Kelley