RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,322 Commits 49 Branches 132 Tags 148 MiB
Commit Graph

2704 Commits

Author SHA1 Message Date
Adrian 5270f3cf99
Add a depth_reset request metadata key (#7913) 2026-08-09 23:07:10 +05:00
Adrian d41bfaec05
Document how to disallow subdomains of allowed domains (#7903) 2026-08-09 23:05:43 +05:00
Adrian aadfd15327
Do not mangle log messages when a log formatter returns empty or tuple args (#7936) 2026-08-09 22:58:35 +05:00
Adrian bee31890a2
Report exceptions from Spider.start() (#7884) 2026-08-09 22:56:03 +05:00
Adrian a5614544a2
Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
Adrian 786ab10494
Document how to write a custom item exporter (#7931) 2026-08-09 22:31:53 +05:00
Adrian 7d9516f332
Canonicalize each extracted link once (#7961) 2026-08-09 22:30:30 +05:00
Adrian 609f64c55d
Make brotli a hard dependency (#7929) 2026-08-09 19:24:14 +02:00
Adrian 95fe7acc86
Warn only once per request about the download warn size (#7963) 2026-08-09 21:14:20 +05:00
Adrian 18ed0c0f7c
Fall back to the response encoding in TextResponse.json() (#7897) 2026-08-09 16:58:09 +05:00
Adrian 59ce27afdd
Send the bytes_received and headers_received signals over HTTP/2 (#7896) 2026-08-09 16:55:02 +05:00
Adrian 63d5ce6272
Fix cookie lookup for dotless hosts and IP addresses (#7900) 2026-08-09 16:52:32 +05:00
Adrian cd7f422a1e
Use client.bucket() in GCSFeedStorage so object-level GCS permissions suffice (#7945) 2026-08-09 14:57:11 +05:00
Mridankan Mandal f123c7a1cc
Simplify cmdline settings test (#7853) 
Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
 2026-08-09 14:55:58 +05:00
Adrian 0e324f3d4a
Let spiders change allowed_domains at run time (#7912) 2026-08-09 14:17:44 +05:00
Adrian fc9c505e79
Add URL benchmarks (#7914) 2026-08-09 14:12:44 +05:00
Adrian 4a69e48f0f
Log the first depth-limited link only (#7916) 2026-08-09 14:08:49 +05:00
Adrian 56f4afd84e
Fix the telnet console shutdown error after a failed start (#7910) 2026-08-09 14:03:57 +05:00
Adrian 1e92635a18
Benchmark item processing and item concurrency (#7954) 2026-08-09 14:01:06 +05:00
Adrian 63485522b6
Remove the job directory of a download slot once it drains (#7955) 2026-08-09 13:58:59 +05:00
Adrian 9d2dea7a8d
Stop skipping the IPv6 resolver tests (#7935) 2026-08-09 13:57:44 +05:00
Adrian e0128c20c3
Extend benchmarks (#7887) 2026-08-05 11:51:22 +05:00
Adrian 54da6c88aa
Deprecate the download_delay spider attribute, and fix the suggested replacement for max_concurrent_requests (#7833) 
* Deprecate the download_delay and max_concurrent_requests spider attributes

* Fix the deprecation entry of max_concurrent_requests
 2026-08-03 23:46:41 +05:00
Adrian a7385d6e51
Type tests.mockserver and other resource-defining files (#7865) 
* Type tests.mockserver and other resource-defining files

* Address pylint issues
 2026-08-03 23:44:28 +05:00
Adrian 298c9e610e
Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
Adrian cde7af87fa
Cover passing spider to a deprecated Downloader.fetch() (#7863) 2026-08-03 13:55:39 +02:00
Adrian 2b2e18199b
Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
Laerte Pereira 8caaac6ecb
fix(shell): Run IPython prompt in a thread under a running loop (#7816) 2026-07-31 18:05:22 +02:00
Adrian 14478e3f24
Support CONCURRENT_REQUESTS = 0 for unlimited concurrency (#7840) 2026-07-31 20:19:07 +05:00
Adrian 1f03fbc17e
Add scrapy.utils.asyncio.sleep() (#7843) 2026-07-31 18:59:38 +05:00
Adrian 6f87d3f863
Rename the current benchmark for the future (#7839) 2026-07-31 16:46:33 +05:00
Adrian aa5ded2539
Set up CodSpeed (#7831) 
* Set up CodSpeed

* CodSpeed: update permissions
 2026-07-31 13:27:14 +05:00
Adrian 3180116cd0
Improve test coverage for scrapy.commands (#7829) 2026-07-31 12:19:49 +05:00
Adrian 37661508db
Add RobotParser.crawl_delay() and a robots_parsed signal (#7830) 
* Add RobotParser.crawl_delay() and a robots_parsed signal

* Improve test coverage
 2026-07-31 11:59:52 +05:00
Adrian 259be5d2dd
CI: Run coverage in as few jobs as necessary (#7834) 
* CI: Run coverage in as few jobs as necessary

* Complete test coverage
 2026-07-31 11:20:21 +05:00
Adrian 7436afc95f
Improve test coverage for scrapy.extensions (#7809) 2026-07-30 20:27:27 +05:00
Janit Rajkarnikar 3fc7148c5e
Don't evaluate annotations when inspecting signatures (#7818) 
Since Python 3.14 (PEP 649) annotations are evaluated lazily, so
inspect.signature() raises NameError for callables whose annotations
reference names imported only under TYPE_CHECKING. This broke
middleware registration (via argument_is_required()) and custom stats
collectors (via _warn_spider_arg) for user code with such annotations.

Use annotation_format=Format.FORWARDREF on 3.14+: parameter names,
kinds and defaults are unchanged, and unresolvable annotations become
ForwardRef proxies instead of raising.

Resolves #7796.
 2026-07-30 20:18:45 +05:00
Adrian 433603e6ca
Add AWS_MAX_POOL_CONNECTIONS (#7794) 2026-07-30 15:51:45 +02:00
Adrian 98696efa80
Export item fields in declaration order (#7824) 2026-07-30 15:46:45 +05:00
Fandu b2d4eedea8
Fix immediate delivery of full feed export batches (#7730) (#7733) 
* Store full feed batches before spider closes (#7730)

Start closing and storing each batch as soon as it reaches the configured item count. Track unfinished close tasks so spider shutdown still waits for all deliveries before emitting the exporter-closed signal.

Add an end-to-end regression test that verifies the first batch is stored while the crawl is still running.

* Remove the issue reference

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-07-29 19:30:03 +05:00
Laerte Pereira 01447f9965
fix(commands/parse): Restore request callback before invoking spider (#7803) 
* fix(commands/parse): restore request callback before invoking spider

* Add other test
 2026-07-29 16:00:24 +05:00
Adrian 0cbb20e8e8
Treat broken cache records as cache misses (#7805) 2026-07-29 14:39:27 +05:00
Adrian 8b5147ae2e
Improve test coverage for scrapy.pipelines (#7798) 
* Improve test coverage for scrapy.pipelines

* Restore old Pillow support
 2026-07-29 14:32:18 +05:00
Youssef Mohamed bc5b5fb1f6
Add Request.to_curl() (#7743) (#7802) 2026-07-29 13:47:15 +05:00
Andrey Rakhmatullin e7d8b34e73
Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
Adrian ad816d2b3a
Improve test coverage for scrapy.cmdline (#7795) 2026-07-28 17:04:54 +05:00
Adrian 5a65bdcc18
Improve spiders coverage (#7768) 
* Skip the deprecated scrapy.mail in coverage data

* Improve CrawlSpider coverage

* Improve XMLFeedSpider coverage

* Improve SitemapSpider coverage

* Solve mypy issues

* Align new spider tests with the shared test helper structure
 2026-07-28 15:24:04 +05:00
Shadow_Lu 59ebb26e60
Fix CaseInsensitiveDict.copy() sharing state with the original (#7783) 
* Fix CaseInsensitiveDict.copy() sharing state with the original

* Address review: don't re-normalise in __copy__, keep _keys in sync in __ior__

UserDict.__ior__ writes self.data directly, bypassing __setitem__, so
_keys never learned about the new keys.
 2026-07-28 15:21:55 +05:00
Andrey Rakhmatullin 7e8b58a2b2
Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
Adrian f3693aa8ba
Fix port issue with cached DNS (#7772) 
* Fix port issue with cached DNS

* Keep ports off the cache

* Complete coverage
 2026-07-27 21:23:56 +05:00