RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,250 Commits 49 Branches 132 Tags 152 MiB
Commit Graph

11250 Commits

Author SHA1 Message Date
Adrian f02a99fe71
Add doc sections for callbacks and errbacks (#7821) 2026-07-30 20:15:15 +05:00
Adrian 433603e6ca
Add AWS_MAX_POOL_CONNECTIONS (#7794) 2026-07-30 15:51:45 +02:00
Adrian 98696efa80
Export item fields in declaration order (#7824) 2026-07-30 15:46:45 +05:00
Laerte Pereira 5b4828a012
docs(practices): Remove scrapoxy mention (#7817) 2026-07-30 08:02:54 +02:00
Fandu b2d4eedea8
Fix immediate delivery of full feed export batches (#7730) (#7733) 
* Store full feed batches before spider closes (#7730)

Start closing and storing each batch as soon as it reaches the configured item count. Track unfinished close tasks so spider shutdown still waits for all deliveries before emitting the exporter-closed signal.

Add an end-to-end regression test that verifies the first batch is stored while the crawl is still running.

* Remove the issue reference

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-07-29 19:30:03 +05:00
Laerte Pereira 01447f9965
fix(commands/parse): Restore request callback before invoking spider (#7803) 
* fix(commands/parse): restore request callback before invoking spider

* Add other test
 2026-07-29 16:00:24 +05:00
Adrian 0cbb20e8e8
Treat broken cache records as cache misses (#7805) 2026-07-29 14:39:27 +05:00
Adrian 8b5147ae2e
Improve test coverage for scrapy.pipelines (#7798) 
* Improve test coverage for scrapy.pipelines

* Restore old Pillow support
 2026-07-29 14:32:18 +05:00
Youssef Mohamed bc5b5fb1f6
Add Request.to_curl() (#7743) (#7802) 2026-07-29 13:47:15 +05:00
Andrey Rakhmatullin e7d8b34e73
Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
Adrian ad816d2b3a
Improve test coverage for scrapy.cmdline (#7795) 2026-07-28 17:04:54 +05:00
Adrian 5a65bdcc18
Improve spiders coverage (#7768) 
* Skip the deprecated scrapy.mail in coverage data

* Improve CrawlSpider coverage

* Improve XMLFeedSpider coverage

* Improve SitemapSpider coverage

* Solve mypy issues

* Align new spider tests with the shared test helper structure
 2026-07-28 15:24:04 +05:00
Shadow_Lu 59ebb26e60
Fix CaseInsensitiveDict.copy() sharing state with the original (#7783) 
* Fix CaseInsensitiveDict.copy() sharing state with the original

* Address review: don't re-normalise in __copy__, keep _keys in sync in __ior__

UserDict.__ior__ writes self.data directly, bypassing __setitem__, so
_keys never learned about the new keys.
 2026-07-28 15:21:55 +05:00
Andrey Rakhmatullin 7e8b58a2b2
Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
Andrey Rakhmatullin a5bc43e34c
Don't generate test keys concurrently with xdist. (#7792) 2026-07-28 09:51:52 +02:00
Adrian f3693aa8ba
Fix port issue with cached DNS (#7772) 
* Fix port issue with cached DNS

* Keep ports off the cache

* Complete coverage
 2026-07-27 21:23:56 +05:00
Adrian 25e6884e2f
Address recursion and media ignore-request reporting issues (#7673) 
* Address recursion and media ignore-request reporting issues

* Improve test coverage

* Address feedback
 2026-07-27 20:34:10 +05:00
Andrey Rakhmatullin cec86f216e
Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
SpiliosDmk 13be37e4b1
Add type hints to test_link.py and test_downloadermiddleware_stats.py (#7785) 2026-07-27 10:07:32 +02:00
SpiliosDmk e710b9c18e
core: fix verb tense in http2 stream comment (#7778) 
'needs to be send' -> 'needs to be sent'. Comment-only, no functional change.
 2026-07-25 10:20:30 +05:00
SpiliosDmk 96195e4a61
docs: fix a/an article before XML in faq.rst (#7779) 
'a XML document' -> 'an XML document' (XML is pronounced with a vowel sound).
 2026-07-25 10:20:18 +05:00
SpiliosDmk e4ae4aad52
docs: fix missing 'of' in faq.rst (#7780) 
'instead joining the strings' -> 'instead of joining the strings'.
 2026-07-25 10:19:53 +05:00
Adrian 58ed9fdccc
Document urlparse_cached (#7777) 2026-07-24 20:08:58 +05:00
Adrian 41bb09741a
Use autodoc for contracts (#7775) 2026-07-24 15:35:54 +02:00
Adrian 0b578c1cbf
trackref: use autodoc (#7771) 2026-07-24 16:24:54 +05:00
Adrian abbc024bbe
Use autodoc for stats.rst (#7769) 2026-07-23 19:48:59 +05:00
Adrian 67e5282684
Use autodoc for exceptions and improve their docs (#7767) 2026-07-23 16:02:41 +05:00
Mridankan Mandal 8489b3dad8
Support iterable values in BaseSettings.update() (#7763) 2026-07-23 12:37:21 +02:00
Andrey Rakhmatullin 628a3afbbd
Remove deprecated xmliter(), deprecate re_rsearch() (#7765) 2026-07-23 10:38:56 +02:00
Andrey Rakhmatullin 56dee203e9
Use httpx2 as a preferred alternative to httpx. (#7762) 2026-07-23 10:29:22 +02:00
Adrian 1157b3e235
Add a middleware to detect response.meta copies (#7588) 2026-07-22 13:27:31 +02:00
Adrian a54c438da1
Provide extras for optional dependencies (#7596) 2026-07-22 10:44:26 +02:00
Andrey Rakhmatullin 64358f547b
Replace import tests with find_spec() or importorskip(). (#7755) 2026-07-22 08:58:05 +02:00
Andrey Rakhmatullin ca21306df7
Reactorless import hook improvements. (#7747) 
* Reactorless import hook improvements.

* Replace prints with logging.

* More tests.

* Uninstall the import hook earlier.

* Add pragma: no branch.
 2026-07-21 19:58:40 +05:00
Mridankan Mandal 1ddf024b89
Log ignored responses exceeding DOWNLOAD_MAXSIZE (#7742) 
* Log ignored responses exceeding DOWNLOAD_MAXSIZE

Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>

* Use caplog in compression warning test

Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>

* Make compression warning test robust

Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>

---------

Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
 2026-07-21 18:15:30 +05:00
Andrey Rakhmatullin 394c2797f3
Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
Andrey Rakhmatullin e322905255
Assorted test fixes, part 3. (#7741) 
* Assorted test fixes, part 3.

* Fix asyncio_enabled_reactor_different_loop.py on 3.14.
 2026-07-21 15:54:43 +05:00
Andrey Rakhmatullin 4ee3676464
Test restructuring (#7736) 2026-07-21 12:44:56 +02:00
Andrey Rakhmatullin 36bf1185e5
Modernize code examples, drop docs for nonexistent `MEMDEBUG_NOTIFY` (#7737) 2026-07-21 12:22:04 +02:00
Andrey Rakhmatullin b1b9efb473
Rewrite GCSFilesStore tests to use mocking. (#7727) 2026-07-21 11:43:36 +02:00
Andrey Rakhmatullin 1c5404dce5
Add docs about running Scrapy from apps and notebooks. (#7751) 2026-07-21 11:23:52 +02:00
Andrey Rakhmatullin 4d4a04f318
Improve a flaky test_stacktracedump_dump_stacktrace(). (#7753) 2026-07-21 13:17:56 +05:00
Andrey Rakhmatullin e80f94fe8a
Tests for extensions.debug and extensions.memdebug. (#7749) 
* Tests for extensions.debug and extensions.memdebug.

* Use a unique class in test_crawl_sets_stats.

* Don't assume a single log message.
 2026-07-20 15:21:28 +02:00
Madan kumar 7faf20c6b5
Merge repeated curl -d/--data options in Request.from_curl() (#7728) 
curl merges repeated -d/--data/--data-raw options into a single request
body joined with "&" (e.g. `curl -d a=1 -d b=2` sends `a=1&b=2`). scrapy's
DataAction kept only the last value, silently dropping the earlier ones, so
`Request.from_curl("curl -d a=1 -d b=2 ...")` produced a body of just `b=2`.
Accumulate the values with "&" to match curl. -H/--header and -b/--cookie
already accumulate via action="append"; -d was the only repeatable data flag
that didn't.
 2026-07-20 15:18:55 +02:00
Andrey Rakhmatullin a591d15c04
Deprecate Spider.log(). (#7739) 2026-07-20 14:59:55 +02:00
Andrey Rakhmatullin 11d7a05a6f
Deprecate is_secure=False for s3://. (#7738) 2026-07-20 14:59:11 +02:00
Andrey Rakhmatullin d8ba1571e7
Assorted docs fixes, part 2. (#7725) 
* Assorted docs fixes, part 2.

* Second pass.

* Address feedback.
 2026-07-14 00:49:44 +05:00
Adrian b3670369b8
Solve the feed Path issue (#7674) 
* Solve the feed Path issue

* Address additional, related issues
 2026-07-13 22:46:41 +05:00
Andrey Rakhmatullin c9446931a8
Use external mitmproxy. (#7720) 2026-07-09 17:02:05 +02:00
Shaquon Kelley 9d9950df69
Fix silent data loss in CsvItemExporter (closes #4002) (#7651) 
Adds state tracking to CsvItemExporter to detect when auto-detected fields drop data on items. Injects a warning instructing users to configure FEED_EXPORT_FIELDS. Includes unit tests. Corrects the false positive logic in stalled PR #7613.
 2026-07-08 17:49:29 +05:00