RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

11208 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 6ba6b032ad
Fix handling meta["download_slot"] == None. (#7172) 2025-12-11 09:46:51 +01:00
Andrey Rakhmatullin 9bfa58e36c
Deprecate send_catch_log_deferred(). (#7161) 
* Add a test for not having pending tasks.

* Refactor TestFeedExporterSignals.

* Refactor FeedExporter.close_spider().

* More engine start/stop robustness.

* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().

* Add pragma: no cover.

* Warn on signal handlers returning a Deferred.

* Make _pending_close_coros an instance attribute.

* Remove an unused function.

* Remove the outdated TODO.
 2025-12-10 14:42:49 +05:00
Andrey Rakhmatullin 5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws (#7069) 
* Restore coverage in MiddlewareManager spider arg handling.

* Move MiddlewareManager._warn_spider_arg() tests.

* Add DownloaderMiddlewareManager.download_async().

* Make download_func async.

* Inline MiddlewareManager._warn_spider_arg().

* Deprecate returning deferreds from downloader middlewares.

* Update the downloader mw doc about async def.

* Use global_object_name() instead of repr().

* Update docs/topics/downloader-middleware.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update downloader-middleware.rst

---------

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2025-12-09 20:07:36 +05:00
Andrey Rakhmatullin 7ed20ee7f3
Limit the queue size in _parallel_asyncio(). (#7159) 2025-12-09 15:24:40 +01:00
Andrey Rakhmatullin 483e059d59
Deprecate returning Deferreds from spider callbacks and errbacks. (#7151) 2025-12-09 15:03:32 +01:00
Laerte Pereira 11073c8680
Deprecate spider attributes that can be replaced by settings, round 2 (#7039) 
* Move duplicate code to utils

* move new function to end

* draft

* update docs

* Update tests

* Update test name

* rename test

* rollback signature

* leftover

* sort

* Rollback some changes

* Rollback download_delay warning

* Rollback

* fix checks

* Remove unused imports

* Add pragma: no cover.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-12-01 01:26:17 +05:00
Leonardo Tozzo 1e8de24380
Merge pull request #7145 from Icorebleidd/master 
Fix duplicate assertion in test_utils_deprecate
 2025-11-28 01:16:02 +05:00
Andrey Rakhmatullin 0adc561348
Add a list of remaining Deferred APIs. (#7146) 2025-11-27 20:13:04 +05:00
Rui Xi 813fd9f1ac
Restore brotlicffi support (#7160) 
* restore brotlicffi dependency

* pre-commit
 2025-11-25 12:06:55 +05:00
Andrey Rakhmatullin 4cb0144b39 Merge tag '2.13.4' 
Bump version: 2.13.3 → 2.13.4
 2025-11-17 16:47:54 +05:00
Andrey Rakhmatullin 2f62ab532d Bump version: 2.13.3 → 2.13.4 2025-11-17 16:30:37 +05:00
Andrey Rakhmatullin 31a9c03c24
Release notes for 2.13.4. (#7144) 
* Release notes for 2.13.4.

* Formatting.
 2025-11-17 16:30:04 +05:00
Rui Xi c44b8df6c7 Cherry-pick: Mitigate brotli and deflate decompression bombs DoS (#7134) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-11-17 16:25:00 +05:00
Rui Xi 14737e91ed
Mitigate brotli and deflate decompression bombs DoS (#7134) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-11-17 14:21:50 +05:00
Adrián Chaves d091256c58 Remove the deprecated sphinx-hoverxref (#6922) 2025-11-14 15:39:21 +05:00
Andrey Rakhmatullin c83ca70db3 Don't force the unavailable sysmon coverage core. (#7137) 2025-11-13 23:26:08 +05:00
Andrey Rakhmatullin 85e4e6c42b Pin Twisted to <= 25.5.0 due to internal API changes. 2025-11-13 23:25:51 +05:00
Andrey Rakhmatullin 426aafddca
Don't force the unavailable sysmon coverage core. (#7137) 2025-11-13 23:22:52 +05:00
Mikhail Korobov db37040a09
Merge pull request #7142 from wRAR/pin-twisted 
Pin Twisted to <= 25.5.0 due to internal API changes.
 2025-11-13 22:48:33 +05:00
Andrey Rakhmatullin ba30e8b82c Pin Twisted to <= 25.5.0 due to internal API changes. 2025-11-10 00:58:27 +05:00
Vasiliy Kiryanov 8c5fa6e6ae
Line buffering (#7094) 2025-10-27 15:42:43 +01:00
Andrey Rakhmatullin 804ae167df
Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
Andrey Rakhmatullin 61b4befc60
Remove code deprecated in 2.12.0 (#7126) 
* Remove smaller things.

* Remove from_settings().

* Remove old MediaPipeline init code.

* Cleanup.

* Typo.
 2025-10-27 17:14:38 +05:00
Andrey Rakhmatullin d414d393d4
Remove Python 3.9 support (#7121) 2025-10-27 12:37:49 +01:00
Andrey Rakhmatullin e48a1bdde3
Speed up and refactor command tests (#7118) 2025-10-27 12:28:16 +01:00
Andrey Rakhmatullin 9cce6c30db
Undeprecate download_delay. (#7117) 2025-10-24 17:44:43 +05:00
Andrew Murray 5afc9b0221
Automatically transpose images using EXIF data (#6975) 
* Automatically transpose images using EXIF data

* Added test

---------

Co-authored-by: Andrew Murray <radarhere@users.noreply.github.com>
 2025-10-24 16:08:23 +05:00
Adrián Chaves eb496470f1
Add missing doc deps for new examples (#7116) 2025-10-24 11:58:18 +02:00
Michael a5bbeb2586
Expanded documentation for issue #6676 (#6957) 2025-10-24 10:30:33 +02:00
sstegmanbsd 2d073a9c0d
Fix typo in logging documentation (#7109) 2025-10-20 15:52:46 +02:00
Wes Galbraith e98c1644ce
Add unit tests that check download handlers behave like basic http clients (#6946) (#7059) 2025-10-20 13:08:26 +02:00
Andrey Rakhmatullin b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. (#7079) 2025-10-16 13:15:07 +02:00
Vasiliy Kiryanov 7b215c6578
UTF-8 BOM at the beginning of the file ignored (#7095) 2025-10-15 18:15:07 +04:00
adglee 4865a500b6
resolves #6876 (#6977) 2025-10-09 22:14:34 +02:00
Sebastien Dionne a02abdcf63
Fix typos in documentation (#7076) 2025-10-06 11:43:36 +02:00
Andrey Rakhmatullin c577771838
Drop tests/upper-constraints.txt. (#7073) 2025-10-06 11:32:20 +02:00
Andrey Rakhmatullin 10850e7d29
Fix extra stdout output in tests (#7070) 2025-10-06 11:19:34 +02:00
Andrey Rakhmatullin 1c6ba00fd0
Move LogCounterHandler into a new extension (#7046) 
* Move LogCounterHandler into Crawler.crawl().

* Add the LogCount extension.

* Update test_spider_custom_settings_log_level().

* Typo.

* Update docs.
 2025-10-03 20:14:11 +05:00
Andrey Rakhmatullin 798390c096
Improve performance of deprecated spider arg checks. (#7045) 2025-10-03 14:17:46 +02:00
Andrey Rakhmatullin dd0b071bcc
Deprecate sync methods of ItemPipelineManager/ITEM_PROCESSOR. (#7043) 2025-10-03 14:15:42 +02:00
Laerte Pereira f2531808f3
docs: Add Edit on GitHub (#7058) 2025-09-28 13:16:47 +05:00
Andrey Rakhmatullin 393d715205
Drop PyPy 3.10 support, restore coverage for subprocesses (#7050) 2025-09-15 11:24:09 +02:00
Ashar Khan d239fcf936
Fix get_meta_refresh to correctly resolve relative URLs using <base> tag (#7047) 2025-09-10 15:17:16 +02:00
Andrey Rakhmatullin 2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. (#7037) 
* Deprecate the spider arg of downloader middleware methods.

* Add a test for deprecated spider args in downloader mws.
 2025-09-02 21:51:47 +05:00
Andrey Rakhmatullin c097921c44
Extract @_warn_spider_arg. (#7033) 
* Extract @_warn_spider_arg.

* Also deprecate passing spider=None.
 2025-09-01 20:17:13 +05:00
Andrey Rakhmatullin 80beec41b5
Log when AsyncCrawlerProcess or CrawlerProcess is instantiated. (#7034) 2025-09-01 13:51:11 +02:00
M Ikram Ullah Khan 00b2be0943
Add unit tests for the StatsMailer extension (#7013) 2025-09-01 11:43:57 +02:00
M Ikram Ullah Khan 3ee4a52aa1
Raise NotConfigured when FILES_STORE or IMAGES_STORE is None or empty (#6969) 2025-09-01 11:39:39 +02:00
Andrey Rakhmatullin ed63fa94d6
Deprecate spider args of middlewares and pipelines. (#7006) 
* Deprecate spider args of spider middleware methods.

* Deprecate the spider arg of pipeline process_item().

* Fix a typing issue.

* Deprecate the spider arg of pipeline {open,close}_spider().

* Cleanup.

* Update docs.

* Add pragma: no cover to some deprecated code.

* More tests.

* More tests.

* More tests.

* Simplify _process_parallel().
 2025-08-21 13:23:10 +05:00
Andrey Rakhmatullin b68330811b
Deprecate spider arguments of StatsCollector methods (#7011) 2025-08-20 09:52:22 +02:00