RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,990 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

2883 Commits

Author SHA1 Message Date
Adrian Chaves 498b4fc1a4 Bump version: 2.14.1 → 2.14.2 2026-03-12 15:45:16 +01:00
Adrian Chaves 378bb68039 Proofread the release notes 2026-03-12 15:44:54 +01:00
Adrian Chaves 945b787a26 Merge remote-tracking branch 'cwxj-rr6w-m6w7/fix-referer-policy-handling' into 2.14 2026-03-12 13:37:24 +01:00
Adrian Chaves 8974580e43 Reword the release note entry to consider the 301 redirect fix a security bug fix 2026-02-03 20:38:11 +01:00
Adrian Chaves ba3d7bc7a8 Remove the non-standard 307/308 handling, and align other aspects with the standard 2026-01-21 18:46:05 +01:00
Adrián Chaves a39545195e
allow to override → allow overriding 2026-01-20 12:20:37 +01:00
Adrián Chaves 1b9c8b55da
not → now 2026-01-20 12:13:44 +01:00
Adrian Chaves 2651d48f20 Cover the 301 behavior change in the release notes 2026-01-19 19:25:00 +01:00
Adrian Chaves 99d5d58e20 Add a release notes entry 2026-01-19 19:19:02 +01:00
Adrian Chaves 6f03f3250b Add versionadded 2026-01-19 16:46:53 +01:00
Adrian Chaves b6e5c58ae7 Fix arbitrary callable execution from the Referrer-Policy response header 2026-01-19 16:44:03 +01:00
Eugenio Lacuesta 0b9d8da09d
Bump version: 2.14.0 → 2.14.1 2026-01-12 16:25:43 -03:00
Eugenio Lacuesta c9fbf6c599
Release notes for 2.14.1 (#7217) 
* Release notes for 2.14.1

* Update docs/news.rst

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* Update docs/news.rst

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>

* Update docs/news.rst

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* Update message about github action change

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-01-12 16:23:51 -03:00
Andrey Rakhmatullin 3ac1192f35 Bump version: 2.13.4 → 2.14.0 2026-01-05 15:06:13 +05:00
Andrey Rakhmatullin 7bef98b4f1
Release notes for 2.14.0 (#7035) 
* Initial release notes for 2.14.0, up to ed63fa9.

* Cover 2.14 in the release notes up to 393d715.

* Cover 2.14 in the release notes up to eb49647.

* Cover 2.14 in the release notes up to 426aafd.

* Cover 2.14 in the release notes up to 1e8de24.

* Cover 2.14 in the release notes up to 5a7e132.

* Bump sphinx-lint.

* Finalize the 2.14.0 release notes.

* Drop more of the old versionadded directives.

* Address feedback.
 2026-01-05 14:51:05 +05:00
Andrey Rakhmatullin d1bd8eb49f
Async API for download handlers. (#7164) 2025-12-31 19:03:16 +01:00
Andrey Rakhmatullin a2463325db
Allow Crawler.crawl_async() without asyncio. (#7202) 2025-12-31 18:33:59 +01:00
Adham 3llam 9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation (#7198) 2025-12-22 10:00:39 +01:00
Andrey Rakhmatullin 180ca39b23
Deprecate returning Deferreds from pipeline methods (#7179) 
* Add tests for exceptions in pipelines.

* Deprecate returning Deferreds from pipeline process_item().

* Deprecate returning Deferreds from pipeline {open,close}_spider().

* Update the custom pipeline docs.
 2025-12-16 13:35:07 +05:00
Andrey Rakhmatullin c49ae2115a
Refactor MediaPipeline.process_item() to async def (#7177) 2025-12-15 10:02:04 +01:00
Andrey Rakhmatullin 588f3d4f65
Remove docs mentioning deprecated spider attrs. (#7176) 2025-12-15 09:51:56 +01:00
Thalison Fernandes d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue (#6940) 
* Change default priority queue to DownloaderAwarePriorityQueue

* Fix documentation building

* Simplify test_start_already_running_exception changes.

* Modernize the test.

* Fix TestEngineCloseSpider.

* Fix typing.

* Remove special slot=None handling.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-12-11 11:25:17 +01:00
Andrey Rakhmatullin 9bfa58e36c
Deprecate send_catch_log_deferred(). (#7161) 
* Add a test for not having pending tasks.

* Refactor TestFeedExporterSignals.

* Refactor FeedExporter.close_spider().

* More engine start/stop robustness.

* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().

* Add pragma: no cover.

* Warn on signal handlers returning a Deferred.

* Make _pending_close_coros an instance attribute.

* Remove an unused function.

* Remove the outdated TODO.
 2025-12-10 14:42:49 +05:00
Andrey Rakhmatullin 5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws (#7069) 
* Restore coverage in MiddlewareManager spider arg handling.

* Move MiddlewareManager._warn_spider_arg() tests.

* Add DownloaderMiddlewareManager.download_async().

* Make download_func async.

* Inline MiddlewareManager._warn_spider_arg().

* Deprecate returning deferreds from downloader middlewares.

* Update the downloader mw doc about async def.

* Use global_object_name() instead of repr().

* Update docs/topics/downloader-middleware.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update downloader-middleware.rst

---------

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2025-12-09 20:07:36 +05:00
Andrey Rakhmatullin 0adc561348
Add a list of remaining Deferred APIs. (#7146) 2025-11-27 20:13:04 +05:00
Andrey Rakhmatullin 4cb0144b39 Merge tag '2.13.4' 
Bump version: 2.13.3 → 2.13.4
 2025-11-17 16:47:54 +05:00
Andrey Rakhmatullin 2f62ab532d Bump version: 2.13.3 → 2.13.4 2025-11-17 16:30:37 +05:00
Andrey Rakhmatullin 31a9c03c24
Release notes for 2.13.4. (#7144) 
* Release notes for 2.13.4.

* Formatting.
 2025-11-17 16:30:04 +05:00
Adrián Chaves d091256c58 Remove the deprecated sphinx-hoverxref (#6922) 2025-11-14 15:39:21 +05:00
Andrey Rakhmatullin 804ae167df
Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
Andrey Rakhmatullin 61b4befc60
Remove code deprecated in 2.12.0 (#7126) 
* Remove smaller things.

* Remove from_settings().

* Remove old MediaPipeline init code.

* Cleanup.

* Typo.
 2025-10-27 17:14:38 +05:00
Andrey Rakhmatullin d414d393d4
Remove Python 3.9 support (#7121) 2025-10-27 12:37:49 +01:00
Adrián Chaves eb496470f1
Add missing doc deps for new examples (#7116) 2025-10-24 11:58:18 +02:00
Michael a5bbeb2586
Expanded documentation for issue #6676 (#6957) 2025-10-24 10:30:33 +02:00
sstegmanbsd 2d073a9c0d
Fix typo in logging documentation (#7109) 2025-10-20 15:52:46 +02:00
Andrey Rakhmatullin b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. (#7079) 2025-10-16 13:15:07 +02:00
Sebastien Dionne a02abdcf63
Fix typos in documentation (#7076) 2025-10-06 11:43:36 +02:00
Andrey Rakhmatullin 1c6ba00fd0
Move LogCounterHandler into a new extension (#7046) 
* Move LogCounterHandler into Crawler.crawl().

* Add the LogCount extension.

* Update test_spider_custom_settings_log_level().

* Typo.

* Update docs.
 2025-10-03 20:14:11 +05:00
Laerte Pereira f2531808f3
docs: Add Edit on GitHub (#7058) 2025-09-28 13:16:47 +05:00
Andrey Rakhmatullin 2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. (#7037) 
* Deprecate the spider arg of downloader middleware methods.

* Add a test for deprecated spider args in downloader mws.
 2025-09-02 21:51:47 +05:00
Andrey Rakhmatullin ed63fa94d6
Deprecate spider args of middlewares and pipelines. (#7006) 
* Deprecate spider args of spider middleware methods.

* Deprecate the spider arg of pipeline process_item().

* Fix a typing issue.

* Deprecate the spider arg of pipeline {open,close}_spider().

* Cleanup.

* Update docs.

* Add pragma: no cover to some deprecated code.

* More tests.

* More tests.

* More tests.

* Simplify _process_parallel().
 2025-08-21 13:23:10 +05:00
Andrey Rakhmatullin b68330811b
Deprecate spider arguments of StatsCollector methods (#7011) 2025-08-20 09:52:22 +02:00
Andrey Rakhmatullin edba1ad572
Refactor RobotsTxtMiddleware to use async def process_request(). (#6802) 2025-08-18 18:40:46 +02:00
Felipe Benevolo 89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() (#6842) 
* added async_download()

* add case tests for ExecutionEngine download() and async_download()

* change async_download() to download_async() in Engine class and subsequent tests

* fixed reactor import in test_engine.py

* Simplify TestEngineDownloadAsync.

* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks

* Simplify ExecutionEngine.download().

* Remove an unneeded import.

* Refactor ExecutionEngine.download{,_async}() tests.

* Deprecate ExecutionEngine.download().

* Update doc examples to use download_async().

* Replace uses of download() with download_async().

* Fix test_downloadermiddleware_robotstxt.py.

* Remove a stray file.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-07-27 23:17:51 +05:00
Andrey Rakhmatullin 552f2fb91e
Generic improvements related to reactorless tests (#6968) 
* Skip doctests that import the reactor.

* Extract FTPDownloadHandler tests.

* Skip TestHttps2ClientProtocol via pytestmark.

* Replace some explicit reactor.callLater() in tests.

* Simplify TestRequestSendOrder._test_request_order().
 2025-07-25 13:35:00 +05:00
Andrey Rakhmatullin d9ef0350d8 Merge remote-tracking branch 'origin/2.13' 2025-07-02 20:46:02 +05:00
Andrey Rakhmatullin 155a504f24 Bump version: 2.13.2 → 2.13.3 2025-07-02 20:39:46 +05:00
Andrey Rakhmatullin cf465bf644
Release notes for 2.13.3. (#6934) 
* Release notes for 2.13.3.

* Address feedback.
 2025-07-02 20:36:01 +05:00
Laerte Pereira dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting (#6921) 
* Deprecate CONCURRENT_REQUESTS_PER_IP setting

* Typing fix

* PR review

* tests

* oops

* Update scrapy/settings/default_settings.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* update logic

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-06-30 16:51:05 +05:00
Andrey Rakhmatullin 3843091c5f
Mark abstract base classes as such. (#6930) 
* Mark abstract base classes as such.

* Remove an unneeded method.

* Fix exporter test coverage.
 2025-06-30 15:14:34 +05:00