RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,990 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

10990 Commits

Author SHA1 Message Date
Adrian Chaves 498b4fc1a4 Bump version: 2.14.1 → 2.14.2 2026-03-12 15:45:16 +01:00
Adrian Chaves 378bb68039 Proofread the release notes 2026-03-12 15:44:54 +01:00
Adrian Chaves 8e28f938d2 Make test_no_warning_when_referer_middleware_present less brittle 2026-03-12 14:29:51 +01:00
Adrian Chaves 886131c7b2 Run pre-commit 2026-03-12 13:54:00 +01:00
Adrian Chaves 945b787a26 Merge remote-tracking branch 'cwxj-rr6w-m6w7/fix-referer-policy-handling' into 2.14 2026-03-12 13:37:24 +01:00
Adrian Chaves 8974580e43 Reword the release note entry to consider the 301 redirect fix a security bug fix 2026-02-03 20:38:11 +01:00
Adrian Chaves ba3d7bc7a8 Remove the non-standard 307/308 handling, and align other aspects with the standard 2026-01-21 18:46:05 +01:00
Adrián Chaves 04db6a5424
Add a docstring to _load_policy_class() 2026-01-20 12:35:00 +01:00
Adrián Chaves a39545195e
allow to override → allow overriding 2026-01-20 12:20:37 +01:00
Adrián Chaves 842d0becf0
Rename test function 2026-01-20 12:15:25 +01:00
Adrián Chaves 1b9c8b55da
not → now 2026-01-20 12:13:44 +01:00
Adrian Chaves 2651d48f20 Cover the 301 behavior change in the release notes 2026-01-19 19:25:00 +01:00
Adrian Chaves 99d5d58e20 Add a release notes entry 2026-01-19 19:19:02 +01:00
Adrian Chaves f7f18123eb Remove body on 307/308 redirects and switch to GET on 301 2026-01-19 19:15:20 +01:00
Adrian Chaves 6f03f3250b Add versionadded 2026-01-19 16:46:53 +01:00
Adrian Chaves b6e5c58ae7 Fix arbitrary callable execution from the Referrer-Policy response header 2026-01-19 16:44:03 +01:00
Eugenio Lacuesta 0b9d8da09d
Bump version: 2.14.0 → 2.14.1 2026-01-12 16:25:43 -03:00
Eugenio Lacuesta c9fbf6c599
Release notes for 2.14.1 (#7217) 
* Release notes for 2.14.1

* Update docs/news.rst

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* Update docs/news.rst

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>

* Update docs/news.rst

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* Update message about github action change

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-01-12 16:23:51 -03:00
Santiago Beltrán Ruiz e30ba7d4ca
(#7180) Update codecov test results action to v5 on all CI workflows. (#7215) 2026-01-12 08:31:53 +01:00
Andrey Rakhmatullin 0f07b2e38c
Pass the spider arg to custom stat collectors {open,close}_spider(). (#7213) 2026-01-09 16:57:53 +01:00
Andrey Rakhmatullin 1af283387f
Deprecate maybeDeferred_coro(). (#7212) 2026-01-09 01:09:39 +05:00
Andrey Rakhmatullin 3ac1192f35 Bump version: 2.13.4 → 2.14.0 2026-01-05 15:06:13 +05:00
Andrey Rakhmatullin 7bef98b4f1
Release notes for 2.14.0 (#7035) 
* Initial release notes for 2.14.0, up to ed63fa9.

* Cover 2.14 in the release notes up to 393d715.

* Cover 2.14 in the release notes up to eb49647.

* Cover 2.14 in the release notes up to 426aafd.

* Cover 2.14 in the release notes up to 1e8de24.

* Cover 2.14 in the release notes up to 5a7e132.

* Bump sphinx-lint.

* Finalize the 2.14.0 release notes.

* Drop more of the old versionadded directives.

* Address feedback.
 2026-01-05 14:51:05 +05:00
Andrey Rakhmatullin d1bd8eb49f
Async API for download handlers. (#7164) 2025-12-31 19:03:16 +01:00
Andrey Rakhmatullin a2463325db
Allow Crawler.crawl_async() without asyncio. (#7202) 2025-12-31 18:33:59 +01:00
Adham 3llam 9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation (#7198) 2025-12-22 10:00:39 +01:00
Andrey Rakhmatullin 180ca39b23
Deprecate returning Deferreds from pipeline methods (#7179) 
* Add tests for exceptions in pipelines.

* Deprecate returning Deferreds from pipeline process_item().

* Deprecate returning Deferreds from pipeline {open,close}_spider().

* Update the custom pipeline docs.
 2025-12-16 13:35:07 +05:00
Andrey Rakhmatullin 5a7e132486
Restore and deprecate the spider arg of Downloader._get_slot(). (#7178) 2025-12-15 11:42:24 +01:00
Andrey Rakhmatullin c49ae2115a
Refactor MediaPipeline.process_item() to async def (#7177) 2025-12-15 10:02:04 +01:00
Andrey Rakhmatullin 588f3d4f65
Remove docs mentioning deprecated spider attrs. (#7176) 2025-12-15 09:51:56 +01:00
Thalison Fernandes d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue (#6940) 
* Change default priority queue to DownloaderAwarePriorityQueue

* Fix documentation building

* Simplify test_start_already_running_exception changes.

* Modernize the test.

* Fix TestEngineCloseSpider.

* Fix typing.

* Remove special slot=None handling.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-12-11 11:25:17 +01:00
DenisC 1a3e343dc4
Update conftest.py to refactor autouse pytest.skip() fixtures into pytest_runtest_setup() (#7173) 2025-12-11 09:48:25 +01:00
Andrey Rakhmatullin 6ba6b032ad
Fix handling meta["download_slot"] == None. (#7172) 2025-12-11 09:46:51 +01:00
Andrey Rakhmatullin 9bfa58e36c
Deprecate send_catch_log_deferred(). (#7161) 
* Add a test for not having pending tasks.

* Refactor TestFeedExporterSignals.

* Refactor FeedExporter.close_spider().

* More engine start/stop robustness.

* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().

* Add pragma: no cover.

* Warn on signal handlers returning a Deferred.

* Make _pending_close_coros an instance attribute.

* Remove an unused function.

* Remove the outdated TODO.
 2025-12-10 14:42:49 +05:00
Andrey Rakhmatullin 5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws (#7069) 
* Restore coverage in MiddlewareManager spider arg handling.

* Move MiddlewareManager._warn_spider_arg() tests.

* Add DownloaderMiddlewareManager.download_async().

* Make download_func async.

* Inline MiddlewareManager._warn_spider_arg().

* Deprecate returning deferreds from downloader middlewares.

* Update the downloader mw doc about async def.

* Use global_object_name() instead of repr().

* Update docs/topics/downloader-middleware.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update downloader-middleware.rst

---------

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2025-12-09 20:07:36 +05:00
Andrey Rakhmatullin 7ed20ee7f3
Limit the queue size in _parallel_asyncio(). (#7159) 2025-12-09 15:24:40 +01:00
Andrey Rakhmatullin 483e059d59
Deprecate returning Deferreds from spider callbacks and errbacks. (#7151) 2025-12-09 15:03:32 +01:00
Laerte Pereira 11073c8680
Deprecate spider attributes that can be replaced by settings, round 2 (#7039) 
* Move duplicate code to utils

* move new function to end

* draft

* update docs

* Update tests

* Update test name

* rename test

* rollback signature

* leftover

* sort

* Rollback some changes

* Rollback download_delay warning

* Rollback

* fix checks

* Remove unused imports

* Add pragma: no cover.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-12-01 01:26:17 +05:00
Leonardo Tozzo 1e8de24380
Merge pull request #7145 from Icorebleidd/master 
Fix duplicate assertion in test_utils_deprecate
 2025-11-28 01:16:02 +05:00
Andrey Rakhmatullin 0adc561348
Add a list of remaining Deferred APIs. (#7146) 2025-11-27 20:13:04 +05:00
Rui Xi 813fd9f1ac
Restore brotlicffi support (#7160) 
* restore brotlicffi dependency

* pre-commit
 2025-11-25 12:06:55 +05:00
Andrey Rakhmatullin 4cb0144b39 Merge tag '2.13.4' 
Bump version: 2.13.3 → 2.13.4
 2025-11-17 16:47:54 +05:00
Andrey Rakhmatullin 2f62ab532d Bump version: 2.13.3 → 2.13.4 2025-11-17 16:30:37 +05:00
Andrey Rakhmatullin 31a9c03c24
Release notes for 2.13.4. (#7144) 
* Release notes for 2.13.4.

* Formatting.
 2025-11-17 16:30:04 +05:00
Rui Xi c44b8df6c7 Cherry-pick: Mitigate brotli and deflate decompression bombs DoS (#7134) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-11-17 16:25:00 +05:00
Rui Xi 14737e91ed
Mitigate brotli and deflate decompression bombs DoS (#7134) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-11-17 14:21:50 +05:00
Adrián Chaves d091256c58 Remove the deprecated sphinx-hoverxref (#6922) 2025-11-14 15:39:21 +05:00
Andrey Rakhmatullin c83ca70db3 Don't force the unavailable sysmon coverage core. (#7137) 2025-11-13 23:26:08 +05:00
Andrey Rakhmatullin 85e4e6c42b Pin Twisted to <= 25.5.0 due to internal API changes. 2025-11-13 23:25:51 +05:00
Andrey Rakhmatullin 426aafddca
Don't force the unavailable sysmon coverage core. (#7137) 2025-11-13 23:22:52 +05:00