RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,067 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

2911 Commits

Author SHA1 Message Date
Adrian Chaves b90842379d Extend download error coverage 2026-04-22 17:04:32 +02:00
NI283105 508367664f
Fix missing load_object import and grammar in docs (#7417) 
Co-authored-by: NI <Nidotexe@gmail.com>
 2026-04-09 15:31:01 +02:00
Andrey Rakhmatullin 47e25fbbb5 Bump version: 2.14.2 → 2.15.0 2026-04-09 17:00:40 +05:00
Andrey Rakhmatullin 1432455d35 Add a missing release notes change. 2026-04-09 16:59:18 +05:00
Andrey Rakhmatullin 7e881ce2d7
Release notes for 2.15.0 (#7373) 
* Cover 2.15.0 in the release notes (up to 31bf7c3).

* Replace VERSION.

* Document unified download handler exceptions.

* Updates.

* Cover 2.15.0 in the release notes (up to 0c6ccf5).

* Cover 2.15.0 in the release notes (up to b68f267).

* Run linkcheck and fix issues.

* Cleanup unused link targets.

* Cleanup.
 2026-04-09 16:56:30 +05:00
Andrey Rakhmatullin b9be5ce053
Reactorless shell and other small shell improvements (#7395) 
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.

* defer.Deferred -> Deferred.

* Allow Shell with TWISTED_REACTOR_ENABLED=False.

* Shell workflow notes.

* Remove an unused argument.

* Shell.fetch_available.

* Shell._use_reactor.

* Some more comments.

* Simplify _schedule() and fix the spider shell var.

* More async def.

* Reactorless shell support.

* Add pragma: no cover.

* More notes.

* Remove a TODO.
 2026-04-08 18:24:51 +05:00
Albert Eduardovich N. 13a014d2e6
Response now uses less memory (#7374) 2026-04-08 14:25:11 +02:00
Kailesh a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER (#7361) 2026-04-08 13:58:35 +02:00
Ahmed Arshad 830eaeab5d
docs: fix typos in addons.rst (#7408) 
Co-authored-by: Abdul Rahman <abdouls@hotmail.com>
 2026-04-08 09:38:09 +02:00
Adrian 010faf1722
Use sphinx-scrapy 0.7.1 (#7406) 2026-04-06 15:29:59 +02:00
Adrian f8d103a65a
Add llms.txt and llms-full.txt generation (#7380) 2026-04-06 10:24:21 +02:00
Andrey Rakhmatullin b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin 0c6ccf50b3
Docs for the reactorless mode (#7385) 
* Reactorless mode docs.

* Unset the ftp handler in the reactorless mode.

* Cleanup reactorless subprocess tests.

* Better handling of incompatible TWISTED_ENABLED cases.

* Improvements.

* Improve early error logging in test scripts.

* Address feedback.

* Address feedback.
 2026-04-01 20:40:34 +05:00
Andrey Rakhmatullin fa76ca52e9
Move TLS logging from _ScrapyClientTLSOptions to handlers. (#7387) 
* Move TLS logging from _ScrapyClientTLSOptions to handlers.

* Update the docs.

* Fix typing.

* Update new docs.

* Add the comment back.
 2026-04-01 16:02:55 +05:00
Andrey Rakhmatullin eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES (#7379) 
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.

* Deprecate BrowserLikeContextFactory.

* Add DOWNLOAD_VERIFY_CERTIFICATES.

* Cleanup, add acceptableCiphers to the verifying path.
 2026-04-01 15:18:43 +05:00
Adrian Chaves c4c0555ccf Enable parallel execution in the other internal Sphinx plugin 2026-03-30 18:01:29 +02:00
Adrian Chaves 299993b62a Fix the docs build 2026-03-30 17:46:55 +02:00
Andrey Rakhmatullin 31bf7c3892
Docs for HttpxDownloadHandler, handle httpx not being installed (#7368) 
* Add docs for HttpxDownloadHandler.

* Handle httpx not being installed.

* Revert test_not_configured_without_asyncio().

* Restore the comment.

* Cleanup.
 2026-03-27 21:06:55 +05:00
Kaileshwar R K 54a4c3af89
Deprecate the mail API (#7263) 2026-03-24 21:31:12 +01:00
Andrey Rakhmatullin d42b23d78a
Await persist_file() in media pipelines. (#7182) 2026-03-24 10:52:56 +01:00
Julian Hamze c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers (#7283) 2026-03-19 09:16:40 +01:00
Adrian Chaves 4e3df249f2 Merge branch '2.14' 2026-03-12 16:02:37 +01:00
Adrian Chaves 498b4fc1a4 Bump version: 2.14.1 → 2.14.2 2026-03-12 15:45:16 +01:00
Adrian Chaves 378bb68039 Proofread the release notes 2026-03-12 15:44:54 +01:00
Adrian Chaves 945b787a26 Merge remote-tracking branch 'cwxj-rr6w-m6w7/fix-referer-policy-handling' into 2.14 2026-03-12 13:37:24 +01:00
Yuval Elbar da1a6b7ebc
Add security warning about pickle files in JOBDIR (#7274) 
* Add security warning about pickle files in JOBDIR

The job directory contains files serialized with pickle (spider.state,
request queues), which can execute arbitrary code when loaded. Add a
warning so users know to treat JOBDIR with the same trust level as
their project code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Make JOBDIR security warning more concise and accurate

Simplify the warning to focus on treating the job directory
as trusted, without mentioning specific serialization details.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-20 19:36:26 +05:00
Anant Murmu 2c3ecbff71
fix: missing self argument in ScreenshotPipeline constructor (#7248) 
added missing self argument in __init__ method constructor
 2026-02-13 09:42:26 +01:00
Andrey Rakhmatullin fc30c47f38
Add HttpxDownloader. (#7239) 2026-02-12 09:30:44 +01:00
Martin Schimandl 66fe5de139
Add Jobdir documentation (#5260) 
* Add Jobdir documentation

* Fix link format

* remove doubtful comments

* some more edits

* Keep the information in jobs.rst an example, and provide details in the reference docs of the corresponding components

* Minor edits

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
 2026-02-06 22:33:02 +04:00
Michael 6a42bc6450
Item docs: mention Pydantic support (#6966) 
* Add Pydantic support documentation

* Fix Sphinx references in Pydantic section for non-Sphinx docs

* Minor reformatting

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
 2026-02-05 22:18:01 +04:00
Adrian Chaves 8974580e43 Reword the release note entry to consider the 301 redirect fix a security bug fix 2026-02-03 20:38:11 +01:00
Andrey Rakhmatullin 2347138ba4
Unified download handler exceptions (#7208) 
* Add DownloadTimeoutError.

* Don't expect ConnectionAborted in tests.

* Add DownloadCancelledError.

* Add ResponseDataLoss.

* Add more download handler tests.

* Add DownloadConnectionRefusedError.

* Add UnsupportedURLScheme.

* Add CannotResolveHostError.

* Add DownloadFailedError.

* Remove wrapped Twisted exceptions from lists.

* Update a test expectation.

* Extract wrap_twisted_exceptions().

* Wrap TxTimeoutError.

* Rename ResponseDataLoss and UnsupportedURLScheme.
 2026-01-21 23:09:54 +05:00
Adrian Chaves ba3d7bc7a8 Remove the non-standard 307/308 handling, and align other aspects with the standard 2026-01-21 18:46:05 +01:00
Adrián Chaves a39545195e
allow to override → allow overriding 2026-01-20 12:20:37 +01:00
Adrián Chaves 1b9c8b55da
not → now 2026-01-20 12:13:44 +01:00
Adrian Chaves 2651d48f20 Cover the 301 behavior change in the release notes 2026-01-19 19:25:00 +01:00
Adrian Chaves 99d5d58e20 Add a release notes entry 2026-01-19 19:19:02 +01:00
Adrian Chaves 6f03f3250b Add versionadded 2026-01-19 16:46:53 +01:00
Adrian Chaves b6e5c58ae7 Fix arbitrary callable execution from the Referrer-Policy response header 2026-01-19 16:44:03 +01:00
Eugenio Lacuesta 0b9d8da09d
Bump version: 2.14.0 → 2.14.1 2026-01-12 16:25:43 -03:00
Eugenio Lacuesta c9fbf6c599
Release notes for 2.14.1 (#7217) 
* Release notes for 2.14.1

* Update docs/news.rst

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* Update docs/news.rst

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>

* Update docs/news.rst

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* Update message about github action change

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-01-12 16:23:51 -03:00
Andrey Rakhmatullin 3ac1192f35 Bump version: 2.13.4 → 2.14.0 2026-01-05 15:06:13 +05:00
Andrey Rakhmatullin 7bef98b4f1
Release notes for 2.14.0 (#7035) 
* Initial release notes for 2.14.0, up to ed63fa9.

* Cover 2.14 in the release notes up to 393d715.

* Cover 2.14 in the release notes up to eb49647.

* Cover 2.14 in the release notes up to 426aafd.

* Cover 2.14 in the release notes up to 1e8de24.

* Cover 2.14 in the release notes up to 5a7e132.

* Bump sphinx-lint.

* Finalize the 2.14.0 release notes.

* Drop more of the old versionadded directives.

* Address feedback.
 2026-01-05 14:51:05 +05:00
Andrey Rakhmatullin d1bd8eb49f
Async API for download handlers. (#7164) 2025-12-31 19:03:16 +01:00
Andrey Rakhmatullin a2463325db
Allow Crawler.crawl_async() without asyncio. (#7202) 2025-12-31 18:33:59 +01:00
Adham 3llam 9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation (#7198) 2025-12-22 10:00:39 +01:00
Andrey Rakhmatullin 180ca39b23
Deprecate returning Deferreds from pipeline methods (#7179) 
* Add tests for exceptions in pipelines.

* Deprecate returning Deferreds from pipeline process_item().

* Deprecate returning Deferreds from pipeline {open,close}_spider().

* Update the custom pipeline docs.
 2025-12-16 13:35:07 +05:00
Andrey Rakhmatullin c49ae2115a
Refactor MediaPipeline.process_item() to async def (#7177) 2025-12-15 10:02:04 +01:00
Andrey Rakhmatullin 588f3d4f65
Remove docs mentioning deprecated spider attrs. (#7176) 2025-12-15 09:51:56 +01:00
Thalison Fernandes d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue (#6940) 
* Change default priority queue to DownloaderAwarePriorityQueue

* Fix documentation building

* Simplify test_start_already_running_exception changes.

* Modernize the test.

* Fix TestEngineCloseSpider.

* Fix typing.

* Remove special slot=None handling.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-12-11 11:25:17 +01:00