RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,101 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

2926 Commits

Author SHA1 Message Date
Adrián Chaves 8d69a7c865
Deprecate FormRequest in favor of form2request (#6438) 
* WIP

* Add docs

* Remove FormRequest from tests

* Silence mypy issue

* Address docs-tests issues

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-05-04 22:38:29 +05:00
Andrey Rakhmatullin 9ca206da64
Merge pull request #7487 from wRAR/deprecation-removals 
Remove most of the 2.13.0 deprecations
 2026-05-04 12:33:21 +05:00
Andrey Rakhmatullin d05b241f64
Fix a code block in leaks.rst. (#7489) 2026-05-04 08:59:27 +02:00
Adrian 55c61646da
Merge branch 'master' into deprecation-removals 2026-05-04 08:52:12 +02:00
Sanjay M a9324fbf76
Fix small documentation wording issues (#7480) 2026-05-04 08:43:21 +02:00
Andrey Rakhmatullin af7dcabebb Drop AjaxCrawlMiddleware and escape_ajax(). 2026-05-03 14:12:18 +05:00
Adrian b9c2240040
sphinx-scrapy: 0.8.4 → 0.8.5 (#7472) 2026-04-29 14:17:26 +02:00
Andrey Rakhmatullin 30a54b72f0
Merge pull request #7470 from scrapy/2.15 
Merge 2.15.2 into the main branch
 2026-04-28 21:19:29 +05:00
Adrian Chaves 3d5ca9f433 Bump version: 2.15.1 → 2.15.2 2026-04-28 15:28:51 +02:00
Adrian 3a88cd0e2b
2.15.2 (#7468) 
* sphinx-scrapy: 0.8.3 → 0.8.4 (#7467)

* Release notes for 2.15.2
 2026-04-28 15:26:42 +02:00
Adrian 528745b059
sphinx-scrapy: 0.8.3 → 0.8.4 (#7467) 2026-04-28 12:17:12 +02:00
Andrey Rakhmatullin 4e25686b20 Merge branch '2.15' 2026-04-23 21:08:38 +05:00
Andrey Rakhmatullin f3c5a6e75f Bump version: 2.15.0 → 2.15.1 2026-04-23 19:57:44 +05:00
Andrey Rakhmatullin 416a454dc5
Release notes for 2.15.1. (#7456) 2026-04-23 19:51:00 +05:00
Adrian 3561748280 sphinx-scrapy: 0.7.1 → 0.8.3 (#7454) 2026-04-23 13:20:30 +05:00
Adrian 41f43f4649
sphinx-scrapy: 0.7.1 → 0.8.3 (#7454) 2026-04-23 10:03:09 +02:00
NI283105 508367664f
Fix missing load_object import and grammar in docs (#7417) 
Co-authored-by: NI <Nidotexe@gmail.com>
 2026-04-09 15:31:01 +02:00
Andrey Rakhmatullin 47e25fbbb5 Bump version: 2.14.2 → 2.15.0 2026-04-09 17:00:40 +05:00
Andrey Rakhmatullin 1432455d35 Add a missing release notes change. 2026-04-09 16:59:18 +05:00
Andrey Rakhmatullin 7e881ce2d7
Release notes for 2.15.0 (#7373) 
* Cover 2.15.0 in the release notes (up to 31bf7c3).

* Replace VERSION.

* Document unified download handler exceptions.

* Updates.

* Cover 2.15.0 in the release notes (up to 0c6ccf5).

* Cover 2.15.0 in the release notes (up to b68f267).

* Run linkcheck and fix issues.

* Cleanup unused link targets.

* Cleanup.
 2026-04-09 16:56:30 +05:00
Andrey Rakhmatullin b9be5ce053
Reactorless shell and other small shell improvements (#7395) 
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.

* defer.Deferred -> Deferred.

* Allow Shell with TWISTED_REACTOR_ENABLED=False.

* Shell workflow notes.

* Remove an unused argument.

* Shell.fetch_available.

* Shell._use_reactor.

* Some more comments.

* Simplify _schedule() and fix the spider shell var.

* More async def.

* Reactorless shell support.

* Add pragma: no cover.

* More notes.

* Remove a TODO.
 2026-04-08 18:24:51 +05:00
Albert Eduardovich N. 13a014d2e6
Response now uses less memory (#7374) 2026-04-08 14:25:11 +02:00
Kailesh a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER (#7361) 2026-04-08 13:58:35 +02:00
Ahmed Arshad 830eaeab5d
docs: fix typos in addons.rst (#7408) 
Co-authored-by: Abdul Rahman <abdouls@hotmail.com>
 2026-04-08 09:38:09 +02:00
Adrian 010faf1722
Use sphinx-scrapy 0.7.1 (#7406) 2026-04-06 15:29:59 +02:00
Adrian f8d103a65a
Add llms.txt and llms-full.txt generation (#7380) 2026-04-06 10:24:21 +02:00
Andrey Rakhmatullin b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin 0c6ccf50b3
Docs for the reactorless mode (#7385) 
* Reactorless mode docs.

* Unset the ftp handler in the reactorless mode.

* Cleanup reactorless subprocess tests.

* Better handling of incompatible TWISTED_ENABLED cases.

* Improvements.

* Improve early error logging in test scripts.

* Address feedback.

* Address feedback.
 2026-04-01 20:40:34 +05:00
Andrey Rakhmatullin fa76ca52e9
Move TLS logging from _ScrapyClientTLSOptions to handlers. (#7387) 
* Move TLS logging from _ScrapyClientTLSOptions to handlers.

* Update the docs.

* Fix typing.

* Update new docs.

* Add the comment back.
 2026-04-01 16:02:55 +05:00
Andrey Rakhmatullin eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES (#7379) 
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.

* Deprecate BrowserLikeContextFactory.

* Add DOWNLOAD_VERIFY_CERTIFICATES.

* Cleanup, add acceptableCiphers to the verifying path.
 2026-04-01 15:18:43 +05:00
Adrian Chaves c4c0555ccf Enable parallel execution in the other internal Sphinx plugin 2026-03-30 18:01:29 +02:00
Adrian Chaves 299993b62a Fix the docs build 2026-03-30 17:46:55 +02:00
Andrey Rakhmatullin 31bf7c3892
Docs for HttpxDownloadHandler, handle httpx not being installed (#7368) 
* Add docs for HttpxDownloadHandler.

* Handle httpx not being installed.

* Revert test_not_configured_without_asyncio().

* Restore the comment.

* Cleanup.
 2026-03-27 21:06:55 +05:00
Kaileshwar R K 54a4c3af89
Deprecate the mail API (#7263) 2026-03-24 21:31:12 +01:00
Andrey Rakhmatullin d42b23d78a
Await persist_file() in media pipelines. (#7182) 2026-03-24 10:52:56 +01:00
Julian Hamze c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers (#7283) 2026-03-19 09:16:40 +01:00
Adrian Chaves 4e3df249f2 Merge branch '2.14' 2026-03-12 16:02:37 +01:00
Adrian Chaves 498b4fc1a4 Bump version: 2.14.1 → 2.14.2 2026-03-12 15:45:16 +01:00
Adrian Chaves 378bb68039 Proofread the release notes 2026-03-12 15:44:54 +01:00
Adrian Chaves 945b787a26 Merge remote-tracking branch 'cwxj-rr6w-m6w7/fix-referer-policy-handling' into 2.14 2026-03-12 13:37:24 +01:00
Yuval Elbar da1a6b7ebc
Add security warning about pickle files in JOBDIR (#7274) 
* Add security warning about pickle files in JOBDIR

The job directory contains files serialized with pickle (spider.state,
request queues), which can execute arbitrary code when loaded. Add a
warning so users know to treat JOBDIR with the same trust level as
their project code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Make JOBDIR security warning more concise and accurate

Simplify the warning to focus on treating the job directory
as trusted, without mentioning specific serialization details.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-20 19:36:26 +05:00
Anant Murmu 2c3ecbff71
fix: missing self argument in ScreenshotPipeline constructor (#7248) 
added missing self argument in __init__ method constructor
 2026-02-13 09:42:26 +01:00
Andrey Rakhmatullin fc30c47f38
Add HttpxDownloader. (#7239) 2026-02-12 09:30:44 +01:00
Martin Schimandl 66fe5de139
Add Jobdir documentation (#5260) 
* Add Jobdir documentation

* Fix link format

* remove doubtful comments

* some more edits

* Keep the information in jobs.rst an example, and provide details in the reference docs of the corresponding components

* Minor edits

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
 2026-02-06 22:33:02 +04:00
Michael 6a42bc6450
Item docs: mention Pydantic support (#6966) 
* Add Pydantic support documentation

* Fix Sphinx references in Pydantic section for non-Sphinx docs

* Minor reformatting

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
 2026-02-05 22:18:01 +04:00
Adrian Chaves 8974580e43 Reword the release note entry to consider the 301 redirect fix a security bug fix 2026-02-03 20:38:11 +01:00
Andrey Rakhmatullin 2347138ba4
Unified download handler exceptions (#7208) 
* Add DownloadTimeoutError.

* Don't expect ConnectionAborted in tests.

* Add DownloadCancelledError.

* Add ResponseDataLoss.

* Add more download handler tests.

* Add DownloadConnectionRefusedError.

* Add UnsupportedURLScheme.

* Add CannotResolveHostError.

* Add DownloadFailedError.

* Remove wrapped Twisted exceptions from lists.

* Update a test expectation.

* Extract wrap_twisted_exceptions().

* Wrap TxTimeoutError.

* Rename ResponseDataLoss and UnsupportedURLScheme.
 2026-01-21 23:09:54 +05:00
Adrian Chaves ba3d7bc7a8 Remove the non-standard 307/308 handling, and align other aspects with the standard 2026-01-21 18:46:05 +01:00
Adrián Chaves a39545195e
allow to override → allow overriding 2026-01-20 12:20:37 +01:00
Adrián Chaves 1b9c8b55da
not → now 2026-01-20 12:13:44 +01:00