RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,108 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

369 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 2d007bc450
Make more of the internal handler helpers private. (#7510) 2026-05-12 22:03:14 +02:00
Andrey Rakhmatullin 7fc84d372a
Cleanup/clarifications of proxy support and improvements for proxy tests (#7496) 
* Run mitmproxy-based tests for every handler and improve them.

* Fixes for H2DownloadHandler.

* Revert "Fixes for H2DownloadHandler."

This reverts commit bcdbd097cd.

* Add TestHttpsProxy for HTTP11DownloadHandler.

* Use the configured context factory in ScrapyProxyAgent.

* Reword.

* Reword.

* pragma: no cover for H2DownloadHandler proxy code.

* Raise an exception for HTTPS proxies instead of using HTTP.

* Remove non-working proxy support and explicitly forbid HTTP.

* pragma: no cover

* Add a docs note about HTTPS proxies.

* Rename ScrapyProxyAgent.
 2026-05-12 17:45:54 +05:00
Adrian Chaves 3d5ca9f433 Bump version: 2.15.1 → 2.15.2 2026-04-28 15:28:51 +02:00
Adrian 3a88cd0e2b
2.15.2 (#7468) 
* sphinx-scrapy: 0.8.3 → 0.8.4 (#7467)

* Release notes for 2.15.2
 2026-04-28 15:26:42 +02:00
Andrey Rakhmatullin f3c5a6e75f Bump version: 2.15.0 → 2.15.1 2026-04-23 19:57:44 +05:00
Andrey Rakhmatullin 416a454dc5
Release notes for 2.15.1. (#7456) 2026-04-23 19:51:00 +05:00
Andrey Rakhmatullin 47e25fbbb5 Bump version: 2.14.2 → 2.15.0 2026-04-09 17:00:40 +05:00
Andrey Rakhmatullin 1432455d35 Add a missing release notes change. 2026-04-09 16:59:18 +05:00
Andrey Rakhmatullin 7e881ce2d7
Release notes for 2.15.0 (#7373) 
* Cover 2.15.0 in the release notes (up to 31bf7c3).

* Replace VERSION.

* Document unified download handler exceptions.

* Updates.

* Cover 2.15.0 in the release notes (up to 0c6ccf5).

* Cover 2.15.0 in the release notes (up to b68f267).

* Run linkcheck and fix issues.

* Cleanup unused link targets.

* Cleanup.
 2026-04-09 16:56:30 +05:00
Albert Eduardovich N. 13a014d2e6
Response now uses less memory (#7374) 2026-04-08 14:25:11 +02:00
Kailesh a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER (#7361) 2026-04-08 13:58:35 +02:00
Andrey Rakhmatullin 0c6ccf50b3
Docs for the reactorless mode (#7385) 
* Reactorless mode docs.

* Unset the ftp handler in the reactorless mode.

* Cleanup reactorless subprocess tests.

* Better handling of incompatible TWISTED_ENABLED cases.

* Improvements.

* Improve early error logging in test scripts.

* Address feedback.

* Address feedback.
 2026-04-01 20:40:34 +05:00
Andrey Rakhmatullin eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES (#7379) 
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.

* Deprecate BrowserLikeContextFactory.

* Add DOWNLOAD_VERIFY_CERTIFICATES.

* Cleanup, add acceptableCiphers to the verifying path.
 2026-04-01 15:18:43 +05:00
Andrey Rakhmatullin d42b23d78a
Await persist_file() in media pipelines. (#7182) 2026-03-24 10:52:56 +01:00
Adrian Chaves 4e3df249f2 Merge branch '2.14' 2026-03-12 16:02:37 +01:00
Adrian Chaves 498b4fc1a4 Bump version: 2.14.1 → 2.14.2 2026-03-12 15:45:16 +01:00
Adrian Chaves 378bb68039 Proofread the release notes 2026-03-12 15:44:54 +01:00
Adrian Chaves 945b787a26 Merge remote-tracking branch 'cwxj-rr6w-m6w7/fix-referer-policy-handling' into 2.14 2026-03-12 13:37:24 +01:00
Andrey Rakhmatullin fc30c47f38
Add HttpxDownloader. (#7239) 2026-02-12 09:30:44 +01:00
Adrian Chaves 8974580e43 Reword the release note entry to consider the 301 redirect fix a security bug fix 2026-02-03 20:38:11 +01:00
Adrian Chaves ba3d7bc7a8 Remove the non-standard 307/308 handling, and align other aspects with the standard 2026-01-21 18:46:05 +01:00
Adrián Chaves 1b9c8b55da
not → now 2026-01-20 12:13:44 +01:00
Adrian Chaves 2651d48f20 Cover the 301 behavior change in the release notes 2026-01-19 19:25:00 +01:00
Adrian Chaves 99d5d58e20 Add a release notes entry 2026-01-19 19:19:02 +01:00
Adrian Chaves b6e5c58ae7 Fix arbitrary callable execution from the Referrer-Policy response header 2026-01-19 16:44:03 +01:00
Eugenio Lacuesta 0b9d8da09d
Bump version: 2.14.0 → 2.14.1 2026-01-12 16:25:43 -03:00
Eugenio Lacuesta c9fbf6c599
Release notes for 2.14.1 (#7217) 
* Release notes for 2.14.1

* Update docs/news.rst

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* Update docs/news.rst

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>

* Update docs/news.rst

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* Update message about github action change

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-01-12 16:23:51 -03:00
Andrey Rakhmatullin 3ac1192f35 Bump version: 2.13.4 → 2.14.0 2026-01-05 15:06:13 +05:00
Andrey Rakhmatullin 7bef98b4f1
Release notes for 2.14.0 (#7035) 
* Initial release notes for 2.14.0, up to ed63fa9.

* Cover 2.14 in the release notes up to 393d715.

* Cover 2.14 in the release notes up to eb49647.

* Cover 2.14 in the release notes up to 426aafd.

* Cover 2.14 in the release notes up to 1e8de24.

* Cover 2.14 in the release notes up to 5a7e132.

* Bump sphinx-lint.

* Finalize the 2.14.0 release notes.

* Drop more of the old versionadded directives.

* Address feedback.
 2026-01-05 14:51:05 +05:00
Andrey Rakhmatullin d1bd8eb49f
Async API for download handlers. (#7164) 2025-12-31 19:03:16 +01:00
Andrey Rakhmatullin c49ae2115a
Refactor MediaPipeline.process_item() to async def (#7177) 2025-12-15 10:02:04 +01:00
Andrey Rakhmatullin 4cb0144b39 Merge tag '2.13.4' 
Bump version: 2.13.3 → 2.13.4
 2025-11-17 16:47:54 +05:00
Andrey Rakhmatullin 2f62ab532d Bump version: 2.13.3 → 2.13.4 2025-11-17 16:30:37 +05:00
Andrey Rakhmatullin 31a9c03c24
Release notes for 2.13.4. (#7144) 
* Release notes for 2.13.4.

* Formatting.
 2025-11-17 16:30:04 +05:00
Andrey Rakhmatullin 1c6ba00fd0
Move LogCounterHandler into a new extension (#7046) 
* Move LogCounterHandler into Crawler.crawl().

* Add the LogCount extension.

* Update test_spider_custom_settings_log_level().

* Typo.

* Update docs.
 2025-10-03 20:14:11 +05:00
Andrey Rakhmatullin ed63fa94d6
Deprecate spider args of middlewares and pipelines. (#7006) 
* Deprecate spider args of spider middleware methods.

* Deprecate the spider arg of pipeline process_item().

* Fix a typing issue.

* Deprecate the spider arg of pipeline {open,close}_spider().

* Cleanup.

* Update docs.

* Add pragma: no cover to some deprecated code.

* More tests.

* More tests.

* More tests.

* Simplify _process_parallel().
 2025-08-21 13:23:10 +05:00
Andrey Rakhmatullin edba1ad572
Refactor RobotsTxtMiddleware to use async def process_request(). (#6802) 2025-08-18 18:40:46 +02:00
Andrey Rakhmatullin d9ef0350d8 Merge remote-tracking branch 'origin/2.13' 2025-07-02 20:46:02 +05:00
Andrey Rakhmatullin 155a504f24 Bump version: 2.13.2 → 2.13.3 2025-07-02 20:39:46 +05:00
Andrey Rakhmatullin cf465bf644
Release notes for 2.13.3. (#6934) 
* Release notes for 2.13.3.

* Address feedback.
 2025-07-02 20:36:01 +05:00
Andrey Rakhmatullin 3843091c5f
Mark abstract base classes as such. (#6930) 
* Mark abstract base classes as such.

* Remove an unneeded method.

* Fix exporter test coverage.
 2025-06-30 15:14:34 +05:00
Andrey Rakhmatullin 03fe7a6424 Add a deprecation notice for the offsite spider middleware. 2025-06-29 02:35:48 +05:00
Andrey Rakhmatullin 2c1c10e923
Remove deprecated offsite spider middleware. (#6926) 
* Remove deprecated offsite spider middleware.

* Add a deprecation notice for the offsite spider middleware.
 2025-06-29 02:35:20 +05:00
Andrey Rakhmatullin 95172659af
Add sphinx-lint. (#6920) 2025-06-28 01:37:20 +02:00
Andrey Rakhmatullin daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts (#6888) 
* Remove the chdir fixture.

* Re-enable fancy pytest asserts.

* Remove doc files from pytest ignores.

* Restore docs/_ext in test collection ignores.

* Skip a doctest that fails on Windows.

* Fix tests that were writing to the current dir.
 2025-06-14 14:01:20 +05:00
Andrey Rakhmatullin 0390176ecd Merge branch '2.13' 2025-06-09 15:54:35 +05:00
Andrey Rakhmatullin c6740604a4 Bump version: 2.13.1 → 2.13.2 2025-06-09 15:21:06 +05:00
Andrey Rakhmatullin 7400868ad5
Release notes for 2.13.2. (#6868) 
* Release notes for 2.13.2.

* Update release notes.
 2025-06-09 15:19:02 +05:00
Andrey Rakhmatullin 405d9bc8a2
More docs for the is_asyncio_reactor_installed() behavior change. (#6866) 2025-06-06 12:59:49 +02:00
Andrey Rakhmatullin 3aa5e75787
Use AsyncCrawlerProcess in commands. (#6845) 
* Use AsyncCrawlerProcess in commands.

* Ignore coverage of abstract methods.

* Address feedback.
 2025-06-03 17:47:52 +05:00