RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,204 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

11204 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 7fc84d372a
Cleanup/clarifications of proxy support and improvements for proxy tests (#7496) 
* Run mitmproxy-based tests for every handler and improve them.

* Fixes for H2DownloadHandler.

* Revert "Fixes for H2DownloadHandler."

This reverts commit bcdbd097cd.

* Add TestHttpsProxy for HTTP11DownloadHandler.

* Use the configured context factory in ScrapyProxyAgent.

* Reword.

* Reword.

* pragma: no cover for H2DownloadHandler proxy code.

* Raise an exception for HTTPS proxies instead of using HTTP.

* Remove non-working proxy support and explicitly forbid HTTP.

* pragma: no cover

* Add a docs note about HTTPS proxies.

* Rename ScrapyProxyAgent.
 2026-05-12 17:45:54 +05:00
Adrian 7f15ca92fc
sphinx-scrapy: 0.8.5 → 0.8.6 (#7507) 2026-05-12 12:49:23 +05:00
Andrey Rakhmatullin 5223dbe3fd
Pin pyOpenSSL to the versions allowing mutable contexts. (#7494) 2026-05-06 10:42:24 +02:00
Andrey Rakhmatullin fc14a0ce59
Enable mypy warn_return_any. (#7492) 2026-05-04 22:24:47 +02:00
Andrey Rakhmatullin 33452f3aeb
Silence deprecation warnings in TestFormRequest. (#7491) 2026-05-04 22:20:21 +02:00
Albert Eduardovich N. 9776a72a6a
chore: simplify some code, get rid of nested fns where it's makes sence + pylint (#7401) 
* chore: simplify some code, get rid of nested fns where it's makes sence

reasoning: https://wemake-python-styleguide.readthedocs.io/en/latest/pages/usage/violations/best_practices.html#wemake_python_styleguide.violations.best_practices.NestedFunctionViolation

* chore: address some pylint rules

* remove `pylint: disable=deprecated-class` from `reactor.py`

* fix typo

* fix another typo :)

* revert backward incompatible arg renames

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-05-04 23:32:46 +05:00
Adrián Chaves 8d69a7c865
Deprecate FormRequest in favor of form2request (#6438) 
* WIP

* Add docs

* Remove FormRequest from tests

* Silence mypy issue

* Address docs-tests issues

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-05-04 22:38:29 +05:00
Andrey Rakhmatullin f3868e11fb
Drop start_requests() support. (#7490) 2026-05-04 17:34:24 +05:00
Andrey Rakhmatullin 9ca206da64
Merge pull request #7487 from wRAR/deprecation-removals 
Remove most of the 2.13.0 deprecations
 2026-05-04 12:33:21 +05:00
Andrey Rakhmatullin d05b241f64
Fix a code block in leaks.rst. (#7489) 2026-05-04 08:59:27 +02:00
Adrian 55c61646da
Merge branch 'master' into deprecation-removals 2026-05-04 08:52:12 +02:00
Andrey Rakhmatullin c62a81d7dd
Drop HTTP/1.0 code. (#7486) 2026-05-04 08:50:41 +02:00
Sanjay M a9324fbf76
Fix small documentation wording issues (#7480) 2026-05-04 08:43:21 +02:00
Andrey Rakhmatullin 9f02f6c16a Drop spider args of Scraper methods. 2026-05-03 14:12:18 +05:00
Andrey Rakhmatullin 6cb2fe1fc3 Drop scrapy_components_versions(). 2026-05-03 14:12:18 +05:00
Andrey Rakhmatullin f24bc749ea Require start_queue_cls. 2026-05-03 14:12:18 +05:00
Andrey Rakhmatullin 5fc40f07f3 Drop w3lib.url reexports. 2026-05-03 14:12:18 +05:00
Andrey Rakhmatullin af7dcabebb Drop AjaxCrawlMiddleware and escape_ajax(). 2026-05-03 14:12:18 +05:00
Andrey Rakhmatullin 8ecfd20fcd Remove InitSpider. 2026-05-03 14:12:16 +05:00
Andrey Rakhmatullin 14f49ab63c Remove deprecated test utils. 2026-05-03 14:11:55 +05:00
Adrian b9c2240040
sphinx-scrapy: 0.8.4 → 0.8.5 (#7472) 2026-04-29 14:17:26 +02:00
Albert Eduardovich N. dd36fb7859
Improve handling of time deltas (#7377) 2026-04-29 09:53:40 +02:00
Andrey Rakhmatullin 30a54b72f0
Merge pull request #7470 from scrapy/2.15 
Merge 2.15.2 into the main branch
 2026-04-28 21:19:29 +05:00
Adrian Chaves 3d5ca9f433 Bump version: 2.15.1 → 2.15.2 2026-04-28 15:28:51 +02:00
Adrian 3a88cd0e2b
2.15.2 (#7468) 
* sphinx-scrapy: 0.8.3 → 0.8.4 (#7467)

* Release notes for 2.15.2
 2026-04-28 15:26:42 +02:00
Andrey Rakhmatullin 988afe1454
Clarify dataloss/http2 tests. (#7466) 2026-04-28 14:35:41 +02:00
Adrian 528745b059
sphinx-scrapy: 0.8.3 → 0.8.4 (#7467) 2026-04-28 12:17:12 +02:00
Andrey Rakhmatullin 068aa69b35
Fixes for Python 3.14. (#7460) 2026-04-27 18:47:52 +05:00
Andrey Rakhmatullin fc4c57e795
Fix the substitution pattern in open_in_browser(). (#7459) 
* Fix the substitution pattern in open_in_browser().

* Formatting.
 2026-04-24 13:54:37 +05:00
Andrey Rakhmatullin 4e25686b20 Merge branch '2.15' 2026-04-23 21:08:38 +05:00
Andrey Rakhmatullin f3c5a6e75f Bump version: 2.15.0 → 2.15.1 2026-04-23 19:57:44 +05:00
Andrey Rakhmatullin 416a454dc5
Release notes for 2.15.1. (#7456) 2026-04-23 19:51:00 +05:00
Adrian 3561748280 sphinx-scrapy: 0.7.1 → 0.8.3 (#7454) 2026-04-23 13:20:30 +05:00
Adrian 41f43f4649
sphinx-scrapy: 0.7.1 → 0.8.3 (#7454) 2026-04-23 10:03:09 +02:00
Andrey Rakhmatullin b7cd42da39 Revert sharing of the SSL context in _ScrapyClientContextFactory. (#7450) 2026-04-22 19:02:01 +05:00
Adrian da6dfae750 Restore 2.14 getwithbase, add a new method for class key deduplication (#7449) 
* Restore 2.14 getwithbase, add a new method for class key deduplication

* Use the original getwithbase code, not some equivalent

* Fix mocking

* Test key exceptions
 2026-04-22 19:01:59 +05:00
Andrey Rakhmatullin 320e40a044
Revert sharing of the SSL context in _ScrapyClientContextFactory. (#7450) 2026-04-22 19:01:01 +05:00
Adrian 294abed138
Restore 2.14 getwithbase, add a new method for class key deduplication (#7449) 
* Restore 2.14 getwithbase, add a new method for class key deduplication

* Use the original getwithbase code, not some equivalent

* Fix mocking

* Test key exceptions
 2026-04-22 19:00:39 +05:00
Andrey Rakhmatullin 27092b2cb7
Small download handler fixes and improvements (#7441) 
* Popen cleanup.

* Improve the test for response headers.

* Fix default headers of Httpx and H2 handlers, add tests.

* Check Location in redirect tests.

* Simplify bindaddress tests.

* Add download_latency to HttpxDownloadHandler, add tests.

* Remove explicit method="GET".

* Don't expect a specific value in test_download_with_maxsize_very_large_file

* Use a non-existent scheme in test_unsupported_scheme.

* Remporarily rollback the test_download_latency test check.

* Better check in test_download_latency.

* Improve logic for StreamCloseReason.CANCELLED.
 2026-04-21 18:27:16 +05:00
Hamzah Alshawwaf 9da14cdff1
Fix file extension extraction for URLs with query parameters (#4225) (#7414) 
* Fix file extension extraction for URLs with query parameters (#4225)

* Prioritize parsed path for file extensions and add tests (#4225)
 2026-04-14 20:04:38 +05:00
NI283105 508367664f
Fix missing load_object import and grammar in docs (#7417) 
Co-authored-by: NI <Nidotexe@gmail.com>
 2026-04-09 15:31:01 +02:00
Andrey Rakhmatullin 47e25fbbb5 Bump version: 2.14.2 → 2.15.0 2026-04-09 17:00:40 +05:00
Andrey Rakhmatullin 1432455d35 Add a missing release notes change. 2026-04-09 16:59:18 +05:00
Andrey Rakhmatullin 7e881ce2d7
Release notes for 2.15.0 (#7373) 
* Cover 2.15.0 in the release notes (up to 31bf7c3).

* Replace VERSION.

* Document unified download handler exceptions.

* Updates.

* Cover 2.15.0 in the release notes (up to 0c6ccf5).

* Cover 2.15.0 in the release notes (up to b68f267).

* Run linkcheck and fix issues.

* Cleanup unused link targets.

* Cleanup.
 2026-04-09 16:56:30 +05:00
Adrian b68f26726a
Fix DownloaderAwarePriorityQueue tie-breaking across slots (#7351) 2026-04-08 16:10:21 +02:00
Andrey Rakhmatullin 2b174e348d
Silence the CertificateOptions method warning. (#7410) 
* Silence the CertificateOptions method warning.

* Fix typing.
 2026-04-08 18:54:13 +05:00
Andrey Rakhmatullin b9be5ce053
Reactorless shell and other small shell improvements (#7395) 
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.

* defer.Deferred -> Deferred.

* Allow Shell with TWISTED_REACTOR_ENABLED=False.

* Shell workflow notes.

* Remove an unused argument.

* Shell.fetch_available.

* Shell._use_reactor.

* Some more comments.

* Simplify _schedule() and fix the spider shell var.

* More async def.

* Reactorless shell support.

* Add pragma: no cover.

* More notes.

* Remove a TODO.
 2026-04-08 18:24:51 +05:00
Albert Eduardovich N. 5b37613618
Improve commands (#7376) 2026-04-08 14:59:00 +02:00
Albert Eduardovich N. 13a014d2e6
Response now uses less memory (#7374) 2026-04-08 14:25:11 +02:00
Albert Eduardovich N. 9fffcc1b82
Optimize SitemapSpider memory usage (#7007) 2026-04-08 13:59:16 +02:00