Commit Graph

11126 Commits

Author SHA1 Message Date
Adrian Chaves 8896371192 Rewrite tests 2026-06-05 11:04:47 +02:00
Adrian Chaves 39d75d2275 Merge remote-tracking branch 'origin/master' into force-url 2026-06-05 10:39:14 +02:00
Ayush Singh 13c1c1faf8
Close GCS feed temp files after upload (#7546) 2026-06-04 20:11:50 +05:00
Ethan Kuo df2f3d708e
fix open_in_browser() logic issue plus add new tests (#7506) 2026-06-04 16:24:50 +02:00
Andrey Rakhmatullin fed75a6c76
Refactor test_crawl.py and test_crawler.py. (#7566) 2026-06-03 14:48:12 +02:00
Andrey Rakhmatullin 90deebe75e
Convert tests that fail with testfixtures 12.0.0 (#7545) 2026-06-03 09:16:24 +02:00
Fardin Alizadeh 44406806f8
fix typos (#7564) 2026-06-02 11:04:38 +02:00
SpiliosDmk 4a16550859
DOC -> Add missing documentation for CloseSpider & CoreStats (#7421)
* DOC -> Add missing documentation for CloseSpider & CoreStats

* Apply the suggestion from the PR review.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-20 13:27:24 +05:00
Andrey Rakhmatullin abe9c63841 Bump version: 2.15.2 → 2.16.0 2026-05-19 17:27:16 +05:00
Andrey Rakhmatullin a84b7850fc
Release notes for Scrapy 2.16.0. (#7536) 2026-05-19 17:26:21 +05:00
Andrey Rakhmatullin 55c17a8985
Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
Andrey Rakhmatullin f875af4a86
Add BaseStreamingDownloadHandler and migrate HttpxDownloadHandler to it. (#7524) 2026-05-18 15:52:13 +02:00
Andrey Rakhmatullin a8a8f20d9c
Remove support for sync process_spider_output(). (#7504)
* Remove support for sync process_spider_output().

* Slight check fix.

* Don't expect exceptions from calling process_spider_output().

* Remove dead code.

* Fix test_deprecated_mw_spider_arg() to run all methods.

* Fix typing.

* Typos.

* Remove references to the removed section, move the universal section to the spider middleware docs and write it from a different perspective

* Fix indentation issue

* Minor rewordings

* older versions → lower versions

* Update news.rst

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-05-15 16:36:05 +05:00
Andrey Rakhmatullin 85c616c5c7
Fix certificate issuer verification on new Twisted. (#7520) 2026-05-14 19:56:05 +02:00
Olivia Choi ae4a8e39e1
Document TLS method setting in avoiding bans guide (#7518)
* Document TLS method setting in avoiding bans guide

Mention DOWNLOADER_CLIENT_TLS_METHOD in the avoiding getting banned section.

* Update practices.rst

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-14 16:27:43 +05:00
Adrian 4cfe7a08cd
Add CITATION.cff (#7519) 2026-05-14 13:07:13 +02:00
Andrey Rakhmatullin 2d007bc450
Make more of the internal handler helpers private. (#7510) 2026-05-12 22:03:14 +02:00
Andrey Rakhmatullin 2798c03bb0
Add Python 3.14 to CI. (#6604)
* Add Python 3.14 (alpha3) to CI.

* Disable mitmproxy on 3.14 for now.

* 3.14.0-alpha.4.

* 3.14.0-alpha.5

* 3.14.0-beta2.

* 3.14 release.

* Fix test_non_pickable_object.

* Fix handling of file:/path feed URIs.

* Better mocking of streams for TextTestResult.

* Do not use .php in test_file_path() as it's now a known extension.

* Fix the URL in TestFeedExporterSignals.

* Fix typing.

* Bump more envs to 3.14.

* Silence pylint.

* Fix another test for .php handling change.

* Remove test_install_asyncio_reactor.

* More bumps to 3.14.

* Revert docs-tests to use 3.13.

* Debug options for Windows.

* Re-enable xdist.

* Revert Windows PYTEST_ADDOPTS.

* Silence loop policy deprecation warnings.

* Restore a lost pylint suppression.

* Update asyncio_enabled_reactor_same_loop.py to new Twisted.

* Fix RobotFileParser tests for Python 3.14.5.
2026-05-12 23:49:37 +05:00
Abhinav W 3b34ab88c0
docs: document daily log file rotation (#7501)
* docs: document daily log file rotation

* Skip the new snippet in doc tests

---------

Co-authored-by: Adrian <adrian@zyte.com>
2026-05-12 19:34:51 +05:00
Andrey Rakhmatullin f7db039d1c
Update the TLS code to Twisted 26.4.0 (#7347)
* Unpin Twisted, add a twisted-trunk tox env (without http2 for now).

* Hide the _setAcceptableProtocols import.

* Silence ScrapyClientContextFactory HTTP/1.0 warnings.

* Update ScrapyClientTLSOptions for unreleased Twisted.

* Add twisted-trunk to CI.

* Update for new changes in Twisted trunk.

* Xfail a test failing with newer Twisted.

* Test Twisted trunk with extra-deps, update the ALPN code.

* Cleanup.

* Fix typing.

* Update relevant type hints for Twisted trunk.

* Update test_no_context_sharing().

* Silence a weird mypy error.

* Update Twisted versions, unpin pyOpenSSL.

* Update a comment.

* Silence pylint.

* Make a factory fixture.

* Improve the _setAcceptableProtocols comment.

* Set OP_LEGACY_SERVER_CONNECT on new Twisted too.

* Add Twisted[http2] to extra-deps-pinned.
2026-05-12 18:22:31 +05:00
Andrey Rakhmatullin 7fc84d372a
Cleanup/clarifications of proxy support and improvements for proxy tests (#7496)
* Run mitmproxy-based tests for every handler and improve them.

* Fixes for H2DownloadHandler.

* Revert "Fixes for H2DownloadHandler."

This reverts commit bcdbd097cd.

* Add TestHttpsProxy for HTTP11DownloadHandler.

* Use the configured context factory in ScrapyProxyAgent.

* Reword.

* Reword.

* pragma: no cover for H2DownloadHandler proxy code.

* Raise an exception for HTTPS proxies instead of using HTTP.

* Remove non-working proxy support and explicitly forbid HTTP.

* pragma: no cover

* Add a docs note about HTTPS proxies.

* Rename ScrapyProxyAgent.
2026-05-12 17:45:54 +05:00
Adrian 7f15ca92fc
sphinx-scrapy: 0.8.5 → 0.8.6 (#7507) 2026-05-12 12:49:23 +05:00
Andrey Rakhmatullin 5223dbe3fd
Pin pyOpenSSL to the versions allowing mutable contexts. (#7494) 2026-05-06 10:42:24 +02:00
Andrey Rakhmatullin fc14a0ce59
Enable mypy warn_return_any. (#7492) 2026-05-04 22:24:47 +02:00
Andrey Rakhmatullin 33452f3aeb
Silence deprecation warnings in TestFormRequest. (#7491) 2026-05-04 22:20:21 +02:00
Albert Eduardovich N. 9776a72a6a
chore: simplify some code, get rid of nested fns where it's makes sence + pylint (#7401)
* chore: simplify some code, get rid of nested fns where it's makes sence

reasoning: https://wemake-python-styleguide.readthedocs.io/en/latest/pages/usage/violations/best_practices.html#wemake_python_styleguide.violations.best_practices.NestedFunctionViolation

* chore: address some pylint rules

* remove `pylint: disable=deprecated-class` from `reactor.py`

* fix typo

* fix another typo :)

* revert backward incompatible arg renames

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 23:32:46 +05:00
Adrián Chaves 8d69a7c865
Deprecate FormRequest in favor of form2request (#6438)
* WIP

* Add docs

* Remove FormRequest from tests

* Silence mypy issue

* Address docs-tests issues

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 22:38:29 +05:00
Andrey Rakhmatullin f3868e11fb
Drop start_requests() support. (#7490) 2026-05-04 17:34:24 +05:00
Andrey Rakhmatullin 9ca206da64
Merge pull request #7487 from wRAR/deprecation-removals
Remove most of the 2.13.0 deprecations
2026-05-04 12:33:21 +05:00
Andrey Rakhmatullin d05b241f64
Fix a code block in leaks.rst. (#7489) 2026-05-04 08:59:27 +02:00
Adrian 55c61646da
Merge branch 'master' into deprecation-removals 2026-05-04 08:52:12 +02:00
Andrey Rakhmatullin c62a81d7dd
Drop HTTP/1.0 code. (#7486) 2026-05-04 08:50:41 +02:00
Sanjay M a9324fbf76
Fix small documentation wording issues (#7480) 2026-05-04 08:43:21 +02:00
Andrey Rakhmatullin 9f02f6c16a Drop spider args of Scraper methods. 2026-05-03 14:12:18 +05:00
Andrey Rakhmatullin 6cb2fe1fc3 Drop scrapy_components_versions(). 2026-05-03 14:12:18 +05:00
Andrey Rakhmatullin f24bc749ea Require start_queue_cls. 2026-05-03 14:12:18 +05:00
Andrey Rakhmatullin 5fc40f07f3 Drop w3lib.url reexports. 2026-05-03 14:12:18 +05:00
Andrey Rakhmatullin af7dcabebb Drop AjaxCrawlMiddleware and escape_ajax(). 2026-05-03 14:12:18 +05:00
Andrey Rakhmatullin 8ecfd20fcd Remove InitSpider. 2026-05-03 14:12:16 +05:00
Andrey Rakhmatullin 14f49ab63c Remove deprecated test utils. 2026-05-03 14:11:55 +05:00
Adrian Chaves cc61611cb1 Upgrade w3lib in pypy3-pinned to support new tests 2026-04-30 16:26:34 +02:00
Adrian Chaves 4a93f26746 Add verbatim_url 2026-04-30 14:42:11 +02:00
Adrian b9c2240040
sphinx-scrapy: 0.8.4 → 0.8.5 (#7472) 2026-04-29 14:17:26 +02:00
Albert Eduardovich N. dd36fb7859
Improve handling of time deltas (#7377) 2026-04-29 09:53:40 +02:00
Andrey Rakhmatullin 30a54b72f0
Merge pull request #7470 from scrapy/2.15
Merge 2.15.2 into the main branch
2026-04-28 21:19:29 +05:00
Adrian Chaves 3d5ca9f433 Bump version: 2.15.1 → 2.15.2 2026-04-28 15:28:51 +02:00
Adrian 3a88cd0e2b
2.15.2 (#7468)
* sphinx-scrapy: 0.8.3 → 0.8.4 (#7467)

* Release notes for 2.15.2
2026-04-28 15:26:42 +02:00
Andrey Rakhmatullin 988afe1454
Clarify dataloss/http2 tests. (#7466) 2026-04-28 14:35:41 +02:00
Adrian 528745b059
sphinx-scrapy: 0.8.3 → 0.8.4 (#7467) 2026-04-28 12:17:12 +02:00
Andrey Rakhmatullin 068aa69b35
Fixes for Python 3.14. (#7460) 2026-04-27 18:47:52 +05:00