Commit Graph

9 Commits

Author SHA1 Message Date
Adrian 361f689df7
Improve test coverage for crawler.py (#7682)
* Improve test coverage for crawler.py

* Silence mypy warnings

* Improve test coverage for crawler.py
2026-07-01 11:53:07 +05:00
Andrey Rakhmatullin 4b40d2d06a
Close various garbage collectible resources (mostly in tests) (#7644)
* Set TELNETCONSOLE_ENABLED=False in get_crawler().

* Make _get_console_and_portal() a context manager.

* Allow PYTHONTRACEMALLOC in tox.

* Close the event loop in test_custom_asyncio_loop_enabled_false().

* Close the handler in _uninstall_scrapy_root_handler().

* Close queues in test_squeues.py.

* Close empty queues in init_prios().

* Close PopenSpawn stdin and stdout explicitly.

* Silence the unclosed socket warning.

* Implement more methods in CustomStatsCollector.

* Link the Twisted issue.

* Close the connection on download_maxsize.

* Fix typing.

* Add a comment about CustomStatsCollector.

* Restore the coverage.

* Properly close fixture queues.

* More robust file closing in feed storages.

* Close the file in TestMarshalItemExporter.test_nonstring_types_item().

* Cleanup temporary file handling in test_exporters.py.

* Close the file in test_stats_file_failed().

* Use an explicit TextIOWrapper in XmlItemExporter.
2026-06-26 23:05:08 +05:00
Andrey Rakhmatullin fed75a6c76
Refactor test_crawl.py and test_crawler.py. (#7566) 2026-06-03 14:48:12 +02:00
Andrey Rakhmatullin 2798c03bb0
Add Python 3.14 to CI. (#6604)
* Add Python 3.14 (alpha3) to CI.

* Disable mitmproxy on 3.14 for now.

* 3.14.0-alpha.4.

* 3.14.0-alpha.5

* 3.14.0-beta2.

* 3.14 release.

* Fix test_non_pickable_object.

* Fix handling of file:/path feed URIs.

* Better mocking of streams for TextTestResult.

* Do not use .php in test_file_path() as it's now a known extension.

* Fix the URL in TestFeedExporterSignals.

* Fix typing.

* Bump more envs to 3.14.

* Silence pylint.

* Fix another test for .php handling change.

* Remove test_install_asyncio_reactor.

* More bumps to 3.14.

* Revert docs-tests to use 3.13.

* Debug options for Windows.

* Re-enable xdist.

* Revert Windows PYTEST_ADDOPTS.

* Silence loop policy deprecation warnings.

* Restore a lost pylint suppression.

* Update asyncio_enabled_reactor_same_loop.py to new Twisted.

* Fix RobotFileParser tests for Python 3.14.5.
2026-05-12 23:49:37 +05:00
Andrey Rakhmatullin f7db039d1c
Update the TLS code to Twisted 26.4.0 (#7347)
* Unpin Twisted, add a twisted-trunk tox env (without http2 for now).

* Hide the _setAcceptableProtocols import.

* Silence ScrapyClientContextFactory HTTP/1.0 warnings.

* Update ScrapyClientTLSOptions for unreleased Twisted.

* Add twisted-trunk to CI.

* Update for new changes in Twisted trunk.

* Xfail a test failing with newer Twisted.

* Test Twisted trunk with extra-deps, update the ALPN code.

* Cleanup.

* Fix typing.

* Update relevant type hints for Twisted trunk.

* Update test_no_context_sharing().

* Silence a weird mypy error.

* Update Twisted versions, unpin pyOpenSSL.

* Update a comment.

* Silence pylint.

* Make a factory fixture.

* Improve the _setAcceptableProtocols comment.

* Set OP_LEGACY_SERVER_CONNECT on new Twisted too.

* Add Twisted[http2] to extra-deps-pinned.
2026-05-12 18:22:31 +05:00
Andrey Rakhmatullin b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin 0c6ccf50b3
Docs for the reactorless mode (#7385)
* Reactorless mode docs.

* Unset the ftp handler in the reactorless mode.

* Cleanup reactorless subprocess tests.

* Better handling of incompatible TWISTED_ENABLED cases.

* Improvements.

* Improve early error logging in test scripts.

* Address feedback.

* Address feedback.
2026-04-01 20:40:34 +05:00
Andrey Rakhmatullin fee20b7858
AsyncCrawlerProcess._start_asyncio() improvements. (#7366) 2026-03-27 12:21:25 +01:00
Andrey Rakhmatullin 584d99af30
Split long test files (#7329) 2026-03-13 10:30:25 +01:00