RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,204 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

102 Commits

Author SHA1 Message Date
Andrey Rakhmatullin c9446931a8
Use external mitmproxy. (#7720) 2026-07-09 17:02:05 +02:00
Adrian 2d0a898e2d
tox.ini: pinned, typing → min, mypy (#7595) 2026-06-11 12:10:48 +05:00
Andrey Rakhmatullin 2798c03bb0
Add Python 3.14 to CI. (#6604) 
* Add Python 3.14 (alpha3) to CI.

* Disable mitmproxy on 3.14 for now.

* 3.14.0-alpha.4.

* 3.14.0-alpha.5

* 3.14.0-beta2.

* 3.14 release.

* Fix test_non_pickable_object.

* Fix handling of file:/path feed URIs.

* Better mocking of streams for TextTestResult.

* Do not use .php in test_file_path() as it's now a known extension.

* Fix the URL in TestFeedExporterSignals.

* Fix typing.

* Bump more envs to 3.14.

* Silence pylint.

* Fix another test for .php handling change.

* Remove test_install_asyncio_reactor.

* More bumps to 3.14.

* Revert docs-tests to use 3.13.

* Debug options for Windows.

* Re-enable xdist.

* Revert Windows PYTEST_ADDOPTS.

* Silence loop policy deprecation warnings.

* Restore a lost pylint suppression.

* Update asyncio_enabled_reactor_same_loop.py to new Twisted.

* Fix RobotFileParser tests for Python 3.14.5.
 2026-05-12 23:49:37 +05:00
Sanjay M a9324fbf76
Fix small documentation wording issues (#7480) 2026-05-04 08:43:21 +02:00
Adrian f8d103a65a
Add llms.txt and llms-full.txt generation (#7380) 2026-04-06 10:24:21 +02:00
Adrian Chaves 74c33e5172 Use the spam label when auto-closing PRs 2026-03-30 17:43:59 +02:00
Adrian 6cef717dad
Add a template for PRs (#7381) 2026-03-30 17:33:21 +02:00
Andrey Rakhmatullin a8e99aeb2e
Fix botocore-pinned and pypy3-* (#7363) 2026-03-26 08:21:58 +01:00
Andrey Rakhmatullin 6fe27ba33e
Add more no-reactor tests (#7259) 
* Generic changes and scrapy bench.

* scrapy check.

* scrapy crawl.

* scrapy fetch.

* scrapy parse.

* scrapy runspider.

* scrapy shell.

* Skip httpx tests on default-reactor.

* Review requires_reactor marks.

* Make test functions that require an event loop async def.

* Improve test_pending_asyncio_tasks().

* Add Mac OS DNS error.

* Refactor most of test_scheduler.py.

* Finish refactoring DownloaderAwareSchedulerTestMixin.

* Refactor test_engine_loop.py.

* Add the no-reactor-extra-deps tox env, run no-reactor on macos.

* Skip unhandled CancelledError when shutting down the engine.

* Fix typing and pre-commit checks.

* Fix typing problems in master.

---------

Co-authored-by: Adrian <adrian@zyte.com>
 2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin 6e0a0e476a
Fix running tests with -n auto, enable it on CI (#7257) 
* Make test_start_deprecated_super() more robust.

* Work around the FTPFilesStore.FTP_* initialization.

* Enable xdist on CI.

* Add PYTEST_ADDOPTS to tox passenv.
 2026-02-13 22:12:57 +05:00
Andrey Rakhmatullin 3186ccf5d5
Reactorless tests 1: the test env (#6952) 
* Reactorless tests.

* Updates after merging master.

* Update a new import.

* Updates after merging master.

* Update testenv:no-reactor-pinned.

* Foundations for the reactorless mode.

* Fixes after merging master.

* Updates after merging reactorless-base.

* Add simple subprocess tests for reactorless AsyncCrawler*.

* More reactorless tests.

* Refactor AsyncCrawlerProcess.start().

* More checks.

* Fix test_reactorless_import_hook.

* More tests.

* Call install_reactor() before asyncio.run().

* Cleanup.

* Rephrase.

* Fix a test.

* Fixes.

* Rephrasing.

* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.

* Fix a new import.

* Clarify --reactor marks.

* Cleanup.

* More docs/comments.

* Update test_fallback_workflow().

* Update TestDeferredFFromCoroF.

* Typing improvements.
 2026-02-02 19:25:30 +05:00
Pádraic Slattery 49930dfec5
chore: Update outdated GitHub Actions versions (#7223) 2026-01-23 12:26:58 +01:00
Santiago Beltrán Ruiz e30ba7d4ca
(#7180) Update codecov test results action to v5 on all CI workflows. (#7215) 2026-01-12 08:31:53 +01:00
Rui Xi 14737e91ed
Mitigate brotli and deflate decompression bombs DoS (#7134) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-11-17 14:21:50 +05:00
Andrey Rakhmatullin 804ae167df
Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
Andrey Rakhmatullin d414d393d4
Remove Python 3.9 support (#7121) 2025-10-27 12:37:49 +01:00
Andrey Rakhmatullin 393d715205
Drop PyPy 3.10 support, restore coverage for subprocesses (#7050) 2025-09-15 11:24:09 +02:00
Andrey Rakhmatullin 4899d416e7
Add PyPy 3.11 to CI. (#6697) 2025-05-06 12:31:28 +02:00
Suejung Shin e50914e0f5
Codecov: Add test analytics (#6741) 2025-03-21 12:28:47 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Adrián Chaves 26ecc93228
Run CI only on the main branch, on release branches and on PRs (#6720) 2025-03-11 14:12:11 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Andrey Rakhmatullin d4b152bbf6
Drop PyPy 3.9, add a pypy3-extra-deps CI job. (#6613) 2025-01-23 09:22:18 +01:00
Andrey Rakhmatullin 5d3aa80ad1
Switch CI to codecov/codecov-action and enable it on Windows. (#6609) 2025-01-07 10:52:26 +01:00
Andrey Rakhmatullin 4869315d10
Install libjpeg-dev on pinned envs to be able to install Pillow. (#6607) 2025-01-07 10:46:12 +01:00
Andrey Rakhmatullin 4a0c05749c Bump tool versions. 2024-12-29 14:29:27 +05:00
Daniel Graña dc4d6d16ea Verified PyPI releases (a.k.a. PEP740) 2024-11-15 00:09:00 -03:00
Andrey Rakhmatullin 845b1ffd44 Merge remote-tracking branch 'origin/master' into py313 2024-10-16 14:50:12 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Klaus Rettinghaus 5ef5474172
update gh-action-pypi-publish 2024-10-09 20:38:50 +02:00
Andrey Rakhmatullin 29bb869284 Remove the beta block. 2024-10-08 15:43:17 +05:00
Andrey Rakhmatullin df6c51af0f Use the 3.13 release. 2024-10-08 15:37:49 +05:00
Andrey Rakhmatullin 1e68d3c0bf Merge branch 'master' into py313 2024-08-11 13:46:51 +05:00
Andrey Rakhmatullin 435686830c Bump the Python version for RTD. 2024-07-11 12:25:13 +05:00
Andrey Rakhmatullin 129dbfa0bf Bump tool versions. 2024-07-11 12:23:43 +05:00
Andrey Rakhmatullin 5755e224d5 Help with building lxml on 3.13beta1 for checks too. 2024-05-28 14:50:09 +05:00
Andrey Rakhmatullin 986d1ee1dd
Move CI from the decommissioned macos-11 to macos-latest. (#6372) 2024-05-28 10:37:19 +02:00
Andrey Rakhmatullin b6d3d9076f Help with building lxml on 3.13beta1. 2024-05-22 13:20:48 +05:00
Andrey Rakhmatullin 534a66e954 Bump 3.13 to beta1. 2024-05-22 13:16:00 +05:00
Andrey Rakhmatullin 85d7458651 Merge branch 'master' into py313 2024-05-22 13:14:59 +05:00
Andrey Rakhmatullin 93f0628530
Improve typing for Spider.parse(). (#6274) 2024-05-13 11:55:45 +02:00
Adrián Chaves c66b517706 Add Python 3.13 alpha to CI 2023-12-01 10:36:27 +01:00
Andrey Rakhmatullin 991121fa91
Re-enable uvloop tests on 3.12 (#6098) 2023-10-17 19:24:44 +02:00
Andrey Rakhmatullin c65567988d
Change supported PyPy versions to 3.9 and 3.10. (#6087) 
* Change support PyPy versions to 3.9 and 3.10.

* Update the RTD URL for coverage.

* Move pypy3-pinned back to pypy3.8.
 2023-10-06 11:27:02 +02:00
Andrey Rakhmatullin 5b0b00212e
Merge pull request #6083 from wRAR/py3.12-release 
Adapt to the Python 3.12 final release
 2023-10-03 20:20:12 +04:00
Klaus Rettinghaus 9ae8d97d81
Update GitHub Actions checkout action to version 4 (#6084) 2023-10-03 13:04:18 +02:00
Andrey Rakhmatullin 60d5f391c4 Use the 3.12 release, enable it for all OSes. 2023-10-02 21:35:45 +04:00
Andrew Armbruster 884840e3a3 Use concurrency groups to limit duplicate jobs 
Leverage concurrency groups along with cancel-in-progress to favor running the most recent job.

Concurrency groups are on a per workflow, per branch/tag basis. So, pushing newer updates to a branch, e.g. as part of a PR, should cancel any in progress runs of workflows that have been retriggered.

See:
- https://docs.github.com/en/enterprise-cloud@latest/actions/using-workflows/workflow-syntax-for-github-actions#concurrency
- https://docs.github.com/en/enterprise-cloud@latest/actions/learn-github-actions/contexts#github-context
 2023-10-01 12:12:24 +02:00
Adrián Chaves 4c98d6068a Add a template for question/help issues 2023-09-25 10:18:35 +02:00
Andrey Rakhmatullin da39fbd270
Update tool versions (#6046) 2023-09-12 20:07:10 +02:00