RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,935 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

2852 Commits

Author SHA1 Message Date
Andrey Rakhmatullin d414d393d4
Remove Python 3.9 support (#7121) 2025-10-27 12:37:49 +01:00
Adrián Chaves eb496470f1
Add missing doc deps for new examples (#7116) 2025-10-24 11:58:18 +02:00
Michael a5bbeb2586
Expanded documentation for issue #6676 (#6957) 2025-10-24 10:30:33 +02:00
sstegmanbsd 2d073a9c0d
Fix typo in logging documentation (#7109) 2025-10-20 15:52:46 +02:00
Andrey Rakhmatullin b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. (#7079) 2025-10-16 13:15:07 +02:00
Sebastien Dionne a02abdcf63
Fix typos in documentation (#7076) 2025-10-06 11:43:36 +02:00
Andrey Rakhmatullin 1c6ba00fd0
Move LogCounterHandler into a new extension (#7046) 
* Move LogCounterHandler into Crawler.crawl().

* Add the LogCount extension.

* Update test_spider_custom_settings_log_level().

* Typo.

* Update docs.
 2025-10-03 20:14:11 +05:00
Laerte Pereira f2531808f3
docs: Add Edit on GitHub (#7058) 2025-09-28 13:16:47 +05:00
Andrey Rakhmatullin 2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. (#7037) 
* Deprecate the spider arg of downloader middleware methods.

* Add a test for deprecated spider args in downloader mws.
 2025-09-02 21:51:47 +05:00
Andrey Rakhmatullin ed63fa94d6
Deprecate spider args of middlewares and pipelines. (#7006) 
* Deprecate spider args of spider middleware methods.

* Deprecate the spider arg of pipeline process_item().

* Fix a typing issue.

* Deprecate the spider arg of pipeline {open,close}_spider().

* Cleanup.

* Update docs.

* Add pragma: no cover to some deprecated code.

* More tests.

* More tests.

* More tests.

* Simplify _process_parallel().
 2025-08-21 13:23:10 +05:00
Andrey Rakhmatullin b68330811b
Deprecate spider arguments of StatsCollector methods (#7011) 2025-08-20 09:52:22 +02:00
Andrey Rakhmatullin edba1ad572
Refactor RobotsTxtMiddleware to use async def process_request(). (#6802) 2025-08-18 18:40:46 +02:00
Felipe Benevolo 89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() (#6842) 
* added async_download()

* add case tests for ExecutionEngine download() and async_download()

* change async_download() to download_async() in Engine class and subsequent tests

* fixed reactor import in test_engine.py

* Simplify TestEngineDownloadAsync.

* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks

* Simplify ExecutionEngine.download().

* Remove an unneeded import.

* Refactor ExecutionEngine.download{,_async}() tests.

* Deprecate ExecutionEngine.download().

* Update doc examples to use download_async().

* Replace uses of download() with download_async().

* Fix test_downloadermiddleware_robotstxt.py.

* Remove a stray file.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-07-27 23:17:51 +05:00
Andrey Rakhmatullin 552f2fb91e
Generic improvements related to reactorless tests (#6968) 
* Skip doctests that import the reactor.

* Extract FTPDownloadHandler tests.

* Skip TestHttps2ClientProtocol via pytestmark.

* Replace some explicit reactor.callLater() in tests.

* Simplify TestRequestSendOrder._test_request_order().
 2025-07-25 13:35:00 +05:00
Andrey Rakhmatullin d9ef0350d8 Merge remote-tracking branch 'origin/2.13' 2025-07-02 20:46:02 +05:00
Andrey Rakhmatullin 155a504f24 Bump version: 2.13.2 → 2.13.3 2025-07-02 20:39:46 +05:00
Andrey Rakhmatullin cf465bf644
Release notes for 2.13.3. (#6934) 
* Release notes for 2.13.3.

* Address feedback.
 2025-07-02 20:36:01 +05:00
Laerte Pereira dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting (#6921) 
* Deprecate CONCURRENT_REQUESTS_PER_IP setting

* Typing fix

* PR review

* tests

* oops

* Update scrapy/settings/default_settings.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* update logic

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-06-30 16:51:05 +05:00
Andrey Rakhmatullin 3843091c5f
Mark abstract base classes as such. (#6930) 
* Mark abstract base classes as such.

* Remove an unneeded method.

* Fix exporter test coverage.
 2025-06-30 15:14:34 +05:00
Andrey Rakhmatullin 03fe7a6424 Add a deprecation notice for the offsite spider middleware. 2025-06-29 02:35:48 +05:00
Andrey Rakhmatullin 2c1c10e923
Remove deprecated offsite spider middleware. (#6926) 
* Remove deprecated offsite spider middleware.

* Add a deprecation notice for the offsite spider middleware.
 2025-06-29 02:35:20 +05:00
Adrián Chaves ba1cedee1b Document the new defaults of throttling settings (#6923) 2025-06-28 18:13:54 +05:00
Adrián Chaves 3019393686
Document the new defaults of throttling settings (#6923) 2025-06-28 18:04:40 +05:00
Adrián Chaves 1473f4d347
Remove the deprecated sphinx-hoverxref (#6922) 2025-06-28 18:04:06 +05:00
Andrey Rakhmatullin 95172659af
Add sphinx-lint. (#6920) 2025-06-28 01:37:20 +02:00
Andrey Rakhmatullin 843ad1afb1 Replace Black with Ruff in contributing.rst. (#6903) 2025-06-23 22:25:56 +05:00
Andrey Rakhmatullin 712e965dbd
Replace Black with Ruff in contributing.rst. (#6903) 2025-06-23 17:56:48 +02:00
Andrey Rakhmatullin daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts (#6888) 
* Remove the chdir fixture.

* Re-enable fancy pytest asserts.

* Remove doc files from pytest ignores.

* Restore docs/_ext in test collection ignores.

* Skip a doctest that fails on Windows.

* Fix tests that were writing to the current dir.
 2025-06-14 14:01:20 +05:00
Andrey Rakhmatullin 0390176ecd Merge branch '2.13' 2025-06-09 15:54:35 +05:00
Andrey Rakhmatullin c6740604a4 Bump version: 2.13.1 → 2.13.2 2025-06-09 15:21:06 +05:00
Andrey Rakhmatullin 7400868ad5
Release notes for 2.13.2. (#6868) 
* Release notes for 2.13.2.

* Update release notes.
 2025-06-09 15:19:02 +05:00
Andrey Rakhmatullin 405d9bc8a2
More docs for the is_asyncio_reactor_installed() behavior change. (#6866) 2025-06-06 12:59:49 +02:00
Andrey Rakhmatullin d99234a33f
Install the reactor explicitly in CrawlerRunner examples. (#6865) 2025-06-06 11:54:21 +02:00
Adrián Chaves b8cd079014 Shorten the README and remove broken links to scrapy.org (#6833) 2025-06-06 12:44:02 +05:00
Adrián Chaves 105c0afb6e Feature the new logo in the README (#6831) 2025-06-06 12:43:56 +05:00
Andrey Rakhmatullin 3aa5e75787
Use AsyncCrawlerProcess in commands. (#6845) 
* Use AsyncCrawlerProcess in commands.

* Ignore coverage of abstract methods.

* Address feedback.
 2025-06-03 17:47:52 +05:00
Rodrigosnrocha 9cc23641cc
Deprecate _parse_response and implement parse_with_rules (#6804) 2025-06-02 17:00:17 +02:00
Andrey Rakhmatullin e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. (#6824) 
* Don't use CrawlerProcess in the commands that don't need it.

* Use a dummy spider loader in runspider.
 2025-05-28 19:57:33 +05:00
Andrey Rakhmatullin 05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 
* Add a basic Crawler.crawl_async().

* Add custom loop tests for *CrawlerRunner.

* Add AsyncCrawlerProcess.

* Update related docs.

* Update practices.rst.

* Address test failures.

* Add a note about AsyncCrawler* to the docs about switching reactors.

* Address feedback.

* Update for TID253.

* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
 2025-05-28 18:55:44 +05:00
Andrey Rakhmatullin dceb85bf3e
Add is_asyncio_available(). (#6827) 
* Add is_asyncio_available().

* Print unexpected warnings in test_install_asyncio_reactor().

* Fix printing warnings.

* Fix printing warnings - 2.

* Skip TestDeferredToFuture on non-asyncio.

* Test the is_asyncio_available() exception.
 2025-05-28 14:46:39 +05:00
Adrián Chaves c480c77f54
Shorten the README and remove broken links to scrapy.org (#6833) 2025-05-28 11:35:18 +02:00
Adrián Chaves f98ffc71d2
2.13.1 (#6832) 2025-05-28 10:58:34 +02:00
Adrián Chaves 7b4cf06b6e
Feature the new logo in the README (#6831) 2025-05-28 10:54:36 +02:00
Adrián Chaves 7fe7f1734a Bump version: 2.13.0 → 2.13.1 2025-05-28 10:12:46 +02:00
Adrián Chaves 0cdb971f63 Complete the release notes 2025-05-28 10:06:04 +02:00
Mikhail Korobov c6643c08ee
Merge pull request #6828 from scrapy/lower-start-priority 
Scrapy 2.13.1: Prioritize other requests over start requests
 2025-05-28 11:29:57 +05:00
Andrey Rakhmatullin 06dec08125 Fix a wrong versionadded usage. (#6822) 2025-05-27 12:34:40 +05:00
Keval Sakhiya 43087fe1df Fix typo in documentation and code: 'needs_backoff' -> 'needs_backout' (#6815) 
Corrected the typo in the code and documentation where 'needs_backoff' was incorrectly used instead of 'needs_backout'.
 2025-05-27 12:34:33 +05:00
Adrián Chaves 05529f3017 Release notes for Scrapy 2.13.1 2025-05-27 08:44:18 +02:00
Andrey Rakhmatullin f2fc177f1f
Fix a wrong versionadded usage. (#6822) 2025-05-23 11:06:33 +02:00