RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,861 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

2827 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 95172659af
Add sphinx-lint. (#6920) 2025-06-28 01:37:20 +02:00
Andrey Rakhmatullin 712e965dbd
Replace Black with Ruff in contributing.rst. (#6903) 2025-06-23 17:56:48 +02:00
Andrey Rakhmatullin daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts (#6888) 
* Remove the chdir fixture.

* Re-enable fancy pytest asserts.

* Remove doc files from pytest ignores.

* Restore docs/_ext in test collection ignores.

* Skip a doctest that fails on Windows.

* Fix tests that were writing to the current dir.
 2025-06-14 14:01:20 +05:00
Andrey Rakhmatullin 0390176ecd Merge branch '2.13' 2025-06-09 15:54:35 +05:00
Andrey Rakhmatullin c6740604a4 Bump version: 2.13.1 → 2.13.2 2025-06-09 15:21:06 +05:00
Andrey Rakhmatullin 7400868ad5
Release notes for 2.13.2. (#6868) 
* Release notes for 2.13.2.

* Update release notes.
 2025-06-09 15:19:02 +05:00
Andrey Rakhmatullin 405d9bc8a2
More docs for the is_asyncio_reactor_installed() behavior change. (#6866) 2025-06-06 12:59:49 +02:00
Andrey Rakhmatullin d99234a33f
Install the reactor explicitly in CrawlerRunner examples. (#6865) 2025-06-06 11:54:21 +02:00
Adrián Chaves b8cd079014 Shorten the README and remove broken links to scrapy.org (#6833) 2025-06-06 12:44:02 +05:00
Adrián Chaves 105c0afb6e Feature the new logo in the README (#6831) 2025-06-06 12:43:56 +05:00
Andrey Rakhmatullin 3aa5e75787
Use AsyncCrawlerProcess in commands. (#6845) 
* Use AsyncCrawlerProcess in commands.

* Ignore coverage of abstract methods.

* Address feedback.
 2025-06-03 17:47:52 +05:00
Rodrigosnrocha 9cc23641cc
Deprecate _parse_response and implement parse_with_rules (#6804) 2025-06-02 17:00:17 +02:00
Andrey Rakhmatullin e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. (#6824) 
* Don't use CrawlerProcess in the commands that don't need it.

* Use a dummy spider loader in runspider.
 2025-05-28 19:57:33 +05:00
Andrey Rakhmatullin 05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 
* Add a basic Crawler.crawl_async().

* Add custom loop tests for *CrawlerRunner.

* Add AsyncCrawlerProcess.

* Update related docs.

* Update practices.rst.

* Address test failures.

* Add a note about AsyncCrawler* to the docs about switching reactors.

* Address feedback.

* Update for TID253.

* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
 2025-05-28 18:55:44 +05:00
Andrey Rakhmatullin dceb85bf3e
Add is_asyncio_available(). (#6827) 
* Add is_asyncio_available().

* Print unexpected warnings in test_install_asyncio_reactor().

* Fix printing warnings.

* Fix printing warnings - 2.

* Skip TestDeferredToFuture on non-asyncio.

* Test the is_asyncio_available() exception.
 2025-05-28 14:46:39 +05:00
Adrián Chaves c480c77f54
Shorten the README and remove broken links to scrapy.org (#6833) 2025-05-28 11:35:18 +02:00
Adrián Chaves f98ffc71d2
2.13.1 (#6832) 2025-05-28 10:58:34 +02:00
Adrián Chaves 7b4cf06b6e
Feature the new logo in the README (#6831) 2025-05-28 10:54:36 +02:00
Adrián Chaves 7fe7f1734a Bump version: 2.13.0 → 2.13.1 2025-05-28 10:12:46 +02:00
Adrián Chaves 0cdb971f63 Complete the release notes 2025-05-28 10:06:04 +02:00
Mikhail Korobov c6643c08ee
Merge pull request #6828 from scrapy/lower-start-priority 
Scrapy 2.13.1: Prioritize other requests over start requests
 2025-05-28 11:29:57 +05:00
Andrey Rakhmatullin 06dec08125 Fix a wrong versionadded usage. (#6822) 2025-05-27 12:34:40 +05:00
Keval Sakhiya 43087fe1df Fix typo in documentation and code: 'needs_backoff' -> 'needs_backout' (#6815) 
Corrected the typo in the code and documentation where 'needs_backoff' was incorrectly used instead of 'needs_backout'.
 2025-05-27 12:34:33 +05:00
Adrián Chaves 05529f3017 Release notes for Scrapy 2.13.1 2025-05-27 08:44:18 +02:00
Andrey Rakhmatullin f2fc177f1f
Fix a wrong versionadded usage. (#6822) 2025-05-23 11:06:33 +02:00
Keval Sakhiya ff7d29654a
Fix typo in documentation and code: 'needs_backoff' -> 'needs_backout' (#6815) 
Corrected the typo in the code and documentation where 'needs_backoff' was incorrectly used instead of 'needs_backout'.
 2025-05-20 07:51:31 +02:00
Andrey Rakhmatullin 3c2cd53abb Skip the doctest. 2025-05-15 22:17:37 +05:00
Andrey Rakhmatullin bf1bfaaa3e Slight improvements for the signal docs. 2025-05-15 20:02:38 +05:00
Andrey Rakhmatullin 1ddcb568e2 Add send_catch_log_async(). 2025-05-15 15:45:55 +05:00
Andrey Rakhmatullin 82acef3051
Add AsyncCrawlerRunner. (#6796) 2025-05-14 16:21:18 +02:00
Andrey Rakhmatullin 82a3245158 Bump version: 2.12.0 → 2.13.0 2025-05-08 14:43:34 +05:00
Andrey Rakhmatullin 0ce693dfa9 Update VERSION strings. 2025-05-08 13:23:19 +05:00
Andrey Rakhmatullin b07e6d4ea8 Merge remote-tracking branch 'origin/master' into relnotes-2.13.0 2025-05-08 13:17:47 +05:00
Andrey Rakhmatullin e86b5051c2 Merge commit '373e501' into relnotes-2.13.0 2025-05-08 10:55:57 +05:00
Andrey Rakhmatullin 5f6d1b464b Cover up to 373e501 in the release notes. 2025-05-08 10:55:40 +05:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Adrián Chaves 373e501f78
Link to scrapy.org from the docs (#6780) 2025-05-07 16:11:22 +02:00
Andrey Rakhmatullin 474f8312ff
Merge pull request #6776 from wRAR/deferred-apis 
Add a list of Deferred-only APIs.
 2025-05-05 21:28:50 +04:00
Andrey Rakhmatullin acb5f895cd
Update docs/topics/coroutines.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2025-05-05 22:28:36 +05:00
Adrián Chaves 523fc25c4d
Document default values set by startproject (#6775) 2025-05-05 18:51:15 +02:00
Andrey Rakhmatullin b93290f28a Add a list of Deferred-only APIs. 2025-05-05 19:38:04 +05:00
Andrey Rakhmatullin 4aba7e5f66 Mention the deprecation of TestSpider. 2025-04-25 20:16:26 +05:00
Andrey Rakhmatullin 095140f134 Cover the current master in the release notes (up to daf9db7). 2025-04-25 19:43:29 +05:00
Andrey Rakhmatullin b1f85b5a17 Release notes for 2.13.0, up to b4c2531021. 2025-04-24 20:03:36 +05:00
Andrey Rakhmatullin daf9db72b2
Base class for universal spider middlewares (#6693) 
* Initial BaseSpiderMiddleware.

* Rename the new methods.

* Remove the spider argument from new BaseSpiderMiddleware methods.

* Add docs for BaseSpiderMiddleware.

* Silence pylint.

* Add BaseSpiderMiddleware tests.

* Add a release note.
 2025-04-23 19:29:04 +05:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Andrey Rakhmatullin 9057bf4e1e
More docs about Deferred<->Future interoperability. (#6734) 2025-03-15 11:47:16 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00