RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,796 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

2803 Commits

Author SHA1 Message Date
Andrey Rakhmatullin f2fc177f1f
Fix a wrong versionadded usage. (#6822) 2025-05-23 11:06:33 +02:00
Keval Sakhiya ff7d29654a
Fix typo in documentation and code: 'needs_backoff' -> 'needs_backout' (#6815) 
Corrected the typo in the code and documentation where 'needs_backoff' was incorrectly used instead of 'needs_backout'.
 2025-05-20 07:51:31 +02:00
Andrey Rakhmatullin 3c2cd53abb Skip the doctest. 2025-05-15 22:17:37 +05:00
Andrey Rakhmatullin bf1bfaaa3e Slight improvements for the signal docs. 2025-05-15 20:02:38 +05:00
Andrey Rakhmatullin 1ddcb568e2 Add send_catch_log_async(). 2025-05-15 15:45:55 +05:00
Andrey Rakhmatullin 82acef3051
Add AsyncCrawlerRunner. (#6796) 2025-05-14 16:21:18 +02:00
Andrey Rakhmatullin 82a3245158 Bump version: 2.12.0 → 2.13.0 2025-05-08 14:43:34 +05:00
Andrey Rakhmatullin 0ce693dfa9 Update VERSION strings. 2025-05-08 13:23:19 +05:00
Andrey Rakhmatullin b07e6d4ea8 Merge remote-tracking branch 'origin/master' into relnotes-2.13.0 2025-05-08 13:17:47 +05:00
Andrey Rakhmatullin e86b5051c2 Merge commit '373e501' into relnotes-2.13.0 2025-05-08 10:55:57 +05:00
Andrey Rakhmatullin 5f6d1b464b Cover up to 373e501 in the release notes. 2025-05-08 10:55:40 +05:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Adrián Chaves 373e501f78
Link to scrapy.org from the docs (#6780) 2025-05-07 16:11:22 +02:00
Andrey Rakhmatullin 474f8312ff
Merge pull request #6776 from wRAR/deferred-apis 
Add a list of Deferred-only APIs.
 2025-05-05 21:28:50 +04:00
Andrey Rakhmatullin acb5f895cd
Update docs/topics/coroutines.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2025-05-05 22:28:36 +05:00
Adrián Chaves 523fc25c4d
Document default values set by startproject (#6775) 2025-05-05 18:51:15 +02:00
Andrey Rakhmatullin b93290f28a Add a list of Deferred-only APIs. 2025-05-05 19:38:04 +05:00
Andrey Rakhmatullin 4aba7e5f66 Mention the deprecation of TestSpider. 2025-04-25 20:16:26 +05:00
Andrey Rakhmatullin 095140f134 Cover the current master in the release notes (up to daf9db7). 2025-04-25 19:43:29 +05:00
Andrey Rakhmatullin b1f85b5a17 Release notes for 2.13.0, up to b4c2531021. 2025-04-24 20:03:36 +05:00
Andrey Rakhmatullin daf9db72b2
Base class for universal spider middlewares (#6693) 
* Initial BaseSpiderMiddleware.

* Rename the new methods.

* Remove the spider argument from new BaseSpiderMiddleware methods.

* Add docs for BaseSpiderMiddleware.

* Silence pylint.

* Add BaseSpiderMiddleware tests.

* Add a release note.
 2025-04-23 19:29:04 +05:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Andrey Rakhmatullin 9057bf4e1e
More docs about Deferred<->Future interoperability. (#6734) 2025-03-15 11:47:16 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves 9b7db1a068
Move some reference docs of Request to the code (#6721) 2025-03-11 12:43:50 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Andrey Rakhmatullin d2e5486d5a Remove the Splash recommendation. 2025-03-07 13:20:42 +05:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin c200458f24
Add more docs for updating sync spider middlewares. (#6688) 2025-02-25 00:16:23 +01:00
anubhav f041f26a6f
Support dark mode in the documentation (#6653) 2025-02-06 18:07:07 +01:00
Laerte Pereira 2eb3c75c69 Remove AjaxCrawlMiddleware mention from built-in downloader middleware 2025-02-05 13:16:51 -03:00
Andrey Rakhmatullin 76a8badd24 Add a deprecation notice to the AjaxCrawlMiddleware docs. 2025-02-03 14:55:10 +05:00
Andrey Rakhmatullin 393ff96e45 Deprecate AjaxCrawlMiddleware. 2025-02-03 14:08:05 +05:00
Rotzbua 7e61ff3524
Upgrade Sphinx (#6624) 2025-01-22 18:09:42 +01:00
guillermo-bondonno 9bc0029d27
Allow updating pre-crawler settings from add-ons (#6568) 2025-01-22 12:07:44 +01:00
Rotzbua e0c828b7f6
chore(docs): refactor config (#6623) 2025-01-20 12:18:30 +01:00
Rotzbua ee4f527f47
fix(docs): pillow domain is shut down permanently 
See https://github.com/python-pillow/Pillow/issues/8585
 2025-01-19 14:58:02 +01:00
Rotzbua d7168577b8
chore(docs): migrate to RTD template v3 
notable change: Drop support for all versions of Internet Explorer.
 2025-01-19 13:50:53 +01:00
anubhav ca345a3b73
Flexible severity of logging level when items are dropped (#6608) 2025-01-15 11:08:18 +01:00
Adrián Chaves 98ba61256d
Deprecate BaseDupeFilter.log() and improve dupefilter docs (#4151) 
* Remove BaseDupeFilter.log()

It is never called because request_seen() always returns False

* Document the interface of DUPEFILTER_CLASS classes

* Remove unnecessary BaseDupeFilter comments and add a short class description

* Improve the documentation related to the DUPEFILTER_CLASS setting

* Deprecate BaseDupeFilter.log

* Update the docs

* Fix the new code example

* Remove typing to keep the example short

Otherwise, it would have required yet another import line (from __future__ or typing).

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-01-14 19:36:56 +05:00
Kevin Lloyd Bernal 1fc91bb462
new `allow_offsite` parameter in OffsiteMiddleware (#6151) 
* new 'allow_offsite' parameter in OffsiteMiddleware

* document deprecated dont_filter flag in OffsiteMiddleware

* avoid deprecating dont_filter in OffsiteMiddleware

* Copy the code to the downloader mw.

* Add tests for allow_offsite in the downloader mw.

* Mark allow_offsite with reqmeta.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-01-08 21:28:51 +05:00
Andrey Rakhmatullin 59fcb9b93c
Improve internal refs to scrapy.Request and scrapy.Selector (#6526) 
* Improve internal refs to scrapy.Selector.

* Improve internal refs to scrapy.Request.

* More scrapy.http fixes.

* Fix FormRequest refs.

* More fixes.

* Simplifications.

* Last fixes.

* Add the parsel intersphinx.
 2025-01-07 16:18:18 +05:00
Andrey Rakhmatullin b70443f2d0 Split ruff and pylint ignores into two categories, some pylint cleanup. 2025-01-02 01:03:42 +05:00
Adrián Chaves 21b9ba717c
Allow customizing logged software versions (#6582) 
Co-authored-by: Grammy Jiang <grammy.jiang@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-12-16 14:46:23 +01:00
Adrián Chaves 5c2df5cf2a
Contributing: add a section on finding work (#6575) 2024-12-12 11:38:30 +01:00
Andrey Rakhmatullin efb53aafdc Fix a typo that broke PDF builds. 2024-11-18 15:44:52 +05:00
Adrián Chaves 10089c6fe2
2.12 release notes (#6226) 
* Cover 2.12 in the release notes up to 9bb973dc54

* Add one more highlight

* Better merge of the news entries.

* Cover 2.12 in the release notes up to 642af40.

* Cover 2.12 in the release notes up to 7a0a34b.

* Cover 2.12 in the release notes up to b4bad97.

* Add not yet merged PRs #6463, #6507, #6511 to the 2.12 release notes.

* Cover 2.12 in the release notes up to d85c39f, small fixes.

* Cover 2.12 in the release notes up to d215669.

* Cover #6527 in the release notes.

* Address PR feedback.

* Cover recent PRs.

* Finalize the 2.12.0 release notes, small additional fixes.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-11-18 13:07:32 +05:00