RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,204 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

144 Commits

Author SHA1 Message Date
Andrey Rakhmatullin d8ba1571e7
Assorted docs fixes, part 2. (#7725) 
* Assorted docs fixes, part 2.

* Second pass.

* Address feedback.
 2026-07-14 00:49:44 +05:00
Andrey Rakhmatullin bdf3067935
Assorted docs fixes, part 1. (#7710) 2026-07-07 18:39:34 +05:00
Adrian a8ffdcf851
Implement HTTP Auth settings and request metadata keys (#7590) 
* Implement HTTP Auth settings and request metadata keys

* Fix doc-tests

* Require the domain to be set when using settings
 2026-06-11 15:56:53 +05:00
Omkar Kabde 5149e2c679
docs: switch `scrapy.Item` examples to dataclasses (#7513) 
* docs: switch `scrapy.Item` examples to dataclasses

* make serializer doc generic

* use modern type hints

* docs/spiders: switch TestItem consumer snippets to attribute access

Since the TestItem migration to @dataclass, the existing
item["id"] = ... assignments would raise TypeError on copy-paste.
Switch to item.id = ... to match the new dataclass declaration.

The snippets sit under .. skip: next so docs-tests still pass either
way, but the change keeps the examples runnable for readers.
 2026-06-08 12:40:28 +05:00
Adrian f8d103a65a
Add llms.txt and llms-full.txt generation (#7380) 2026-04-06 10:24:21 +02:00
Andrey Rakhmatullin 7bef98b4f1
Release notes for 2.14.0 (#7035) 
* Initial release notes for 2.14.0, up to ed63fa9.

* Cover 2.14 in the release notes up to 393d715.

* Cover 2.14 in the release notes up to eb49647.

* Cover 2.14 in the release notes up to 426aafd.

* Cover 2.14 in the release notes up to 1e8de24.

* Cover 2.14 in the release notes up to 5a7e132.

* Bump sphinx-lint.

* Finalize the 2.14.0 release notes.

* Drop more of the old versionadded directives.

* Address feedback.
 2026-01-05 14:51:05 +05:00
Adham 3llam 9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation (#7198) 2025-12-22 10:00:39 +01:00
Andrey Rakhmatullin 588f3d4f65
Remove docs mentioning deprecated spider attrs. (#7176) 2025-12-15 09:51:56 +01:00
Michael a5bbeb2586
Expanded documentation for issue #6676 (#6957) 2025-10-24 10:30:33 +02:00
Sebastien Dionne a02abdcf63
Fix typos in documentation (#7076) 2025-10-06 11:43:36 +02:00
Keval Sakhiya ff7d29654a
Fix typo in documentation and code: 'needs_backoff' -> 'needs_backout' (#6815) 
Corrected the typo in the code and documentation where 'needs_backoff' was incorrectly used instead of 'needs_backout'.
 2025-05-20 07:51:31 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Georgiy Zatserklianyi 6ce0342beb
Allow yielding items from start_requests (#6417) 
Co-authored-by: Georgiy Zatserklianyi <george.zatseklyany@gmail.com>
Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-08-26 20:53:06 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Andrey Rakhmatullin 603aa4924a Improve the docs about Crawler attributes and settings initialization. 2023-11-15 19:51:04 +04:00
Andrey Rakhmatullin 2fa768399a Replace the VERSION vars. 2023-09-15 19:19:42 +04:00
Andrey Rakhmatullin ac201d310b Small improvements, 2023-09-13 16:17:52 +04:00
Andrey Rakhmatullin 61ef37a594 Docs improvements. 2023-09-13 15:47:41 +04:00
Andrey Rakhmatullin 61e6bfc023 Docs improvements. 2023-09-13 12:35:15 +04:00
Andrey Rakhmatullin a3f22046ef Document changing settings in Spider.from_crawler(). 2023-09-12 20:41:36 +04:00
Andrey Rakhmatullin af1be835e4
Apply suggestions from code review 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-08-02 19:46:16 +04:00
Andrey Rakhmatullin 72462a53e2 Add more docs for update_settings(). 2023-08-02 12:32:53 +04:00
Jalil SA 44cdaa442b
Update docs/topics/spiders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-03-17 13:19:03 -06:00
Jalil SA 24f28c415c fix: docs update_settings() 2023-03-17 12:16:08 -06:00
Jalil SA a1fc37cbff
Update docs/topics/spiders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-03-17 12:13:05 -06:00
Jalil SA 495372648c fix: docs update_settings() 2023-03-16 23:14:57 -06:00
Jalil SA 96d51c3afa docs: update 2023-03-08 04:21:21 -06:00
Jalil SA 1d862d0831 fix: remove line breaks 2023-03-08 03:26:38 -06:00
Jalil SA 05893e1796 docs: Spider.update_settings 2023-03-08 02:59:47 -06:00
Andrey Rakhmatullin 76eba9977b
Skip or fix failing code block tests (#5826) 2023-02-14 09:40:38 +01:00
pankaj1707k c1bbb299d7
Add and run pre-commit hook 'blacken-docs' 
Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
 2023-02-01 16:30:57 +05:30
Josh Soref b62aacfee3 spelling: initializing 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
zaid-ismail031 c49b5aaf77 Changed incorrect information regarding the return type of parse/request callback method. 2022-08-26 21:17:31 +02:00
Adrián Chaves aa0306a167
Cover 2.6.0 in the release notes (#5399) 2022-03-01 12:16:37 +01:00
Kian-Meng, Ang 3a263280ba Fix typos 2021-10-11 22:32:42 +08:00
kamran890 e5998fb846 Document spider.state attribute (#5174) 2021-09-22 03:15:12 +05:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Adrián Chaves 73ff9ffd64
spiders.rst: indent warnings into class descriptions 2021-06-26 08:58:29 +02:00
Shinichi Takayanagi bd60c3f41f
More documentation for setting spider atributes 
* docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17

* Update feed-exports.rst

* Update feed-exports.rst

* Reflects the comments

* Remove redundant newline

* Update docs/topics/feed-exports.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Apply suggestions from code review

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 00:58:04 +05:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Eugenio Lacuesta 2be2bdd2df
Merge remote-tracking branch 'upstream/master' into spider.parse 2020-07-08 14:00:54 -03:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Rohit Sharma 0cc0e51ca3 remove uneeded escape sequence 
removed uneeded wscape sequences from method arguments in the docs
folder
 2020-05-24 13:38:44 -07:00
Aditya f37b1bdc56 [docs] update redirect links to python3 2020-03-20 05:31:41 +05:30
Eugenio Lacuesta bde0384dfd
Merge branch 'master' into spider.parse 2020-03-14 19:53:36 -03:00
Adrián Chaves 6aa0ba4553
Write release notes for Scrapy 2.0.0 (#4329) 2020-03-03 09:11:11 +01:00
nyov a34c366fa4 DOC linkcheck run; https and 301 link updates. 
Closes #4359
 2020-02-25 16:51:33 +00:00
Eugenio Lacuesta 8a1dc26d46
[doc] Note about the 'parse' method for CrawlSpider/XMLFeedSpider 2019-12-26 15:14:47 -03:00
Eugenio Lacuesta 5982e3477c
Take keyword arguments in base parsing methods 2019-12-23 14:12:21 -03:00