RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,262 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

2993 Commits

Author SHA1 Message Date
Adrian 1f03fbc17e
Add scrapy.utils.asyncio.sleep() (#7843) 2026-07-31 18:59:38 +05:00
Adrian 6cefaa5434
Add a stats reference (#7814) 2026-07-31 15:43:40 +05:00
Adrian 746bc7548d
Clarify crawl vs runspider in help and docs (#7832) 2026-07-31 12:17:34 +05:00
Adrian 37661508db
Add RobotParser.crawl_delay() and a robots_parsed signal (#7830) 
* Add RobotParser.crawl_delay() and a robots_parsed signal

* Improve test coverage
 2026-07-31 11:59:52 +05:00
Adrian 434fd1154a
Improve pre-crawler setting docs (#7835) 2026-07-31 11:38:33 +05:00
Adrian f02a99fe71
Add doc sections for callbacks and errbacks (#7821) 2026-07-30 20:15:15 +05:00
Adrian 433603e6ca
Add AWS_MAX_POOL_CONNECTIONS (#7794) 2026-07-30 15:51:45 +02:00
Adrian 98696efa80
Export item fields in declaration order (#7824) 2026-07-30 15:46:45 +05:00
Laerte Pereira 5b4828a012
docs(practices): Remove scrapoxy mention (#7817) 2026-07-30 08:02:54 +02:00
Adrian 0cbb20e8e8
Treat broken cache records as cache misses (#7805) 2026-07-29 14:39:27 +05:00
Youssef Mohamed bc5b5fb1f6
Add Request.to_curl() (#7743) (#7802) 2026-07-29 13:47:15 +05:00
SpiliosDmk 96195e4a61
docs: fix a/an article before XML in faq.rst (#7779) 
'a XML document' -> 'an XML document' (XML is pronounced with a vowel sound).
 2026-07-25 10:20:18 +05:00
SpiliosDmk e4ae4aad52
docs: fix missing 'of' in faq.rst (#7780) 
'instead joining the strings' -> 'instead of joining the strings'.
 2026-07-25 10:19:53 +05:00
Adrian 58ed9fdccc
Document urlparse_cached (#7777) 2026-07-24 20:08:58 +05:00
Adrian 41bb09741a
Use autodoc for contracts (#7775) 2026-07-24 15:35:54 +02:00
Adrian 0b578c1cbf
trackref: use autodoc (#7771) 2026-07-24 16:24:54 +05:00
Adrian abbc024bbe
Use autodoc for stats.rst (#7769) 2026-07-23 19:48:59 +05:00
Adrian 67e5282684
Use autodoc for exceptions and improve their docs (#7767) 2026-07-23 16:02:41 +05:00
Andrey Rakhmatullin 628a3afbbd
Remove deprecated xmliter(), deprecate re_rsearch() (#7765) 2026-07-23 10:38:56 +02:00
Andrey Rakhmatullin 56dee203e9
Use httpx2 as a preferred alternative to httpx. (#7762) 2026-07-23 10:29:22 +02:00
Adrian 1157b3e235
Add a middleware to detect response.meta copies (#7588) 2026-07-22 13:27:31 +02:00
Adrian a54c438da1
Provide extras for optional dependencies (#7596) 2026-07-22 10:44:26 +02:00
Andrey Rakhmatullin ca21306df7
Reactorless import hook improvements. (#7747) 
* Reactorless import hook improvements.

* Replace prints with logging.

* More tests.

* Uninstall the import hook earlier.

* Add pragma: no branch.
 2026-07-21 19:58:40 +05:00
Andrey Rakhmatullin 394c2797f3
Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
Andrey Rakhmatullin 36bf1185e5
Modernize code examples, drop docs for nonexistent `MEMDEBUG_NOTIFY` (#7737) 2026-07-21 12:22:04 +02:00
Andrey Rakhmatullin 1c5404dce5
Add docs about running Scrapy from apps and notebooks. (#7751) 2026-07-21 11:23:52 +02:00
Andrey Rakhmatullin a591d15c04
Deprecate Spider.log(). (#7739) 2026-07-20 14:59:55 +02:00
Andrey Rakhmatullin d8ba1571e7
Assorted docs fixes, part 2. (#7725) 
* Assorted docs fixes, part 2.

* Second pass.

* Address feedback.
 2026-07-14 00:49:44 +05:00
Adrian b3670369b8
Solve the feed Path issue (#7674) 
* Solve the feed Path issue

* Address additional, related issues
 2026-07-13 22:46:41 +05:00
Andrey Rakhmatullin c9446931a8
Use external mitmproxy. (#7720) 2026-07-09 17:02:05 +02:00
Andrey Rakhmatullin bdf3067935
Assorted docs fixes, part 1. (#7710) 2026-07-07 18:39:34 +05:00
Adrian c5ec881f1d
Update tools (#7724) 2026-07-07 17:27:26 +05:00
Adrian Chaves feb692f552 Bump version: 2.16.0 → 2.17.0 2026-07-07 12:26:52 +02:00
Adrian 9523e1ec8c
Merge commit from fork 2026-07-07 12:25:36 +02:00
Andrey Rakhmatullin 5ccc8dbe8a
Release notes for 2.17.0 (#7723) 
* Cover 2.17.0 in the release notes up to dd10cb8.

* Fix a copypasted issue number.

* Address minor issues

* Minor fixes

* Cover additional deprecations introduced while improving test coverage

* versionadded/versionchanged improvements

* Add a versionadded to the give_up_log_level docstring

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
 2026-07-07 14:57:42 +05:00
Adrian dd10cb8e9a
LxmlLinkExtractor: add deny_attrs and deny_tags (#7679) 2026-07-05 16:46:22 +05:00
Javier 4b2b56f384
Update on Request objects (#7286) 2026-06-29 15:34:12 +02:00
Adrian cf5607f8bc
Undeprecated the basic FormRequest API (#7671) 2026-06-26 17:35:46 +02:00
Adrian 1b940a75ac
Improve the item pipeline docs (#7676) 2026-06-26 16:18:22 +05:00
Adrian 7ab404c725
Add a documentation page about security (#7678) 2026-06-26 11:58:37 +02:00
Adrian 74e6b61071
Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 
* Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults

* Remove pointless comment

* Remove unnecessary mypy comments
 2026-06-25 16:44:51 +05:00
Adrian c690eac770
Document “logging settings” as special settings (#7668) 2026-06-25 16:34:37 +05:00
smellslikeml fb3455304d
Add content-based image filtering example (#4954) 2026-06-23 09:04:19 +02:00
Adrian f605defefc
Document scrapy-lint, remove start_url check (#7627) 
* Document scrapy-lint, remove start_url check

* Remove the offsite URL check in favor of scrapy-lint
 2026-06-22 20:12:16 +05:00
Adrian a8ffdcf851
Implement HTTP Auth settings and request metadata keys (#7590) 
* Implement HTTP Auth settings and request metadata keys

* Fix doc-tests

* Require the domain to be set when using settings
 2026-06-11 15:56:53 +05:00
Adrian c99f6e2209
Document the SPIDER_MODULES hack to lower startup time and memory usage (#7600) 2026-06-11 12:10:49 +02:00
Labib Bin Salam e0a7de7213
Document cb_kwargs/meta deep-copy when JOBDIR is set (#7573) 
When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.

Closes #6120
 2026-06-11 14:40:15 +05:00
Adrian e7f229f5b2
Clarify the first-class-citizen nature of asyncio support (#7599) 2026-06-11 10:51:46 +02:00
Andrey Rakhmatullin 4cb049cb15
Small docs fixes. (#7598) 2026-06-11 10:51:03 +02:00
Andrey Rakhmatullin beb6c51c17
Update default_settings, deprecate CRAWLSPIDER_FOLLOW_LINKS. (#7592) 2026-06-11 00:22:09 +05:00