RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,292 Commits 49 Branches 132 Tags 149 MiB
Commit Graph

2057 Commits

Author SHA1 Message Date
Adrian 4a69e48f0f
Log the first depth-limited link only (#7916) 2026-08-09 14:08:49 +05:00
Adrian 81d12c6eb8
Document the Referer caveat of DEFAULT_REQUEST_HEADERS (#7917) 2026-08-09 14:08:06 +05:00
Adrian 5b4888a0b1
Document that signal handler order is undefined (#7941) 2026-08-09 14:07:23 +05:00
Adrian 91b70e4db4
Improve the docs about scrapy parse --pipelines (#7876) 2026-08-04 17:30:51 +02:00
Adrian 6e2081ca41
Ask custom download handlers not to use engine.download_async() (#7871) 2026-08-04 14:41:25 +05:00
Adrian 54da6c88aa
Deprecate the download_delay spider attribute, and fix the suggested replacement for max_concurrent_requests (#7833) 
* Deprecate the download_delay and max_concurrent_requests spider attributes

* Fix the deprecation entry of max_concurrent_requests
 2026-08-03 23:46:41 +05:00
Adrian a9f1770306
Docs: sort and compact the component-settings list (#7862) 2026-08-03 13:03:28 +02:00
Adrian e83c709574
Docs: a job directory belongs to one Scrapy version (#7861) 2026-08-03 12:44:48 +02:00
Adrian 14478e3f24
Support CONCURRENT_REQUESTS = 0 for unlimited concurrency (#7840) 2026-07-31 20:19:07 +05:00
Adrian 1f03fbc17e
Add scrapy.utils.asyncio.sleep() (#7843) 2026-07-31 18:59:38 +05:00
Adrian 6cefaa5434
Add a stats reference (#7814) 2026-07-31 15:43:40 +05:00
Adrian 746bc7548d
Clarify crawl vs runspider in help and docs (#7832) 2026-07-31 12:17:34 +05:00
Adrian 37661508db
Add RobotParser.crawl_delay() and a robots_parsed signal (#7830) 
* Add RobotParser.crawl_delay() and a robots_parsed signal

* Improve test coverage
 2026-07-31 11:59:52 +05:00
Adrian 434fd1154a
Improve pre-crawler setting docs (#7835) 2026-07-31 11:38:33 +05:00
Adrian f02a99fe71
Add doc sections for callbacks and errbacks (#7821) 2026-07-30 20:15:15 +05:00
Adrian 433603e6ca
Add AWS_MAX_POOL_CONNECTIONS (#7794) 2026-07-30 15:51:45 +02:00
Adrian 98696efa80
Export item fields in declaration order (#7824) 2026-07-30 15:46:45 +05:00
Laerte Pereira 5b4828a012
docs(practices): Remove scrapoxy mention (#7817) 2026-07-30 08:02:54 +02:00
Adrian 0cbb20e8e8
Treat broken cache records as cache misses (#7805) 2026-07-29 14:39:27 +05:00
Youssef Mohamed bc5b5fb1f6
Add Request.to_curl() (#7743) (#7802) 2026-07-29 13:47:15 +05:00
Adrian 58ed9fdccc
Document urlparse_cached (#7777) 2026-07-24 20:08:58 +05:00
Adrian 41bb09741a
Use autodoc for contracts (#7775) 2026-07-24 15:35:54 +02:00
Adrian 0b578c1cbf
trackref: use autodoc (#7771) 2026-07-24 16:24:54 +05:00
Adrian abbc024bbe
Use autodoc for stats.rst (#7769) 2026-07-23 19:48:59 +05:00
Adrian 67e5282684
Use autodoc for exceptions and improve their docs (#7767) 2026-07-23 16:02:41 +05:00
Andrey Rakhmatullin 56dee203e9
Use httpx2 as a preferred alternative to httpx. (#7762) 2026-07-23 10:29:22 +02:00
Adrian 1157b3e235
Add a middleware to detect response.meta copies (#7588) 2026-07-22 13:27:31 +02:00
Adrian a54c438da1
Provide extras for optional dependencies (#7596) 2026-07-22 10:44:26 +02:00
Andrey Rakhmatullin ca21306df7
Reactorless import hook improvements. (#7747) 
* Reactorless import hook improvements.

* Replace prints with logging.

* More tests.

* Uninstall the import hook earlier.

* Add pragma: no branch.
 2026-07-21 19:58:40 +05:00
Andrey Rakhmatullin 394c2797f3
Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
Andrey Rakhmatullin 36bf1185e5
Modernize code examples, drop docs for nonexistent `MEMDEBUG_NOTIFY` (#7737) 2026-07-21 12:22:04 +02:00
Andrey Rakhmatullin 1c5404dce5
Add docs about running Scrapy from apps and notebooks. (#7751) 2026-07-21 11:23:52 +02:00
Andrey Rakhmatullin a591d15c04
Deprecate Spider.log(). (#7739) 2026-07-20 14:59:55 +02:00
Andrey Rakhmatullin d8ba1571e7
Assorted docs fixes, part 2. (#7725) 
* Assorted docs fixes, part 2.

* Second pass.

* Address feedback.
 2026-07-14 00:49:44 +05:00
Adrian b3670369b8
Solve the feed Path issue (#7674) 
* Solve the feed Path issue

* Address additional, related issues
 2026-07-13 22:46:41 +05:00
Andrey Rakhmatullin bdf3067935
Assorted docs fixes, part 1. (#7710) 2026-07-07 18:39:34 +05:00
Adrian c5ec881f1d
Update tools (#7724) 2026-07-07 17:27:26 +05:00
Andrey Rakhmatullin 5ccc8dbe8a
Release notes for 2.17.0 (#7723) 
* Cover 2.17.0 in the release notes up to dd10cb8.

* Fix a copypasted issue number.

* Address minor issues

* Minor fixes

* Cover additional deprecations introduced while improving test coverage

* versionadded/versionchanged improvements

* Add a versionadded to the give_up_log_level docstring

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
 2026-07-07 14:57:42 +05:00
Adrian dd10cb8e9a
LxmlLinkExtractor: add deny_attrs and deny_tags (#7679) 2026-07-05 16:46:22 +05:00
Javier 4b2b56f384
Update on Request objects (#7286) 2026-06-29 15:34:12 +02:00
Adrian cf5607f8bc
Undeprecated the basic FormRequest API (#7671) 2026-06-26 17:35:46 +02:00
Adrian 1b940a75ac
Improve the item pipeline docs (#7676) 2026-06-26 16:18:22 +05:00
Adrian 7ab404c725
Add a documentation page about security (#7678) 2026-06-26 11:58:37 +02:00
Adrian 74e6b61071
Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 
* Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults

* Remove pointless comment

* Remove unnecessary mypy comments
 2026-06-25 16:44:51 +05:00
Adrian c690eac770
Document “logging settings” as special settings (#7668) 2026-06-25 16:34:37 +05:00
smellslikeml fb3455304d
Add content-based image filtering example (#4954) 2026-06-23 09:04:19 +02:00
Adrian f605defefc
Document scrapy-lint, remove start_url check (#7627) 
* Document scrapy-lint, remove start_url check

* Remove the offsite URL check in favor of scrapy-lint
 2026-06-22 20:12:16 +05:00
Adrian a8ffdcf851
Implement HTTP Auth settings and request metadata keys (#7590) 
* Implement HTTP Auth settings and request metadata keys

* Fix doc-tests

* Require the domain to be set when using settings
 2026-06-11 15:56:53 +05:00
Adrian c99f6e2209
Document the SPIDER_MODULES hack to lower startup time and memory usage (#7600) 2026-06-11 12:10:49 +02:00
Labib Bin Salam e0a7de7213
Document cb_kwargs/meta deep-copy when JOBDIR is set (#7573) 
When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.

Closes #6120
 2026-06-11 14:40:15 +05:00