RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,244 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

2995 Commits

Author SHA1 Message Date
Adrian Chaves 370ff74dc9 WIP 2026-07-02 15:02:13 +02:00
Adrian Chaves 27f239074c WIP 2026-07-02 10:05:47 +02:00
Adrian Chaves 8f1f66e1eb Address backward compatibility issues, remove CONCURRENT_REQUESTS_PER_IP 2026-07-02 09:28:01 +02:00
Adrian Chaves 7784594f7a Rederepcate PER_IP, deprecate PER_DOMAIN 2026-07-02 08:27:20 +02:00
Adrian Chaves 8e488d837c Clarify backoff per-scope overrides 2026-07-02 06:59:52 +02:00
Adrian Chaves e203c326c2 WIP 2026-07-02 06:48:46 +02:00
Adrian Chaves c21f8dcfc5 WIP 2026-07-01 14:51:10 +02:00
Adrian Chaves 61c9cc33ff WIP 2026-07-01 13:23:00 +02:00
Adrian Chaves 7ae19cc2a2 WIP 2026-07-01 12:26:53 +02:00
Adrian Chaves 074f9e567d WIP 2026-07-01 10:19:27 +02:00
Adrian Chaves ce35abbce7 WIP 2026-07-01 09:14:10 +02:00
Adrian Chaves f87717e05a WIP 2026-06-30 16:45:58 +02:00
Adrian Chaves b3ea60354f WIP 2026-06-30 16:28:21 +02:00
Adrian Chaves 0e7c89a6a2 WIP 2026-06-30 15:33:10 +02:00
Adrian Chaves 3ae5ab40c1 WIP 2026-06-30 13:32:57 +02:00
Adrian Chaves ba8bb9e4e9 WIP 2026-06-30 12:53:18 +02:00
Adrian Chaves e0cc7ed7db WIP 2026-06-30 11:33:37 +02:00
Adrian Chaves 80f161ef3e Merge remote-tracking branch 'origin/master' into throttling 2026-06-30 08:23:04 +02:00
Adrian Chaves 8dfb548415 WIP 2026-06-29 16:29:31 +02:00
Javier 4b2b56f384
Update on Request objects (#7286) 2026-06-29 15:34:12 +02:00
Adrian Chaves 8834250059 WIP 2026-06-26 22:53:11 +02:00
Adrian Chaves fb031dee80 Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 19:01:20 +02:00
Adrian Chaves 5aae8334a5 Improve request and scope delay support 2026-06-26 19:00:06 +02:00
Adrian cf5607f8bc
Undeprecated the basic FormRequest API (#7671) 2026-06-26 17:35:46 +02:00
Adrian Chaves aeb6b4763c Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 14:47:23 +02:00
Adrian 1b940a75ac
Improve the item pipeline docs (#7676) 2026-06-26 16:18:22 +05:00
Adrian Chaves 627cecc93e Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 12:17:22 +02:00
Adrian Chaves 6ce90dde3e Minimum queuelib: 1.4.2 → 1.6.1 2026-06-26 12:15:52 +02:00
Adrian 7ab404c725
Add a documentation page about security (#7678) 2026-06-26 11:58:37 +02:00
Adrian Chaves 3e02b9e5ab Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 11:32:43 +02:00
Adrian Chaves 8190b5b703 Add a throttling-aware scheduler 2026-06-26 10:44:31 +02:00
Adrian Chaves ec18ddc239 Initial implementation draft 2026-06-25 16:01:47 +02:00
Adrian 74e6b61071
Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 
* Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults

* Remove pointless comment

* Remove unnecessary mypy comments
 2026-06-25 16:44:51 +05:00
Adrian Chaves 4af625ac1b Merge remote-tracking branch 'origin/master' into throttling 2026-06-25 13:39:28 +02:00
Adrian Chaves 7fdee93e42 WIP 2026-06-25 13:38:55 +02:00
Adrian c690eac770
Document “logging settings” as special settings (#7668) 2026-06-25 16:34:37 +05:00
Adrian Chaves c93e430ccb Complete docs 2026-06-24 13:35:07 +02:00
Adrian Chaves d18085dc3e Update docs 2026-06-24 11:49:51 +02:00
Adrian Chaves d8d519367d Fix docs/requirements.txt 2026-06-24 10:57:56 +02:00
Adrian Chaves e796beb6fa Merge remote-tracking branch 'origin/master' into throttling 2026-06-24 10:27:28 +02:00
smellslikeml fb3455304d
Add content-based image filtering example (#4954) 2026-06-23 09:04:19 +02:00
Adrian f605defefc
Document scrapy-lint, remove start_url check (#7627) 
* Document scrapy-lint, remove start_url check

* Remove the offsite URL check in favor of scrapy-lint
 2026-06-22 20:12:16 +05:00
Adrian Chaves b21e654800 Align default values in docs with the main branch 2026-06-15 14:10:54 +02:00
Adrian a8ffdcf851
Implement HTTP Auth settings and request metadata keys (#7590) 
* Implement HTTP Auth settings and request metadata keys

* Fix doc-tests

* Require the domain to be set when using settings
 2026-06-11 15:56:53 +05:00
Adrian c99f6e2209
Document the SPIDER_MODULES hack to lower startup time and memory usage (#7600) 2026-06-11 12:10:49 +02:00
Labib Bin Salam e0a7de7213
Document cb_kwargs/meta deep-copy when JOBDIR is set (#7573) 
When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.

Closes #6120
 2026-06-11 14:40:15 +05:00
Adrian e7f229f5b2
Clarify the first-class-citizen nature of asyncio support (#7599) 2026-06-11 10:51:46 +02:00
Andrey Rakhmatullin 4cb049cb15
Small docs fixes. (#7598) 2026-06-11 10:51:03 +02:00
Andrey Rakhmatullin beb6c51c17
Update default_settings, deprecate CRAWLSPIDER_FOLLOW_LINKS. (#7592) 2026-06-11 00:22:09 +05:00
Andrey Rakhmatullin 4e956bd2de
Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs (#7575) 
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.

* Update the docs.

* Trim the tables.

* Restore lost wording.

* Handlers docs improvements and fixes.
 2026-06-09 01:10:44 +05:00