RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 134 MiB
Commit Graph

2770 Commits

Author SHA1 Message Date
Laerte Pereira 144ff6c756 Document missing parts of response.json method 2024-06-05 21:09:10 -03:00
Laerte Pereira 62c89aaf05
Update docs/topics/practices.rst 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-05-23 07:00:45 -03:00
Laerte Pereira 8ec67ca230
Update docs/topics/practices.rst 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-05-23 07:00:35 -03:00
Laerte Pereira 17e623cf0c
Update docs/topics/practices.rst 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-05-23 07:00:24 -03:00
Laerte Pereira 3f66b66e3f fix: checks 2024-05-22 22:01:55 -03:00
Laerte Pereira dc6a495fee
Update docs/topics/practices.rst 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-05-22 18:51:02 -03:00
Laerte Pereira 8210fae25a
Update docs/topics/practices.rst 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-05-22 18:50:50 -03:00
Laerte Pereira e676cd3ce0 docs: Remove top-level reactor imports from CrawlerProces/CrawlerRunner examples 2024-05-22 07:55:53 -03:00
Adrián Chaves 812fd2368f
Allow user-defined secure cookies (#6357) 2024-05-15 11:48:43 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Sanchay Kumar c9ef520936
Add Downloader.get_slot_key() without a spider parameter (#6352) 2024-05-10 14:26:45 +02:00
Adrián Chaves fe163d98ea
Fix typo 2024-05-08 20:41:20 +02:00
aisha-partha 2e13a9b8e1 Update METAREFRESH_IGNORE_TAGS default value in documentation 2024-05-08 23:33:13 +05:30
aisha-partha 3590a1f66b Closes #6342. Update documentation on METAREFRESH_IGNORE_TAGS default value change 2024-05-08 23:23:17 +05:30
aisha-partha 180bc9bad7 Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default 2024-05-08 22:06:46 +05:30
Andrey Rakhmatullin a166e97399
Remove the auto-generated copyright years from the docs footer. (#6322) 2024-04-29 09:39:22 +02:00
TechVest 5f67c01d1d
chore: fix some typos in comments (#6317) 
Signed-off-by: TechVest <techdashen@qq.com>
 2024-04-17 10:56:26 +02:00
Adrián Chaves 1d11ea3a54
Update practices.rst 2024-04-11 12:19:32 +02:00
Adrián Chaves d7581c6b41
Allow disabling the AutoThrottle extension for a given slot (#6246) 2024-03-12 09:44:29 +01:00
Andrey Rakhmatullin 642af40704
Full typing for scrapy/extensions, part 2. (#6279) 2024-03-11 10:09:09 +01:00
Andrey Rakhmatullin 4f9dd998dc Bump pylint, cleanup the ignored tags. 2024-02-28 14:31:21 +05:00
Andrey Rakhmatullin 282767f23b Bump black. 2024-02-28 14:30:38 +05:00
Adrián Chaves 2d46b4acf5
Complete coverage for the AutoThrottle extension (#6245) 2024-02-27 09:28:02 +01:00
Adrián Chaves 6fc7827042
Do not close the underlying file from compression plugins (#6239) 2024-02-21 14:27:42 +01:00
Laerte Pereira ee1189512f
Replace urlparse with urlparse_cached where possible (#6229) 2024-02-20 12:47:29 +01:00
Jalil SA 36f72877ba update: docs/topics/selectors.rst 2024-02-16 10:39:16 -06:00
Jalil SA 3e7b704c08 update: docs/topics/selectors.rst 2024-02-15 16:59:32 -06:00
Jalil SA 660e3b1953 update: docs/topics/items.rst 2024-02-15 16:59:31 -06:00
Adrián Chaves 6bd45bb6d9 Merge remote-tracking branch 'scrapy/2.11' into 2.11-port 2024-02-15 06:39:50 +01:00
Adrián Chaves 6b88b3346c Set the release date of versions 2.11.1 and 1.8.4 2024-02-14 18:16:40 +01:00
Adrián Chaves 479619b340 Merge branch '2.11-redos' into 2.11 2024-02-14 17:18:26 +01:00
Adrián Chaves 809bfac489 Merge branch '2.11-compression-bomb' into 2.11 2024-02-14 17:16:25 +01:00
Adrián Chaves 5bcb8fd501 Merge branch '2.11-authorization' into 2.11 2024-02-14 17:14:29 +01:00
Andrey Rakhmatullin a55e933c11
Release notes for 2.11.1 (#6150) 2024-02-14 17:08:40 +01:00
Andy 1c9d308acc
Cover the deprecation and removal of response_httprepr in the release notes (#6216) 2024-02-05 12:52:01 +01:00
Laerte Pereira 24634f1bb2 Attend PR comments 2024-01-31 12:29:43 -03:00
Laerte Pereira bacaf0db7a Update documentation 2024-01-31 10:14:48 -03:00
Kevin Toms 9074c16497 make suggestion 2024-01-18 09:36:25 -05:00
Kevin Toms 46f94ec9cb Fix test 
Wrap the yield line in a function to prevent throwing error when the code snippet is executed
 2024-01-17 15:49:51 -05:00
Kevin Toms 88285e75b6 Add FAQ on making a blank request 2024-01-17 10:05:22 -05:00
Rotzbua d5233bb57f
chore(docs): update `sphinx` dependencies (#6200) 2024-01-15 14:11:33 +01:00
Adrián Chaves 6127f7d278 Update quotes.toscrape.com page copies (#6190) 2024-01-01 22:58:34 +04:00
Chan Sau Yee badc7c5be9 Update black reference in docs (#6192) 2024-01-01 22:58:21 +04:00
Chan Sau Yee 1902284942
Update black reference in docs (#6192) 2023-12-29 12:32:51 +01:00
Adrián Chaves 34e01a8a93
Update quotes.toscrape.com page copies (#6190) 2023-12-28 12:25:01 +01:00
Jalil SA d25cfe5315
Add JsonResponse (#6174) 2023-12-21 10:36:21 +01:00
Adrián Chaves 48a9a58ff2 Link to Zyte’s export guides 2023-12-20 21:43:21 +04:00
Adrián Chaves 369712ee50 SPM → Zyte API (#6163) 2023-12-20 21:43:13 +04:00
Adrián Chaves b095dd218f Extend Request.meta documentation (#5565) 2023-12-20 21:43:03 +04:00
Adrián Chaves 1864f48e9e Link to Zyte’s export guides 2023-12-20 12:47:34 +01:00
Adrián Chaves bc138ef8e9 Minor release notes fix 2023-12-15 12:24:04 +01:00
Adrián Chaves ce9d290eff Remove the lxml version check for huge_tree on xmliter_lxml 
iterparse supports the option since lxml 2.2.1, it was the HTML parser that only got it in 4.2
 2023-12-15 12:23:26 +01:00
Adrián Chaves cd6846b763 Merge remote-tracking branch 'scrapy/2.11' into 2.11-redos 2023-12-15 11:52:35 +01:00
Adrián Chaves 150d96764b Deprecate xmliter in favor of xmliter_lxml 2023-12-15 11:42:55 +01:00
Adrián Chaves 12b10a7a64 Cover scrapy.downloadermiddlewares.decompression in the release notes 2023-12-13 13:35:05 +01:00
Adrián Chaves 70ba3a0868
SPM → Zyte API (#6163) 2023-11-30 11:01:22 +01:00
Adrián Chaves 731f749556
Extend Request.meta documentation (#5565) 2023-11-30 10:54:09 +01:00
Adrián Chaves 40b3efbbee Remove open_in_browser from the 1.8.4 release notes 2023-11-29 12:47:04 +01:00
Adrián Chaves eb8b2c5197 Mention open_in_browser in the release notes 2023-11-29 12:13:04 +01:00
Adrián Chaves c947f51077 Set an arbitrary upper limit on ReDoS-vulnerable regexps 2023-11-29 11:54:08 +01:00
Adrián Chaves b53ed52a22 Update the release notes 2023-11-23 11:36:45 +01:00
Adrián Chaves 1087bb7b2e Update the docs 2023-11-23 09:11:14 +01:00
Adrián Chaves 1c4e93293a Merge remote-tracking branch 'scrapy/2.11' into 2.11-authorization 2023-11-17 15:42:08 +01:00
Jessica Allman-LaPorte 150f9d6d88 Make shell switching more clear in the tutorial (#6128) 2023-11-16 19:17:29 +04:00
Andrey Rakhmatullin ffbf943e9d Merge pull request #6077 from 11-aryan/11-aryan 2023-11-16 19:13:02 +04:00
Andrey Rakhmatullin 75e99c75b3 Improve the docs about Crawler attributes and settings initialization. 2023-11-16 12:36:57 +04:00
Andrey Rakhmatullin 603aa4924a Improve the docs about Crawler attributes and settings initialization. 2023-11-15 19:51:04 +04:00
Adrián Chaves 080fecd890 Drop the Authorization header on cross-domain redirect 2023-11-15 15:39:30 +01:00
Andrey Rakhmatullin 5fccf370b8 Update the RTD URL for coverage 2023-11-15 15:38:13 +01:00
Jessica Allman-LaPorte 6587556af9
Make shell switching more clear in the tutorial (#6128) 2023-11-03 10:02:18 +01:00
nihilisticneuralnet 04024f1e79
[Solved] JOBDIR= None for when Scheduler initializes disk queue even if JOBDIR is empty string (#6124) 
Co-authored-by: John Doe <johndoe@email.com>
 2023-10-31 17:37:26 +04:00
Andrey Rakhmatullin 39ee8d1ee2
Deprecate ReppyRobotParser (#6099) 2023-10-18 09:10:21 +02:00
Sandesh Pyakurel 064256b059
Typo fixed in news.rst file (#6094) 2023-10-17 18:39:39 +02:00
Andrey Rakhmatullin c65567988d
Change supported PyPy versions to 3.9 and 3.10. (#6087) 
* Change support PyPy versions to 3.9 and 3.10.

* Update the RTD URL for coverage.

* Move pypy3-pinned back to pypy3.8.
 2023-10-06 11:27:02 +02:00
Andrey Rakhmatullin a96989c6f3
Merge pull request #6077 from 11-aryan/11-aryan 2023-10-02 20:40:25 +04:00
Adrián Chaves 42adbb2104
Update docs/topics/request-response.rst 2023-10-02 14:43:30 +02:00
kokobhara 8dc72dfc4d Cover PythonItemExporter backwaird-incompatible changes in 2.11 (#6081) 2023-10-02 14:48:44 +04:00
kokobhara ef1ed4fab7
Cover PythonItemExporter backwaird-incompatible changes in 2.11 (#6081) 2023-10-02 12:14:05 +02:00
Aryan e146c3a2fc removed the entire example for dont_merge_cookies 2023-10-02 15:36:29 +05:30
Aryan fe5ef0a80a Fixed the dont_merge_cookies example in docs 2023-10-01 11:47:51 +05:30
Yash nagarkar 197781e3af Cover the removal of is_botocore on the release notes (#6061) 2023-09-24 18:32:35 +04:00
Yash nagarkar 908da8ba82
Cover the removal of is_botocore on the release notes (#6061) 2023-09-22 10:12:20 +02:00
Andrey Rakhmatullin f2fb4760d2 Bump version: 2.10.1 → 2.11.0 2023-09-18 15:24:28 +04:00
Andrey Rakhmatullin 528911da85 Fix/reword the release notes. 2023-09-18 14:35:28 +04:00
Andrey Rakhmatullin 2fa768399a Replace the VERSION vars. 2023-09-15 19:19:42 +04:00
Andrey Rakhmatullin c2346b4a95 Update the release notes up to current master. 2023-09-15 19:15:05 +04:00
Andrey Rakhmatullin 14c27d2215 Merge branch 'master' into relnotes-2.11 2023-09-14 16:38:15 +04:00
Andrey Rakhmatullin f96a3ed5f0 Cover up to cddb8c15d in the release notes. 2023-09-13 20:46:55 +04:00
Andrey Rakhmatullin 8dd48a08e4 Move PeriodicLog docs from Debugging to General purpose. 2023-09-13 20:46:32 +04:00
Andrey Rakhmatullin ac201d310b Small improvements, 2023-09-13 16:17:52 +04:00
Andrey Rakhmatullin 61ef37a594 Docs improvements. 2023-09-13 15:47:41 +04:00
Andrey Rakhmatullin 61e6bfc023 Docs improvements. 2023-09-13 12:35:15 +04:00
Andrey Rakhmatullin a3f22046ef Document changing settings in Spider.from_crawler(). 2023-09-12 20:41:36 +04:00
Andrey Rakhmatullin d3b5c9be97 Deprecate running Crawler.crawl() twice. 2023-09-06 19:29:44 +04:00
Andrey Rakhmatullin d9efb94b61 Merge branch '2.10' 2023-08-31 13:21:22 +04:00
Mikhail Korobov 0fcb0554d1
Merge pull request #6016 from jxlil/fix/response.json 
Fix: response.json() call makes unnecessary memory allocation
 2023-08-31 14:07:24 +05:00
Andrey Rakhmatullin a320e5f6a4 Bump version: 2.10.0 → 2.10.1 2023-08-30 12:08:36 +04:00
Andrey Rakhmatullin f8f550120b Release notes for 2.10.1. 2023-08-29 18:07:15 +04:00
Jalil SA 4dd32672ed added entry to Backward-incompatible changes 2023-08-21 08:28:08 -06:00
Georgiy Zatserklianyi fe02642980 Merge remote-tracking branch 'origin/periodic_log_2' into periodic_log_2 2023-08-21 17:15:42 +03:00