RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,152 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

9152 Commits

Author SHA1 Message Date
Michel Ace 07d20a8ce4
Fix test_rel_has_nofollow test 2021-08-10 21:21:43 +02:00
Michel Ace 18b6f30a73
Add test for rel_has_nofollow 2021-08-10 21:13:50 +02:00
Michel Ace 1ba0f68483
Allow comma-separated values in the rel tag 
Comma-separated `rel` values are often seen in the wild, because Google allows it (see https://developers.google.com/search/docs/advanced/guidelines/qualify-outbound-links).
 2021-08-10 17:09:37 +02:00
Aaron Tan 954f303590
Update docs/topics/settings.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-08-09 22:23:23 +10:00
Aaron Tan 48eff4ee8f Remove JOBDIR from default settings 2021-08-08 20:52:14 +10:00
Aaron Tan 8e7d2ef133 Document JOBDIR option issue #5173 
Add JOBDIR setting to the settings page.
Add default JOBDIR setting to global defaults in scrapy.settings.default_settings module.
 2021-08-07 11:44:12 +10:00
Mannan2812 94baa4b272
Fix FileFeedStoragePreFeedOptionsTest fails in CI/CD pipeline (#5198) 2021-08-05 21:23:11 +02:00
Andrey Rakhmatullin 93bf1ae7e3 Fix tests for the 3.10 TypeError message change. 2021-08-03 20:16:59 +05:00
Andrey Rakhmatullin ef6fb933b5 Fix a Python 3.10 logging issue. 2021-08-03 20:16:59 +05:00
Andrey Rakhmatullin 2bf2f9d6db Add Python 3.10b4 tests on Ubuntu. 2021-08-03 20:16:59 +05:00
Andrey Rahmatullin 4d1ecc31c9
Merge pull request #5217 from aaron-tan/docs-extension 2021-08-02 17:36:35 +05:00
Aaron Tan 880a4d9493
Update docs/topics/extensions.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-08-01 11:02:27 +10:00
Andrey Rahmatullin b406affd5b
Merge pull request #5210 from laggardkernel/feature/aws-session-token 2021-07-30 15:15:55 +05:00
Eugenio Lacuesta cc89f6be38
Response.attributes (#5218) 2021-07-29 22:12:44 +02:00
Andrey Rahmatullin 22bd01237b
Merge pull request #4613 from elacuesta/docs-headless-browser 
Update headless browser docs
 2021-07-29 15:25:08 +05:00
Eugenio Lacuesta 4b62ac6c3a
Update headless browser docs to mention playwright 2021-07-28 15:00:24 -03:00
Eugenio Lacuesta 9aef9b78eb
Merge remote-tracking branch 'upstream/master' into docs-headless-browser 2021-07-28 14:31:49 -03:00
Eugenio Lacuesta 0e3d50dd18
Update docs/topics/dynamic-content.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-28 14:30:16 -03:00
Eugenio Lacuesta 494e0ad8ff
Update docs/topics/dynamic-content.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-28 14:29:50 -03:00
Aaron Tan d55b6fcad6 Fix for duplicate object description error 2021-07-28 12:10:34 +10:00
Aaron Tan 714aa3970e
Merge branch 'scrapy:master' into docs-extension 2021-07-30 20:36:58 +10:00
laggardkernel 8e7b96d8a2 Tweak doc for setting AWS_SESSION_TOKEN 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-27 19:40:02 +08:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
laggardkernel abe0b37d30
Cleanup leftover boto2 code in S3DownloaderHandler (#5209) 
S3DownloaderHandler.conn is a leftover attribute from 5e99758.
 2021-07-27 11:11:32 +02:00
Rob Banagale b22a004398
Document media pipeline file naming (#5152) 2021-07-26 20:51:32 +02:00
Aaron Tan 209c1fce02
Reference MailSender in StatsMailer 
Added a reference to MailSender in the StatsMailer extension description and included a link to the document detailing how to instantiate MailSender and using Scrapy settings objects.
 2021-07-24 14:50:48 +10:00
Mikhail Korobov ff272d6332
Merge pull request #5215 from pcorpet/types 
Typing: switch to a newer version of MyPy to check types
 2021-07-22 17:52:48 +05:00
Pascal Corpet 70dddfe2b2 Typing: switch to a newer version of MyPy to check types 2021-07-21 17:10:10 +02:00
Andrey Rahmatullin ee2df97bbd
Pin the libxml2 version in CI as a newer one breaks lxml (#5208) 2021-07-16 14:28:32 +02:00
Andrey Rahmatullin 89b654b82d
Make the pylint test pass (#5207) 
Co-authored-by: Vostretsov Nikita <whalebot.helmsman@gmail.com>
 2021-07-16 12:18:14 +02:00
Eugenio Lacuesta bcce066057
Update ItemFilter (#5203) 2021-07-14 17:56:07 +02:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Türkalp Burak KAYRANCIOĞLU fcc6becc58
S3FeedStorage: allow custom endpoint (#4998) 
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2021-07-14 10:00:43 +02:00
D R Siddhartha 4ddc9d6b55
Feeds: Item Filters (#5178) 2021-07-13 17:22:29 +02:00
Eugenio Lacuesta c062ed017a
[CI] fail-fast: false (#5200) 2021-07-12 18:34:22 +02:00
Andrey Rahmatullin 1c46d5aa93
Merge pull request #5191 from ivanprado/master 
CloseSpider can be raised on spider_idle signal handler to set the closing reason
 2021-07-12 18:46:40 +05:00
Eugenio Lacuesta cb08e3644b
Remove trailing whitespaces 2021-07-08 09:22:21 -03:00
Iván de Prado eca641aa3d
Update tests/test_engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:40:20 +01:00
Iván de Prado 6b8f694653
Update scrapy/core/engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:40:02 +01:00
Iván de Prado 7597d860c8
Update scrapy/core/engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:39:17 +01:00
Andrey Rahmatullin 15dfc8a76f
Merge pull request #5194 from scrapy/doc-spiders-warning-indentation-1 
spiders.rst: indent warnings into class descriptions
 2021-06-29 17:54:22 +05:00
Iván de Prado Alonso e94d3ac173 Expanded doc for idle signal 2021-06-29 13:40:43 +01:00
Iván de Prado Alonso f35970778b Test case for raising CloseSpider on spider idle signal handler 2021-06-29 13:21:38 +01:00
Adrián Chaves 73ff9ffd64
spiders.rst: indent warnings into class descriptions 2021-06-26 08:58:29 +02:00
Iván de Prado Alonso ce445f2046 Fix typing 2021-06-24 09:56:05 +01:00
Iván de Prado Alonso 812b4bb518 CloseSpider can be raised on spider_idle signal handler 2021-06-23 17:22:01 +01:00
Andrey Rahmatullin 016c7e92d1
Merge pull request #3579 from maramsumanth/duplicate_key 
[MRG+1] Issue #2919: Fix FormRequest.formdata with GET method duplicates same key in query string
 2021-06-15 13:36:53 +05:00
Andrey Rahmatullin af975a5b0a
Merge pull request #4649 from ajaymittur28/http-proxy-schema 
Support schema-less HTTP proxy (#4504)
 2021-06-15 13:36:17 +05:00
ajaymittur28 eb324fadad Merge branch 'http-proxy-schema' of https://github.com/ajaymittur28/scrapy into http-proxy-schema 2021-06-14 21:39:43 +05:30
ajaymittur28 7d653288e3 Update unittest 2021-06-14 21:39:18 +05:30