RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

10710 Commits

Author SHA1 Message Date
Andrey Rahmatullin 0915be7300
Merge pull request #5228 from wRAR/test-reqs 
Rename tests/requirements-py3.txt to tests/requirements.txt.
 2021-08-11 21:04:10 +05:00
Andrey Rakhmatullin b63369c148 Rename tests/requirements-py3.txt to tests/requirements.txt. 2021-08-11 20:02:45 +05:00
Andrey Rahmatullin 0590c3756c
Merge pull request #5221 from wRAR/3.10-tests 
Fixing tests for upcoming 3.10.
 2021-08-11 14:19:51 +05:00
Andrey Rakhmatullin 74cee38a4e Don't run the asyncio tests on 3.9. 2021-08-11 14:19:08 +05:00
Andrey Rahmatullin 902fce0640
Merge pull request #5225 from divtiply/patch-3 
Allow comma-separated values in the rel attribute
 2021-08-11 10:58:28 +05:00
Andrey Rakhmatullin 983b89ad4f Fix SpiderLoaderTest on Python 3.10. 2021-08-11 10:39:23 +05:00
Michel Ace ce9d6c658b
Add more rel_has_nofollow tests 2021-08-10 22:21:51 +02:00
Michel Ace 295f0e2bf5
Make flake8 happy 2021-08-10 21:38:29 +02:00
Michel Ace 07d20a8ce4
Fix test_rel_has_nofollow test 2021-08-10 21:21:43 +02:00
Michel Ace 18b6f30a73
Add test for rel_has_nofollow 2021-08-10 21:13:50 +02:00
Michel Ace 1ba0f68483
Allow comma-separated values in the rel tag 
Comma-separated `rel` values are often seen in the wild, because Google allows it (see https://developers.google.com/search/docs/advanced/guidelines/qualify-outbound-links).
 2021-08-10 17:09:37 +02:00
Aaron Tan 954f303590
Update docs/topics/settings.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-08-09 22:23:23 +10:00
Aaron Tan 48eff4ee8f Remove JOBDIR from default settings 2021-08-08 20:52:14 +10:00
Aaron Tan 8e7d2ef133 Document JOBDIR option issue #5173 
Add JOBDIR setting to the settings page.
Add default JOBDIR setting to global defaults in scrapy.settings.default_settings module.
 2021-08-07 11:44:12 +10:00
Mannan2812 94baa4b272
Fix FileFeedStoragePreFeedOptionsTest fails in CI/CD pipeline (#5198) 2021-08-05 21:23:11 +02:00
Andrey Rakhmatullin 93bf1ae7e3 Fix tests for the 3.10 TypeError message change. 2021-08-03 20:16:59 +05:00
Andrey Rakhmatullin ef6fb933b5 Fix a Python 3.10 logging issue. 2021-08-03 20:16:59 +05:00
Andrey Rakhmatullin 2bf2f9d6db Add Python 3.10b4 tests on Ubuntu. 2021-08-03 20:16:59 +05:00
Andrey Rahmatullin 4d1ecc31c9
Merge pull request #5217 from aaron-tan/docs-extension 2021-08-02 17:36:35 +05:00
Aaron Tan 880a4d9493
Update docs/topics/extensions.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-08-01 11:02:27 +10:00
Andrey Rahmatullin b406affd5b
Merge pull request #5210 from laggardkernel/feature/aws-session-token 2021-07-30 15:15:55 +05:00
Eugenio Lacuesta cc89f6be38
Response.attributes (#5218) 2021-07-29 22:12:44 +02:00
Andrey Rahmatullin 22bd01237b
Merge pull request #4613 from elacuesta/docs-headless-browser 
Update headless browser docs
 2021-07-29 15:25:08 +05:00
Eugenio Lacuesta 4b62ac6c3a
Update headless browser docs to mention playwright 2021-07-28 15:00:24 -03:00
Eugenio Lacuesta 9aef9b78eb
Merge remote-tracking branch 'upstream/master' into docs-headless-browser 2021-07-28 14:31:49 -03:00
Eugenio Lacuesta 0e3d50dd18
Update docs/topics/dynamic-content.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-28 14:30:16 -03:00
Eugenio Lacuesta 494e0ad8ff
Update docs/topics/dynamic-content.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-28 14:29:50 -03:00
Aaron Tan d55b6fcad6 Fix for duplicate object description error 2021-07-28 12:10:34 +10:00
Aaron Tan 714aa3970e
Merge branch 'scrapy:master' into docs-extension 2021-07-30 20:36:58 +10:00
laggardkernel 8e7b96d8a2 Tweak doc for setting AWS_SESSION_TOKEN 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-27 19:40:02 +08:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
laggardkernel abe0b37d30
Cleanup leftover boto2 code in S3DownloaderHandler (#5209) 
S3DownloaderHandler.conn is a leftover attribute from 5e99758.
 2021-07-27 11:11:32 +02:00
Rob Banagale b22a004398
Document media pipeline file naming (#5152) 2021-07-26 20:51:32 +02:00
Aaron Tan 209c1fce02
Reference MailSender in StatsMailer 
Added a reference to MailSender in the StatsMailer extension description and included a link to the document detailing how to instantiate MailSender and using Scrapy settings objects.
 2021-07-24 14:50:48 +10:00
Mikhail Korobov ff272d6332
Merge pull request #5215 from pcorpet/types 
Typing: switch to a newer version of MyPy to check types
 2021-07-22 17:52:48 +05:00
Pascal Corpet 70dddfe2b2 Typing: switch to a newer version of MyPy to check types 2021-07-21 17:10:10 +02:00
Andrey Rakhmatullin 7306a81188 Disable builtin middlewares in spider middleware tests. 2021-07-19 20:09:11 +05:00
Adrián Chaves 8a2768561f
Merge branch 'master' into asyncio-parse-asyncgen-proper-rebased 2021-07-16 14:32:15 +02:00
Andrey Rahmatullin ee2df97bbd
Pin the libxml2 version in CI as a newer one breaks lxml (#5208) 2021-07-16 14:28:32 +02:00
Andrey Rahmatullin 89b654b82d
Make the pylint test pass (#5207) 
Co-authored-by: Vostretsov Nikita <whalebot.helmsman@gmail.com>
 2021-07-16 12:18:14 +02:00
Eugenio Lacuesta bcce066057
Update ItemFilter (#5203) 2021-07-14 17:56:07 +02:00
Adrián Chaves e7d3fad00e
Merge branch 'master' into master 2021-07-14 11:24:56 +02:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Türkalp Burak KAYRANCIOĞLU fcc6becc58
S3FeedStorage: allow custom endpoint (#4998) 
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2021-07-14 10:00:43 +02:00
D R Siddhartha 4ddc9d6b55
Feeds: Item Filters (#5178) 2021-07-13 17:22:29 +02:00
Eugenio Lacuesta c062ed017a
[CI] fail-fast: false (#5200) 2021-07-12 18:34:22 +02:00
Andrey Rahmatullin 1c46d5aa93
Merge pull request #5191 from ivanprado/master 
CloseSpider can be raised on spider_idle signal handler to set the closing reason
 2021-07-12 18:46:40 +05:00
Eugenio Lacuesta cb08e3644b
Remove trailing whitespaces 2021-07-08 09:22:21 -03:00
Iván de Prado eca641aa3d
Update tests/test_engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:40:20 +01:00
Iván de Prado 6b8f694653
Update scrapy/core/engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:40:02 +01:00