RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,965 Commits 42 Branches 132 Tags 162 MiB
Commit Graph

8965 Commits

Author SHA1 Message Date
Akshay Sharma 22c33adb41
Merge branch 'master' into integrate-mime 2021-08-13 01:07:08 +05:30
Akshay Sharma 3475f0049b Change _guess_content_type 2021-08-13 01:05:37 +05:30
Andrey Rahmatullin 0915be7300
Merge pull request #5228 from wRAR/test-reqs 
Rename tests/requirements-py3.txt to tests/requirements.txt.
 2021-08-11 21:04:10 +05:00
Andrey Rakhmatullin b63369c148 Rename tests/requirements-py3.txt to tests/requirements.txt. 2021-08-11 20:02:45 +05:00
Akshay Sharma 4169fd184f More tests 2021-08-11 17:14:38 +05:30
Andrey Rahmatullin 0590c3756c
Merge pull request #5221 from wRAR/3.10-tests 
Fixing tests for upcoming 3.10.
 2021-08-11 14:19:51 +05:00
Andrey Rakhmatullin 74cee38a4e Don't run the asyncio tests on 3.9. 2021-08-11 14:19:08 +05:00
Andrey Rahmatullin 902fce0640
Merge pull request #5225 from divtiply/patch-3 
Allow comma-separated values in the rel attribute
 2021-08-11 10:58:28 +05:00
Andrey Rakhmatullin 983b89ad4f Fix SpiderLoaderTest on Python 3.10. 2021-08-11 10:39:23 +05:00
Michel Ace ce9d6c658b
Add more rel_has_nofollow tests 2021-08-10 22:21:51 +02:00
Michel Ace 295f0e2bf5
Make flake8 happy 2021-08-10 21:38:29 +02:00
Michel Ace 07d20a8ce4
Fix test_rel_has_nofollow test 2021-08-10 21:21:43 +02:00
Michel Ace 18b6f30a73
Add test for rel_has_nofollow 2021-08-10 21:13:50 +02:00
Michel Ace 1ba0f68483
Allow comma-separated values in the rel tag 
Comma-separated `rel` values are often seen in the wild, because Google allows it (see https://developers.google.com/search/docs/advanced/guidelines/qualify-outbound-links).
 2021-08-10 17:09:37 +02:00
Akshay Sharma 9980a3d134 ADD xtractmime to pinnedtests 2021-08-09 17:28:12 +05:30
Akshay Sharma 2ba2f49ac5 Add more tests 2021-08-08 19:35:48 +05:30
Akshay Sharma 0b52719e5f Add is_binary_data() 2021-08-08 14:19:19 +05:30
Mannan2812 94baa4b272
Fix FileFeedStoragePreFeedOptionsTest fails in CI/CD pipeline (#5198) 2021-08-05 21:23:11 +02:00
Andrey Rakhmatullin 93bf1ae7e3 Fix tests for the 3.10 TypeError message change. 2021-08-03 20:16:59 +05:00
Andrey Rakhmatullin ef6fb933b5 Fix a Python 3.10 logging issue. 2021-08-03 20:16:59 +05:00
Andrey Rakhmatullin 2bf2f9d6db Add Python 3.10b4 tests on Ubuntu. 2021-08-03 20:16:59 +05:00
Andrey Rahmatullin 4d1ecc31c9
Merge pull request #5217 from aaron-tan/docs-extension 2021-08-02 17:36:35 +05:00
Akshay Sharma 841a218431 Small fix 2021-08-02 14:26:54 +05:30
Akshay Sharma 57d76df85c
Merge pull request #2 from akshaysharmajs/response-class 
Use xtractmime.mimegroups to determine response classes
 2021-08-02 14:13:14 +05:30
Akshay Sharma 2e0e99c1e5 Refactor response class 2021-08-01 17:17:20 +05:30
Akshay Sharma f573d11ec2 Logic fix 2021-08-01 14:40:39 +05:30
Aaron Tan 880a4d9493
Update docs/topics/extensions.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-08-01 11:02:27 +10:00
Akshay Sharma 3a20b79de7 Move headers logic 2021-07-31 21:08:42 +05:30
Akshay Sharma 5ee086e7b6
Update scrapy/responsetypes.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-31 20:30:43 +05:30
Akshay Sharma 180b457f77 Format change 2021-07-31 17:38:28 +05:30
Akshay Sharma 5c256d5c5f Add guess content type 2021-07-31 01:57:09 +05:30
Andrey Rahmatullin b406affd5b
Merge pull request #5210 from laggardkernel/feature/aws-session-token 2021-07-30 15:15:55 +05:00
Eugenio Lacuesta cc89f6be38
Response.attributes (#5218) 2021-07-29 22:12:44 +02:00
Andrey Rahmatullin 22bd01237b
Merge pull request #4613 from elacuesta/docs-headless-browser 
Update headless browser docs
 2021-07-29 15:25:08 +05:00
Eugenio Lacuesta 4b62ac6c3a
Update headless browser docs to mention playwright 2021-07-28 15:00:24 -03:00
Eugenio Lacuesta 9aef9b78eb
Merge remote-tracking branch 'upstream/master' into docs-headless-browser 2021-07-28 14:31:49 -03:00
Eugenio Lacuesta 0e3d50dd18
Update docs/topics/dynamic-content.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-28 14:30:16 -03:00
Eugenio Lacuesta 494e0ad8ff
Update docs/topics/dynamic-content.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-28 14:29:50 -03:00
Aaron Tan d55b6fcad6 Fix for duplicate object description error 2021-07-28 12:10:34 +10:00
Aaron Tan 714aa3970e
Merge branch 'scrapy:master' into docs-extension 2021-07-30 20:36:58 +10:00
laggardkernel 8e7b96d8a2 Tweak doc for setting AWS_SESSION_TOKEN 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-27 19:40:02 +08:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
laggardkernel abe0b37d30
Cleanup leftover boto2 code in S3DownloaderHandler (#5209) 
S3DownloaderHandler.conn is a leftover attribute from 5e99758.
 2021-07-27 11:11:32 +02:00
Rob Banagale b22a004398
Document media pipeline file naming (#5152) 2021-07-26 20:51:32 +02:00
Akshay Sharma 1512ed22b2 Pre n post xtractmime tests 2021-07-25 19:42:27 +05:30
Aaron Tan 209c1fce02
Reference MailSender in StatsMailer 
Added a reference to MailSender in the StatsMailer extension description and included a link to the document detailing how to instantiate MailSender and using Scrapy settings objects.
 2021-07-24 14:50:48 +10:00
Mikhail Korobov ff272d6332
Merge pull request #5215 from pcorpet/types 
Typing: switch to a newer version of MyPy to check types
 2021-07-22 17:52:48 +05:00
Pascal Corpet 70dddfe2b2 Typing: switch to a newer version of MyPy to check types 2021-07-21 17:10:10 +02:00
Akshay Sharma 763ec07b7b Deprecation Warnings 2021-07-18 03:56:12 +05:30
Andrey Rahmatullin ee2df97bbd
Pin the libxml2 version in CI as a newer one breaks lxml (#5208) 2021-07-16 14:28:32 +02:00