RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,003 Commits 49 Branches 132 Tags 182 MiB
Commit Graph

9003 Commits

Author SHA1 Message Date
Akshay Sharma 0eb1de90e2 fix static checks 2022-06-20 21:10:06 -04:00
Akshay Sharma b9effde875 Refactored _guess_content_type 2022-06-20 20:15:33 -04:00
Akshay Sharma b2181d76d5 Merge branch 'master' of https://github.com/AKSHAYSHARMAJS/scrapy into integrate-mime 2022-05-08 17:48:57 -04:00
Akshay Sharma 66e3b40043 Add more tests 2021-09-20 22:44:13 -04:00
Akshay Sharma c81605abe6 Move tests to post_xtractmime 2021-08-29 13:55:02 -04:00
Andrey Rahmatullin 8284de5e76
Fix/silence the Pylint messages added in 2.10 (#5235) 2021-08-24 12:15:29 +02:00
D R Siddhartha 43ea21e830
Feed post-processing plugin support (#5190) 2021-08-24 11:48:01 +02:00
Matsievskiy S.V 3f635eb683
Extract domain from genspider URL (#4439) 2021-08-24 11:05:50 +02:00
Andrey Rahmatullin ada539a63a
Merge pull request #4676 from Gallaecio/startproject-existing-folder 
Fix startproject failing for existing folders
 2021-08-24 12:18:49 +05:00
Andrey Rakhmatullin cd17c829cf Revert "Move documentation about avoiding bans into a topic of its own (#4039)" 
This reverts commit 2d2581c68f.
 2021-08-23 19:55:35 +05:00
Andrey Rahmatullin 731f2d3085
Merge pull request #4178 from elacuesta/remove_spider_make_requests_from_url 2021-08-20 16:16:25 +05:00
Andrey Rahmatullin bdf8355ce1
Merge pull request #5231 from umairnsr87/umairnsr87/issues/5226 
updated documentation for python version for reppy
 2021-08-18 21:08:32 +05:00
databender 572d347b3b warning view updated 2021-08-18 16:17:52 +05:30
Adrián Chaves 2d2581c68f
Move documentation about avoiding bans into a topic of its own (#4039) 2021-08-18 12:46:42 +02:00
databender d623ed15d1 indentation updated 2021-08-18 14:51:03 +05:30
databender bcf38a6719 added upstream issue for not supported python version 2021-08-18 14:48:47 +05:30
Andrey Rahmatullin a7bd6fa229
Merge pull request #3669 from Gallaecio/spider-name-collision 2021-08-17 19:42:53 +05:00
databender 79bad0cf82 updated suggested changes after review 2021-08-16 18:10:23 +05:30
databender ebddb77a33 updated suggested changes after review 2021-08-16 18:08:26 +05:30
umair ansari 013ac90f61
Update docs/topics/downloader-middleware.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-08-16 18:00:06 +05:30
databender cc1cb2de0c updated suggested changes 2021-08-16 17:21:47 +05:30
databender 1a8b98843a updated documentation for python version for reppy 2021-08-16 17:00:05 +05:30
databender 8bbaea9892 updated documentation for python version for reppy 2021-08-16 16:57:43 +05:30
Andrey Rahmatullin 2814e0e197
Disable builtin middlewares in spider middleware tests. (#5229) 2021-08-16 13:22:01 +02:00
Andrey Rahmatullin 47a281d231
Merge pull request #5224 from aaron-tan/jobdir-doc 2021-08-16 12:52:02 +05:00
Akshay Sharma 4dbe9e11ed Small fix 2021-08-13 01:15:29 +05:30
Akshay Sharma 22c33adb41
Merge branch 'master' into integrate-mime 2021-08-13 01:07:08 +05:30
Akshay Sharma 3475f0049b Change _guess_content_type 2021-08-13 01:05:37 +05:30
Andrey Rahmatullin 0915be7300
Merge pull request #5228 from wRAR/test-reqs 
Rename tests/requirements-py3.txt to tests/requirements.txt.
 2021-08-11 21:04:10 +05:00
Andrey Rakhmatullin b63369c148 Rename tests/requirements-py3.txt to tests/requirements.txt. 2021-08-11 20:02:45 +05:00
Akshay Sharma 4169fd184f More tests 2021-08-11 17:14:38 +05:30
Andrey Rahmatullin 0590c3756c
Merge pull request #5221 from wRAR/3.10-tests 
Fixing tests for upcoming 3.10.
 2021-08-11 14:19:51 +05:00
Andrey Rakhmatullin 74cee38a4e Don't run the asyncio tests on 3.9. 2021-08-11 14:19:08 +05:00
Andrey Rahmatullin 902fce0640
Merge pull request #5225 from divtiply/patch-3 
Allow comma-separated values in the rel attribute
 2021-08-11 10:58:28 +05:00
Andrey Rakhmatullin 983b89ad4f Fix SpiderLoaderTest on Python 3.10. 2021-08-11 10:39:23 +05:00
Michel Ace ce9d6c658b
Add more rel_has_nofollow tests 2021-08-10 22:21:51 +02:00
Michel Ace 295f0e2bf5
Make flake8 happy 2021-08-10 21:38:29 +02:00
Michel Ace 07d20a8ce4
Fix test_rel_has_nofollow test 2021-08-10 21:21:43 +02:00
Michel Ace 18b6f30a73
Add test for rel_has_nofollow 2021-08-10 21:13:50 +02:00
Michel Ace 1ba0f68483
Allow comma-separated values in the rel tag 
Comma-separated `rel` values are often seen in the wild, because Google allows it (see https://developers.google.com/search/docs/advanced/guidelines/qualify-outbound-links).
 2021-08-10 17:09:37 +02:00
Aaron Tan 954f303590
Update docs/topics/settings.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-08-09 22:23:23 +10:00
Akshay Sharma 9980a3d134 ADD xtractmime to pinnedtests 2021-08-09 17:28:12 +05:30
Akshay Sharma 2ba2f49ac5 Add more tests 2021-08-08 19:35:48 +05:30
Aaron Tan 48eff4ee8f Remove JOBDIR from default settings 2021-08-08 20:52:14 +10:00
Akshay Sharma 0b52719e5f Add is_binary_data() 2021-08-08 14:19:19 +05:30
Aaron Tan 8e7d2ef133 Document JOBDIR option issue #5173 
Add JOBDIR setting to the settings page.
Add default JOBDIR setting to global defaults in scrapy.settings.default_settings module.
 2021-08-07 11:44:12 +10:00
Mannan2812 94baa4b272
Fix FileFeedStoragePreFeedOptionsTest fails in CI/CD pipeline (#5198) 2021-08-05 21:23:11 +02:00
Andrey Rakhmatullin 93bf1ae7e3 Fix tests for the 3.10 TypeError message change. 2021-08-03 20:16:59 +05:00
Andrey Rakhmatullin ef6fb933b5 Fix a Python 3.10 logging issue. 2021-08-03 20:16:59 +05:00
Andrey Rakhmatullin 2bf2f9d6db Add Python 3.10b4 tests on Ubuntu. 2021-08-03 20:16:59 +05:00