RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,508 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

8508 Commits

Author SHA1 Message Date
Andrey Rakhmatullin b2f43d51ac Merge remote-tracking branch 'origin/master' into asyncio-startrequests-asyncgen 2020-08-31 18:02:54 +05:00
Mikhail Korobov be655b855d
Merge pull request #4765 from scrapy/simplify-test_crawl 
Simplify tests/test_crawl
 2020-08-29 13:31:20 +05:00
Mikhail Korobov 5e9cc3298b
Merge pull request #4324 from ammarnajjar/4307-use-f-strings 
refactor: use f-strings
 2020-08-29 13:28:38 +05:00
Ammar Najjar 90ca9350f5
Update scrapy/commands/check.py 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-29 08:03:03 +00:00
Andrey Rakhmatullin 8123c42737 Simplify running spiders in CrawlSpiderTestCase. 2020-08-29 12:01:58 +05:00
Ammar Najjar ffdf6fe100
use f-strings for the newly merged code from master 
Issue: #4324
 2020-08-29 07:21:48 +02:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Mikhail Korobov 7e6476ff0a
Merge pull request #4764 from scrapy/test_crawl-move-py36 
Merge back tests/py36/_test_crawl.py.
 2020-08-29 00:12:53 +05:00
Andrey Rahmatullin 59a0157ef1
Merge pull request #4722 from Gallaecio/umask 
Do not let umask affect the permissions of startproject-generated files
 2020-08-28 18:41:02 +05:00
Andrey Rakhmatullin de640f41ec Merge back tests/py36/_test_crawl.py. 2020-08-28 18:27:36 +05:00
Ammar Najjar 0e57918231
test(Slot): cover __repr__ 
Issue: #4324
 2020-08-28 13:58:32 +02:00
Adrián Chaves d568423b68
Merge branch 'master' into asyncio-startrequests-asyncgen 2020-08-28 09:55:59 +02:00
Mikhail Korobov 64905e3397
Merge pull request #4691 from elacuesta/typing-spider-attributes 
Typing: annotate a few Spider attributes
 2020-08-28 00:03:40 +05:00
Adrián Chaves 3f0a677c04
Cover version directive usage in the documentation policy (#4310) 
* Cover version directives in the documentation policy

* Remove version directives in preparation for Scrapy 2.0

* Update the policy based on the deprecation policy

* Only remove version directives after 3 years
 2020-08-27 23:56:58 +05:00
Andrey Rakhmatullin 7839aee998 Merge remote-tracking branch 'origin/master' into asyncio-startrequests-asyncgen 2020-08-27 20:37:16 +05:00
Adrián Chaves 195f738bba
Update Python version references after dropping support for 3.5 (#4742) 
* Update Python version references after dropping support for 3.5

* Remove outdated test

* Undo change affecting collect_asyncgen

* Undo change to be handled by #4743

* Remove unused import

* Remove unused import

* Update tests/requirements-py3.txt

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-27 15:43:43 +05:00
Mikhail Korobov a6b67cf4a1
Merge pull request #4755 from maranqz/csv-item-exporter-errors 
Add errors parameter for CsvItemExporter with tests
 2020-08-27 12:51:53 +05:00
Andrey Rakhmatullin 33ebfcacca Merge branch 'master' into asyncio-startrequests-asyncgen 2020-08-26 19:24:20 +05:00
Andrey Rakhmatullin c0b2488213 Drop python 3.5 support from this PR. 2020-08-26 19:23:15 +05:00
Ammar Najjar 5ab1a318e8
test: list appears in ValueError Exception message 
Issue #4324
 2020-08-26 15:11:46 +02:00
Andrey Rakhmatullin 2f5cabc134 Fix async def (non-generator) process_start_requests handling. 2020-08-26 17:57:51 +05:00
Eugenio Lacuesta a8114d3731
Typing: annotate a few Spider attributes 2020-08-26 09:00:36 -03:00
Ammar Najjar 2ca8dfb4b0
revert f-string changes for files under sep/ 
Issue #4324
 2020-08-26 13:49:39 +02:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
Ammar Najjar 9aaddcde0a
Update scrapy/utils/conf.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:44:20 +00:00
Ammar Najjar ea03e4254f
Update scrapy/http/request/form.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:43:52 +00:00
Ammar Najjar 92dfa7176d
Update scrapy/extensions/statsmailer.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:43:40 +00:00
Ammar Najjar c77450990d
Update scrapy/commands/version.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 11:41:24 +00:00
maranqz 29725e4b58 Fix tabulation of rst and change documentation link. 2020-08-26 14:38:37 +03:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
maranqz 560c335c07 Add errors parameter in documentation. 2020-08-26 14:00:51 +03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Mikhail Korobov 0ccaf89a6f
Merge pull request #4756 from ivanprado/master 
Support for delegated methods as callbacks
 2020-08-26 00:10:43 +05:00
Iván de Prado 0524df8669 Code simplification. 
Thanks @victor-torres for the suggestion
 2020-08-25 14:37:10 +01:00
Andrey Rahmatullin 067125c303
Merge pull request #4752 from drs-11/master 
DOWNLOAD_MAXSIZE logger level changed from Error to Warning
 2020-08-25 18:35:50 +05:00
Adrián Chaves 39affea93c
Fix style issues 2020-08-25 13:57:48 +02:00
maranqz a2d6fa5adc Add errors parameter for CsvItemExporter with tests 2020-08-25 13:34:43 +03:00
Iván de Prado 3e726b9df7 Support for delegated methods as callbacks 
It can be useful to structure the spiders code around some helper classes.
 2020-08-25 11:31:20 +01:00
Andrey Rahmatullin 64e0ea4ee5
Merge pull request #4735 from wRAR/ciphers-tests 
Re-enable TLS 1.2 in cipher tests.
 2020-08-25 11:48:31 +05:00
drs-11 0b3881d65e Reverted maxsize warning log message 2020-08-24 20:37:40 +05:30
WinterComes 2d8ec9d44f Change DOWNLOAD_MAXSIZE logger level from Error to Warning 2020-08-24 20:37:40 +05:30
Eugenio Lacuesta f1250177dc
Remove Python 3.5 from CI (#4743) 2020-08-22 09:33:35 +02:00
Mikhail Korobov ba0e7f3c85
Merge pull request #4747 from Gallaecio/pylint-update 
Skip checks introduced in Pylint 2.6.0
 2020-08-21 23:18:51 +05:00
Adrián Chaves 7c076122eb Skip checks introduced in Pylint 2.6.0 2020-08-21 17:06:54 +02:00
Andrey Rahmatullin 07b0c591e0
Merge pull request #4745 from Yogendra0Sharma/fix4741 
Removed appveyor.xml no longer needed
 2020-08-21 12:36:13 +05:00
yogendra0sharma 2fbfe2c214 Removed appveyor.xml no longer needed 2020-08-21 12:18:15 +05:30
Grisha Temchenko d68aab992e
Smarter generator check for combined return/yield statements (#4721) 2020-08-20 10:22:07 -03:00
Hugo van Kemenade a57db9e302
Bitbucket no longer supports Mercurial repositories (#4738) 2020-08-19 17:45:24 +02:00
sakshamb2113 42383cc267
Add a setting to customize the asyncio event loop (#4414) 2020-08-19 09:18:14 +02:00
Andrey Rakhmatullin d9e69bfb51 Re-enable TLS 1.2 in cipher tests. 2020-08-17 19:46:24 +05:00