RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,427 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

8427 Commits

Author SHA1 Message Date
Andrey Rakhmatullin de640f41ec Merge back tests/py36/_test_crawl.py. 2020-08-28 18:27:36 +05:00
Mikhail Korobov 64905e3397
Merge pull request #4691 from elacuesta/typing-spider-attributes 
Typing: annotate a few Spider attributes
 2020-08-28 00:03:40 +05:00
Adrián Chaves 3f0a677c04
Cover version directive usage in the documentation policy (#4310) 
* Cover version directives in the documentation policy

* Remove version directives in preparation for Scrapy 2.0

* Update the policy based on the deprecation policy

* Only remove version directives after 3 years
 2020-08-27 23:56:58 +05:00
Adrián Chaves 195f738bba
Update Python version references after dropping support for 3.5 (#4742) 
* Update Python version references after dropping support for 3.5

* Remove outdated test

* Undo change affecting collect_asyncgen

* Undo change to be handled by #4743

* Remove unused import

* Remove unused import

* Update tests/requirements-py3.txt

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-27 15:43:43 +05:00
Mikhail Korobov a6b67cf4a1
Merge pull request #4755 from maranqz/csv-item-exporter-errors 
Add errors parameter for CsvItemExporter with tests
 2020-08-27 12:51:53 +05:00
Eugenio Lacuesta a8114d3731
Typing: annotate a few Spider attributes 2020-08-26 09:00:36 -03:00
maranqz 29725e4b58 Fix tabulation of rst and change documentation link. 2020-08-26 14:38:37 +03:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
maranqz 560c335c07 Add errors parameter in documentation. 2020-08-26 14:00:51 +03:00
Mikhail Korobov 0ccaf89a6f
Merge pull request #4756 from ivanprado/master 
Support for delegated methods as callbacks
 2020-08-26 00:10:43 +05:00
Iván de Prado 0524df8669 Code simplification. 
Thanks @victor-torres for the suggestion
 2020-08-25 14:37:10 +01:00
Andrey Rahmatullin 067125c303
Merge pull request #4752 from drs-11/master 
DOWNLOAD_MAXSIZE logger level changed from Error to Warning
 2020-08-25 18:35:50 +05:00
Adrián Chaves 39affea93c
Fix style issues 2020-08-25 13:57:48 +02:00
maranqz a2d6fa5adc Add errors parameter for CsvItemExporter with tests 2020-08-25 13:34:43 +03:00
Iván de Prado 3e726b9df7 Support for delegated methods as callbacks 
It can be useful to structure the spiders code around some helper classes.
 2020-08-25 11:31:20 +01:00
Andrey Rahmatullin 64e0ea4ee5
Merge pull request #4735 from wRAR/ciphers-tests 
Re-enable TLS 1.2 in cipher tests.
 2020-08-25 11:48:31 +05:00
drs-11 0b3881d65e Reverted maxsize warning log message 2020-08-24 20:37:40 +05:30
WinterComes 2d8ec9d44f Change DOWNLOAD_MAXSIZE logger level from Error to Warning 2020-08-24 20:37:40 +05:30
Eugenio Lacuesta f1250177dc
Remove Python 3.5 from CI (#4743) 2020-08-22 09:33:35 +02:00
Mikhail Korobov ba0e7f3c85
Merge pull request #4747 from Gallaecio/pylint-update 
Skip checks introduced in Pylint 2.6.0
 2020-08-21 23:18:51 +05:00
Adrián Chaves 7c076122eb Skip checks introduced in Pylint 2.6.0 2020-08-21 17:06:54 +02:00
Andrey Rahmatullin 07b0c591e0
Merge pull request #4745 from Yogendra0Sharma/fix4741 
Removed appveyor.xml no longer needed
 2020-08-21 12:36:13 +05:00
yogendra0sharma 2fbfe2c214 Removed appveyor.xml no longer needed 2020-08-21 12:18:15 +05:30
Grisha Temchenko d68aab992e
Smarter generator check for combined return/yield statements (#4721) 2020-08-20 10:22:07 -03:00
Hugo van Kemenade a57db9e302
Bitbucket no longer supports Mercurial repositories (#4738) 2020-08-19 17:45:24 +02:00
sakshamb2113 42383cc267
Add a setting to customize the asyncio event loop (#4414) 2020-08-19 09:18:14 +02:00
Andrey Rakhmatullin d9e69bfb51 Re-enable TLS 1.2 in cipher tests. 2020-08-17 19:46:24 +05:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Mikhail Korobov 282a6d4fc1
Merge pull request #4727 from wRAR/mitmproxy-ca 
Use a longer key in mitmproxy-ca.pem.
 2020-08-17 14:06:58 +05:00
Mikhail Korobov 61459de476
Merge pull request #4688 from ashellunts/obsolete-s3feednokeys 
Remove obsolete S3FeedStorage instancing without AWS credentials
 2020-08-17 13:58:32 +05:00
Adrián Chaves 792d29380e
Merge Twisted’s deprecated HTTPClientFactory into ScrapyHTTPClientFactory (#4165) 
* Port code from Twisted’s deprecated HTTPClientFactory into ScrapyHTTPClientFactory

* Mention that ScrapyHTTPClientFactory has Twisted code
 2020-08-17 10:54:58 +02:00
Mikhail Korobov 5ac135f036
Merge pull request #4718 from Gallaecio/support-pinned-3.6 
Upgrade minimum dependencies for Python 3.6 support
 2020-08-17 13:54:45 +05:00
Grammy Jiang 55edf8d3b8
Add typing hint to httpcache downloadermiddlewares (#4243) 2020-08-17 10:50:52 +02:00
Ajay Mittur a8e08d51cd
Check if file is already present on running `scrapy genspider` and terminate if so (#4623) 2020-08-17 10:45:52 +02:00
Eugenio Lacuesta 2aa4f3cbf9
Conditional request attribute binding for responses (#4632) 2020-08-17 10:39:59 +02:00
Andrey Rakhmatullin 756c368a6b Use a longer key in mitmproxy-ca.pem. 2020-08-14 22:09:24 +05:00
Andrey Rahmatullin acb3b4433a
Merge pull request #4724 from Gallaecio/feed-uri-params 
Document FEED_URI_PARAMS
 2020-08-14 14:47:55 +05:00
Mikhail Korobov 61653418ff
Merge pull request #4090 from Gallaecio/documentation-build 
Fix references to Python types in parameter type fields
 2020-08-14 00:58:06 +05:00
Mikhail Korobov cd0d3fd48d
Merge pull request #4705 from Gallaecio/deprecation-policy 
Cover our deprecation policy in the documentation
 2020-08-14 00:27:37 +05:00
Mikhail Korobov 1f44464a4a
Merge pull request #4723 from Gallaecio/windows 
test_utils_iterators.py: support Windows the right way
 2020-08-14 00:23:55 +05:00
Adrián Chaves 65e0abaea5 Document FEED_URI_PARAMS 2020-08-13 09:05:51 +02:00
Adrián Chaves 24ba5a71ac Maybe the problem is not in the code after all 2020-08-13 06:35:09 +02:00
Adrián Chaves 5f4df622a1 test_utils_iterators.py: support Windows the right way 2020-08-13 05:41:06 +02:00
Adrián Chaves b1de55d37d Fix marker syntax 2020-08-12 12:34:40 +02:00
Adrián Chaves 8e393a0b21 Do not change the mitmproxy version for no-3.6 Python versions 2020-08-12 12:29:51 +02:00
Adrián Chaves 394631fc0a Restore 3.5 support for mitmproxy-based tests 2020-08-12 12:08:09 +02:00
Ajay Mittur 1c4b4cc6b0
Support defining file path based on item in media pipelines (#4686) 2020-08-11 14:12:44 +02:00
Adrián Chaves b2f4df5cb7 Merge remote-tracking branch 'upstream/master' into documentation-build 2020-08-11 13:16:14 +02:00
Adrián Chaves aefd43a6c6 Upgrade minimum dependencies for Python 3.6 support 2020-08-11 12:52:54 +02:00
Andrey Rahmatullin 0cf1340c29
Merge pull request #4663 from ajaymittur28/scrapy-check-test 
Add Tests to `scrapy check` command
 2020-08-10 16:18:29 +05:00