RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,393 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

10393 Commits

Author SHA1 Message Date
Eugenio Lacuesta c54df8253a
[test] Handle keyword args in CrawlSpider.parse 2019-12-26 15:12:19 -03:00
Eugenio Lacuesta 8d4948f6ca
[test] Override CrawlSpider.parse 2019-12-26 14:38:11 -03:00
Andrey Rakhmatullin 37ac47ff80 Fix a deprecation warning. 2019-12-26 20:46:54 +05:00
Andrey Rakhmatullin 87ece066ca Remove conditional asyncio imports. 2019-12-26 20:41:06 +05:00
Eugenio Lacuesta ab54e0d33e
Keyword-only args for S3DownloadHandler 2019-12-23 20:37:18 -03:00
Eugenio Lacuesta 982a66f9fb
[test] Download handler: avoid passing settings if not necessary 2019-12-23 20:28:17 -03:00
Eugenio Lacuesta 9a75b46fb8
Explicit argument names 2019-12-23 20:26:58 -03:00
Eugenio Lacuesta 2fb160e3ba
Use settings instead of crawler 2019-12-23 20:24:16 -03:00
Mikhail Korobov 4d594b8c2b
Merge pull request #4193 from Gallaecio/lgtm 
Use super().__init__ in BaseItemExporter subclasses
 2019-12-23 23:52:43 +05:00
Eugenio Lacuesta 5982e3477c
Take keyword arguments in base parsing methods 2019-12-23 14:12:21 -03:00
nyov b9a58798ee
change Scraper API to call internal `_parse` method 
A Spider class using internal pre-processing can have first dibs
at this and then call a public `parse` method for subclass hooking.
 2019-12-23 13:55:06 -03:00
Eugenio Lacuesta e2e15d6651
Downloader handlers: sort imports 2019-12-23 10:48:19 -03:00
Eugenio Lacuesta a6ec89251e
Downloader handlers: crawler=None in __init__ 2019-12-23 10:47:08 -03:00
Eugenio Lacuesta 8a567e98bb
Remove unnecessary __init__ methods in downloader handler tests 2019-12-23 10:36:46 -03:00
Eugenio Lacuesta 7e6387de40
Use create_instance in FTPDownloadHandler/DataURIDownloadHandler tests 2019-12-23 10:02:58 -03:00
Eugenio Lacuesta fa21d8687a
Use create_instance in S3DownloadHandler tests 2019-12-23 10:00:25 -03:00
Eugenio Lacuesta 9e5d945ef2
Use create_instance in downloader handler tests 2019-12-23 09:55:47 -03:00
Eugenio Lacuesta 342bf3cd35
Explicit keyword arguments 2019-12-23 09:52:55 -03:00
Eugenio Lacuesta 931b7e68d3
Update FileDownloadHandler test 2019-12-23 09:50:28 -03:00
Eugenio Lacuesta 310cc081d0
Merge branch 'master' into from_crawler_downloader_handlers 2019-12-21 17:24:45 -03:00
Andrey Rakhmatullin 8de80f59db Raise an exception if ASYNCIO_ENABLED but the reactor is wrong. 2019-12-21 13:08:29 +05:00
Mikhail Korobov 5577d4d2be
Merge pull request #3869 from elacuesta/detect_return_in_generator_callbacks 
[MRG+1][Py3] Check for 'return' with arguments inside generators
 2019-12-20 22:46:46 +05:00
Andrey Rakhmatullin e342de5038 Remove a stray newline. 2019-12-20 19:37:50 +05:00
Andrey Rakhmatullin 40697dcbfa Remove deferred_from_coro from this PR. 2019-12-20 19:33:44 +05:00
Andrey Rakhmatullin 20289be810 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-20 19:31:20 +05:00
Mikhail Korobov fb3fb17c28
Merge pull request #4045 from Gallaecio/documentation-coverage 
Deprecate FilteringLinkExtractor direct usage; add API docs for scrapy.linkextractors
 2019-12-20 02:01:45 +05:00
Mikhail Korobov 6a32a36e09
Merge pull request #4143 from Gallaecio/undent-rst-doctests 
Do not indent doctests from the documentation unnecessarily
 2019-12-20 01:59:07 +05:00
Mikhail Korobov 1213c0e5f0
Merge pull request #4247 from noviluni/fix_typos 
fix typos
 2019-12-20 01:56:36 +05:00
Mikhail Korobov a611a7c0f2
Merge pull request #4249 from Gallaecio/docs-py37 
Use Python 3.7 to build the documentation
 2019-12-20 01:54:26 +05:00
Adrián Chaves c7f9b955bd
Pylint: ignore not-an-iterable 2019-12-19 12:44:52 +01:00
Adrián Chaves b4a2d985a7
Merge branch 'master' into undent-rst-doctests 2019-12-19 12:39:17 +01:00
Adrián Chaves e22c0c27d9 Revert "Improve FilteringLinkExtractor.__new__" 
This reverts commit ee9881d270.
 2019-12-19 12:15:54 +01:00
Adrián Chaves f6bc1940a3 Use Python 3.7 to build the documentation 2019-12-19 12:06:15 +01:00
Andrey Rahmatullin c841a1f3e7
Merge pull request #4140 from Gallaecio/docs-py38 
Use the latest Python version to build the documentation
 2019-12-19 14:09:02 +05:00
Marc Hernandez Cabot 16b363de31 Merge branch 'master' into fix_typos 2019-12-19 10:02:21 +01:00
Marc Hernandez Cabot 23a67cec27 fix first letter capitalization for Raring and Scrapy 2019-12-19 09:57:17 +01:00
Mikhail Korobov 3bcb14fe4d
Merge pull request #4164 from Gallaecio/speed-up-tests-2 
Improve the performance of the DOWNLOAD_DELAY test
 2019-12-18 23:56:56 +05:00
Mikhail Korobov 8f7faaa63d
Merge pull request #4190 from Gallaecio/doctest 
Make developer-tools doctests pass
 2019-12-18 23:40:21 +05:00
Marc Hernandez Cabot c0d84f0962 fix typos 2019-12-18 19:39:21 +01:00
Mikhail Korobov c976230b5f
Merge pull request #4246 from noviluni/fix_W293_blank_line_contains_whitespace 
Fix W291, W292 and W293 flake8 issues (whitespaces)
 2019-12-18 23:39:04 +05:00
Marc Hernandez Cabot a5de2c64e6 fix W291, W292, W293 (whitespaces) 2019-12-18 16:33:46 +01:00
elacuesta 916382e109 Add errback parameter to scrapy.spiders.crawl.Rule (#4000) 
* Add errback parameter to scrapy.spiders.crawl.Rule

* CrawlSpider: optimize by reducing iterations

* [test] Rule.errback

* [doc] Rule.errback

* [doc] Use autoclass in docs/topics/spiders.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Rule.process_links takes a list

* Fix aesthetic issue reported by Flake8
 2019-12-18 20:05:33 +05:00
Andrey Rahmatullin 0e8ee22a88
Merge pull request #4244 from wRAR/split-EngineTest 
Split a long test in test_engine.py into three.
 2019-12-18 20:00:36 +05:00
Adrián Chaves 00517ec273
Merge branch 'master' into docs-py38 2019-12-18 15:50:49 +01:00
Adrián Chaves 8c5a8a02ae
Merge branch 'master' into pylint 2019-12-18 15:43:40 +01:00
Andrey Rakhmatullin 7ccb169a27 Split a long test in test_engine.py into three. 2019-12-18 19:41:16 +05:00
Mikhail Korobov 053319334c
Merge pull request #4179 from Gallaecio/user-friendlier-tox 
Make tox configuration more user friendly
 2019-12-18 18:35:19 +05:00
Andrey Rahmatullin e2b5cdeb88
Merge pull request #4242 from whalebot-helmsman/single_place_for_dependencies 
Remove requirements-py3.txt
 2019-12-18 18:27:09 +05:00
Vostretsov Nikita 012533924a remove requirements from here too 2019-12-18 11:13:36 +00:00
Mikhail Korobov e90f276903
Merge pull request #4198 from wRAR/deprecate-noconnect 
Deprecate the HTTPS proxy noconnect mode.
 2019-12-18 16:11:24 +05:00