RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,393 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

10393 Commits

Author SHA1 Message Date
Andrey Rakhmatullin c079d5002b Run tests without asyncio support by default, add py35-asyncio and py38-asyncio envs. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 794cf71806 Fix or ignore flake8 problems. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 3ba25ccbd3 Don't use asyncio.iscoroutine, as it is True for generators. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin f41c2f3874 Add py38-no-asyncio to Travis. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin cc19ab5439 Add tests that check asyncio support. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 2fbe7d49dc Log asyncio support on spider start. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin b04b541372 Install the asyncio reactor only in scrapy.cmdline. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 8d8fbddbde Switch to the released version of pytest-twisted. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 63c3c62305 Add utils.deferred_from_coro. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 9777639533 Run tests using the asyncio reactor. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 1b437bbe9f Install the asyncio reactor on "import scrapy". 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 1b35260625 Add a test for downloader middlewares using Deferreds. 2019-12-04 21:32:56 +05:00
Grammy Jiang 74627033c4 Remove the used import and re-arrange the imports (#4208) 
This commit removes unused import and re-arrange the imports in cookies
module
 2019-12-04 14:24:14 +01:00
Grammy Jiang 702333478d Re-arrange the imports in httpcache module (#4209) 
This commit re-arrange the imports in httpcache module to follow pep8
 2019-12-04 14:23:28 +01:00
Grammy Jiang 5d8d4bb7d7 Re-arrange the imports in the httpproxy module (#4210) 
This commit re-arranges the imports in the httpproxy module to follow
pep8
 2019-12-04 14:22:10 +01:00
Eugenio Lacuesta 62778cf23f
Request: remove restriction about errback without callback 2019-12-04 08:53:38 -03:00
Eugenio Lacuesta 2a9f5a0aef
Skip invalid links when passing SelectorLists to Response.follow_all 2019-12-03 16:01:42 -03:00
Mikhail Korobov 9b7452211a
Merge pull request #4099 from BurnzZ/itemloader-docs 
update docs of scrapy.loader.ItemLoader.item
 2019-12-03 13:14:45 +05:00
Eugenio Lacuesta e43f37fff3
Pass args/kwargs in S3DownloadHandler.from_crawler, update tests 2019-12-02 17:36:31 -03:00
Eugenio Lacuesta 3d77f74e40
Download handlers: from_crawler factory method, take crawler instead of settings in __init__ 2019-12-02 17:36:31 -03:00
Adrián Chaves 5980b0f284 Don’t use follow_all where a single item is expected (#4) 2019-12-02 12:47:44 -03:00
Grammy Jiang d1cdfb4701 Use pprint.pformat on overridden settings (#4199) 
Keeps consistency with scrapy.middleware
 2019-11-29 09:13:57 +01:00
Mikhail Korobov 3a5b86220f
Merge pull request #4194 from Gallaecio/intersphinx 
Use InterSphinx for coverage links
 2019-11-28 17:36:19 +05:00
Eugenio Lacuesta 048cd74ae5
Add separate mapping for Twisted API docs 2019-11-27 19:16:18 -03:00
Eugenio Lacuesta 17e6481823
[Docs] Fix Twisted links 2019-11-27 18:42:42 -03:00
Andrey Rakhmatullin 63546cbf3e Deprecate the HTTPS proxy noconnect mode. 2019-11-27 22:42:52 +05:00
Adrián Chaves b73fc99b60 Use InterSphinx for coverage links 2019-11-26 10:31:55 +01:00
Adrián Chaves dd12f5fdcd Use Response.follow_all in the documentation where appropiate 2019-11-25 11:59:59 -03:00
Adrián Chaves ed1e577610 Use super().__init__ in BaseItemExporter subclasses 2019-11-25 13:38:10 +01:00
Adrián Chaves 54b056c4be Make developer-tools doctests pass 2019-11-25 12:32:35 +01:00
Adrián Chaves 6d9ed6146d
Merge branch 'master' into remove-six-code 2019-11-25 10:34:21 +01:00
Andrey Rahmatullin 8a1c99676e
Merge pull request #3899 from elacuesta/py3_single_argument_processors 
[Py3] Item loaders: allow single argument functions as processors
 2019-11-25 13:47:58 +05:00
Eugenio Lacuesta 03af8885ff
LogFormatter.download_error 2019-11-23 20:02:44 -03:00
Eugenio Lacuesta 4756e7c587
LogFormatter.spider_error 2019-11-23 19:33:29 -03:00
Eugenio Lacuesta facb926542
Remove quotes from item_error message 2019-11-23 19:16:41 -03:00
Eugenio Lacuesta 7a7d13b112
Rename LogFormatter.error to item_error 2019-11-23 19:12:03 -03:00
Eugenio Lacuesta 40b5cfc0a4
Item loaders: allow single-argument processors (unbound methods) 2019-11-22 20:47:22 -03:00
Eugenio Lacuesta 6fa8f06b64
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-22 13:05:06 -03:00
Eugenio Lacuesta 55cc5c9068
Skip pickle in bandit check 2019-11-22 12:41:31 -03:00
Eugenio Lacuesta 5bab3c0261
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-22 12:12:29 -03:00
Mikhail Korobov 16e0636dcf
Merge pull request #4186 from Gallaecio/lgtm 
Remove unused imports
 2019-11-22 12:28:29 +05:00
Adrián Chaves 9b5053c564 Undo unintended tox.ini changes 2019-11-21 22:00:34 +01:00
Mikhail Korobov 0d416c6191
Merge pull request #4185 from Gallaecio/intersphinx 
Use InterSphinx for links to the pytest and tox documentation
 2019-11-21 23:27:57 +05:00
Mabel Villalba 070b3a4e84
Merge branch 'master' into 4133-handle-start_url 2019-11-21 17:10:31 +01:00
Mabel Villalba 1718e450ef [start_url] Fixes #4133: Raise AttributeError error when empty 'start_urls' and 'start_url' found. Added test. 2019-11-21 17:06:47 +01:00
Mikhail Korobov 82c01c261f
Merge pull request #4184 from Gallaecio/doctest 
Make debug doctests pass
 2019-11-21 20:53:56 +05:00
Eugenio Lacuesta bf503869e9
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-21 10:58:24 -03:00
Adrián Chaves b232881356 Restore intentional import of unused objects 2019-11-21 14:30:10 +01:00
Mikhail Korobov 91c9576675
Merge pull request #4183 from Gallaecio/sphinx-extensions-2 
Enable sphinx-hoverxref for all references
 2019-11-21 18:25:21 +05:00
Mikhail Korobov d2b73b8e21
Merge pull request #4114 from scrapy/remove-py2-tests 
Remove py2 tests
 2019-11-21 18:22:10 +05:00