RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

9291 Commits

Author SHA1 Message Date
Eugenio Lacuesta 90e3bd8715
[test] Name resolvers 2020-01-16 20:32:40 -03:00
Eugenio Lacuesta 1040f581ec
Name resolvers: do not pass the reactor to the install method 2020-01-16 20:14:52 -03:00
Eugenio Lacuesta 3cfa73b8b1
Name resolvers: install_on_reactor as instance method 2020-01-16 18:01:18 -03:00
Eugenio Lacuesta f45b4c7f8d
from_crawler support for name resolvers 2020-01-16 10:09:34 -03:00
Eugenio Lacuesta 0f155b059a
Make Flake8 happy (remove unused import) 2020-01-16 04:27:13 -03:00
Eugenio Lacuesta e69cf415c8
Ability to choose name resolver 2020-01-16 03:58:07 -03:00
Mikhail Korobov 50310fc0f9
Merge pull request #4270 from wRAR/asyncio-pipelines 
async def support in pipelines
 2020-01-16 03:28:09 +05:00
Eugenio Lacuesta 8c3de288fa
Remove non-working DNS timeout code 2020-01-15 12:38:10 -03:00
Eugenio Lacuesta 55babf9acd
Cache resolution only if the DNS request was successful 2020-01-15 12:38:09 -03:00
Eugenio Lacuesta f1c184631e
Name resolver: timeout 2020-01-15 12:38:09 -03:00
Eugenio Lacuesta 735c0ceb78
Custom name resolver implementing twisted.internet.interfaces.IHostnameResolver 2020-01-15 12:38:08 -03:00
abhishekh2001 03241aa4a6
Fixed artwork/README formatting 2020-01-15 08:54:25 +04:00
Eugenio Lacuesta 9770ca35fb
Spider middleware: simplify deferred errback handling 2020-01-10 18:45:39 -03:00
Eugenio Lacuesta d6e928f472
Remove object as base class for MutableChain 
Plus some minor styling adjustments
 2020-01-10 04:40:10 -03:00
Eugenio Lacuesta c088c04f44
Spider middleware: catch exceptions right after the spider callback 2020-01-10 04:20:55 -03:00
Eugenio Lacuesta 6ce1ad3107
[test] Spider middleware: catch exceptions right after the spider callback 2020-01-10 04:20:37 -03:00
Andrey Rakhmatullin a91a13b443 Support for async def callbacks. 2020-01-09 15:59:19 +05:00
Andrey Rakhmatullin 3faef2d082 Add async def support to signal handlers that already supported Deferreds. 2020-01-09 15:17:45 +05:00
Andrey Rakhmatullin 7d85984880 Use get_from_asyncio_queue in the pipeline test. 2020-01-09 14:49:16 +05:00
Andrey Rakhmatullin 9d8c54c0f2 Fix/ignore flake8 problems. 2020-01-09 14:49:02 +05:00
Andrey Rakhmatullin bdef948aae Mark the asyncio pipelines test as only_asyncio. 2020-01-09 14:19:02 +05:00
Andrey Rakhmatullin bfdd552a32 Add a test for pipelines using asyncio. 2020-01-09 14:19:02 +05:00
Andrey Rakhmatullin 1f9cef787d Add async def support to pipelines. 2020-01-09 14:19:02 +05:00
Andrey Rakhmatullin 8117566974 Add utils.defer.deferred_f_from_coro_f. 2020-01-09 14:19:02 +05:00
Eugenio Lacuesta 2e405d2d5c
Merge branch 'master' into response_follow_all 2020-01-05 00:33:19 -03:00
Mikhail Korobov ce618fb6f2
Merge pull request #4259 from scrapy/asyncio-mw 
Asyncio support in downloader middlewares
 2020-01-03 22:28:41 +05:00
Andrey Rakhmatullin b2dd379bc2 Remove the py35-asyncio env for 3.5 from Travis. 2020-01-03 21:38:05 +05:00
Andrey Rakhmatullin 2b9254c2bd Add a test function that uses asyncio.Queue(). 2019-12-31 17:54:41 +05:00
Andrey Rakhmatullin e3b8ba6188 Run py35-asyncio also on 3.5.2 to test Xenial. 2019-12-31 17:54:01 +05:00
Andrey Rakhmatullin 16787f5bf4 Merge middleware tests back as we don't need to set the setting anymore. 2019-12-30 12:02:19 +05:00
Andrey Rakhmatullin 50aa6ef22c Add deferred_from_coro. 2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin 5cf1ac0005 Move the asyncio downloader mw test to a separate class. 2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin 3603644552 Add a non-asyncio async def middleware test. 2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin 21f50c795a Add async def support to downloader middlewares. 2019-12-30 11:46:45 +05:00
1um0s 14d4428e70 Rephrasing documentation for image and file pipelines (#4252) 
* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Revert image, file pipeline docs. Enhance custom media pipeline docs.

* scrapy#4034 rebase master

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Revert image, file pipeline docs. Enhance custom media pipeline docs.

* scrapy#4034 rebase master

* Rebase master

* Add class to media pipeline docs

Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>

Co-authored-by: elacuesta <elacuesta@users.noreply.github.com>
 2019-12-30 00:56:22 +05:00
Mikhail Korobov f0ae673452
Merge pull request #4258 from atul-g/patch-1 
Edited the link provided to homepage of lxml's website
 2019-12-30 00:55:15 +05:00
Mikhail Korobov bb991cd303
Merge pull request #4010 from scrapy/asyncio-base 
Base support for asyncio
 2019-12-30 00:51:28 +05:00
Atul Gopinathan 82861c73c8
Edited the link of the homepage of lxml website 
The link "https://lxml.de" is redirecting to a completely different and unintended website. I changed the link to the index page of lxml's official website. I thought of changing it to the PyPi page of lxml, but even they are providing the same "https://lxml.de" link which doesn't seem to be working now.
 2019-12-27 22:57:58 +05:30
Andrey Rakhmatullin dc1ee09481 Rename ASYNCIO_ENABLED to ASYNCIO_REACTOR, change the logic accordingly. 2019-12-27 21:56:28 +05:00
Andrey Rakhmatullin f75ccc997a FIx a typo in the only_asyncio fixture. 2019-12-27 19:48:54 +05:00
Andrey Rakhmatullin 30ebd05a5f Simplify the tox asyncio entries. 2019-12-27 00:05:14 +05:00
Eugenio Lacuesta 8a1dc26d46
[doc] Note about the 'parse' method for CrawlSpider/XMLFeedSpider 2019-12-26 15:14:47 -03:00
Eugenio Lacuesta c54df8253a
[test] Handle keyword args in CrawlSpider.parse 2019-12-26 15:12:19 -03:00
Eugenio Lacuesta 8d4948f6ca
[test] Override CrawlSpider.parse 2019-12-26 14:38:11 -03:00
Andrey Rakhmatullin 37ac47ff80 Fix a deprecation warning. 2019-12-26 20:46:54 +05:00
Andrey Rakhmatullin 87ece066ca Remove conditional asyncio imports. 2019-12-26 20:41:06 +05:00
Eugenio Lacuesta ab54e0d33e
Keyword-only args for S3DownloadHandler 2019-12-23 20:37:18 -03:00
Eugenio Lacuesta 982a66f9fb
[test] Download handler: avoid passing settings if not necessary 2019-12-23 20:28:17 -03:00
Eugenio Lacuesta 9a75b46fb8
Explicit argument names 2019-12-23 20:26:58 -03:00
Eugenio Lacuesta 2fb160e3ba
Use settings instead of crawler 2019-12-23 20:24:16 -03:00