RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,800 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

7800 Commits

Author SHA1 Message Date
faizan2700 0641ba0faa SCRAPY_CHECK will be set while running contact 2020-02-02 16:54:22 +05:30
Mikhail Korobov 22f7934fcc
Merge pull request #4269 from wRAR/asyncio-parse 
Support for async def callbacks.
 2020-01-31 23:50:19 +05:00
Andrey Rakhmatullin cc825c21de Test returning items from an async def callback. 2020-01-30 16:17:06 +05:00
Mikhail Korobov 47b9de93a9
Merge pull request #4288 from petervandenabeele/patch-1 
[Docs] 2 typos + 1 clarification in docs
 2020-01-30 03:15:18 +05:00
Mikhail Korobov 534ce07b10
Merge pull request #4293 from edorofeev/fix-accept-encoding-delimiter 
[HttpCompressionMiddleware] fix delimiter for Accept-Encoding header
 2020-01-30 00:19:40 +05:00
Evgeny Dorofeev 4e56571a19 [HttpCompressionMiddleware] fix delimiter for Accept-Encoding header 2020-01-29 15:49:22 +03:00
Kevin Lloyd Bernal a3b168948c Log an error when giving up requests after too many retries (#3566) 2020-01-28 21:53:25 +01:00
Mikhail Korobov a6bf895474
Merge pull request #4286 from petervandenabeele/fix-documentation 
[Docs] Fix variable name `author_page_links`
 2020-01-27 20:57:22 +05:00
Peter Vandenabeele f72d4e93e6
[Docs] 2 typos + 1 clarification in docs 
Fixing 2 small typos and adding 1 word as clarification in the downloader-middlewares.

Also, I was confused with the entries like `ref:Reppy <reppy-parser>` and similar entries.
Are these supposed to be links to other parts of the doc, or is this the intended way of showing these references ?
 2020-01-26 10:48:28 +01:00
Peter Vandenabeele f3374a5047 Fix variable name `author_page_links` 
I did not test this code, but the change from `href` to this
author_page_links seems to have a typo ?
 2020-01-25 16:53:29 +01:00
Mikhail Korobov 8b8df31961
Merge pull request #4227 from elacuesta/name-resolver 
Name resolver with IPv6 support
 2020-01-25 05:03:13 +05:00
Mikhail Korobov 5f407cf657
Merge pull request #3961 from OmarFarrag/ftp_files#3928 
Add FTPFileStore to FilesPipeline
 2020-01-25 04:58:23 +05:00
Mikhail Korobov 7a62bd310c
Merge pull request #4126 from elacuesta/from_crawler_downloader_handlers 
Download handlers: from_crawler factory method, take crawler in __init__
 2020-01-25 04:52:56 +05:00
OmarFarrag 9e6d5573f1 Fix Flake8 errors 2020-01-24 15:58:52 +02:00
OmarFarrag 40e0a11aa8 Fix Flake8 errors 2020-01-24 15:51:48 +02:00
OmarFarrag f5d9eb15f8 use `__future__` imports at the begining of the file 2020-01-24 15:06:40 +02:00
OmarFarrag fc98aa6b67
Merge branch 'master' into ftp_files#3928 2020-01-24 14:52:40 +02:00
OmarFarrag c544c0d2b8 Use context management with `FTP` 2020-01-24 14:36:16 +02:00
Eugenio Lacuesta 9899414300
Name resolver: return result directly 2020-01-23 18:06:59 -03:00
Mikhail Korobov bd54f22fef
Merge pull request #4282 from petervandenabeele/patch-1 
fix logical documentation error with PER_DOMAIN or PER_DOMAIN
 2020-01-24 02:01:22 +05:00
Mikhail Korobov 6a98d660e5
Merge pull request #3551 from jpbalarini/change_scraper_slot 
[MRG+1] Add ability to change max_active_size by setting
 2020-01-23 23:40:05 +05:00
Mikhail Korobov f80c7776ae
Merge pull request #4008 from elacuesta/docs_request_errback 
Request: remove restriction about errback without callback
 2020-01-23 23:12:44 +05:00
Peter Vandenabeele 7d5cebcf77
fix logical documentation error with PER_DOMAIN or PER_DOMAIN 2020-01-23 09:08:21 +01:00
Mikhail Korobov c0a7dfbc01
Merge pull request #4057 from elacuesta/response_follow_all 
Response.follow_all
 2020-01-23 02:15:24 +05:00
Eugenio Lacuesta c75cf15b7a
Update CSS selectors in tutorial 2020-01-22 10:38:59 -03:00
OmarFarrag 06ab668ec7 Use kwargs-only parameters in `ftp_store_file` 2020-01-22 03:48:07 +02:00
OmarFarrag 8ea8f14827
Update scrapy/utils/ftp.py 
Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
 2020-01-20 18:19:36 +02:00
JP Balarini 0f2d871d88 Use PEP 515 style for SCRAPER_SLOT_MAX_ACTIVE_SIZE documentation 2020-01-20 11:28:28 -03:00
Juan Pablo Balarini eaa8ed02d0 Add ability to change max_active_size by settings 2020-01-20 11:27:58 -03:00
Eugenio Lacuesta b471765d40
[doc] FAQ entry about the IPv6 and the DNS_RESOLVER setting 2020-01-18 01:52:29 -03:00
Eugenio Lacuesta 302d3f552b
[doc] DNS_RESOLVER setting 2020-01-18 01:46:24 -03:00
Eugenio Lacuesta 41f7ebf3ad
CachingThreadedResolver: No need to store the reactor as an instance attribute 
It's already done in the parent class
 2020-01-17 12:40:59 -03:00
Eugenio Lacuesta dee420a69c
Fix name resolvers tests 2020-01-16 23:48:16 -03:00
Eugenio Lacuesta d487498cff
Update name resolvers tests 2020-01-16 22:02:01 -03:00
Eugenio Lacuesta 90e3bd8715
[test] Name resolvers 2020-01-16 20:32:40 -03:00
Eugenio Lacuesta 1040f581ec
Name resolvers: do not pass the reactor to the install method 2020-01-16 20:14:52 -03:00
Eugenio Lacuesta 3cfa73b8b1
Name resolvers: install_on_reactor as instance method 2020-01-16 18:01:18 -03:00
Eugenio Lacuesta f45b4c7f8d
from_crawler support for name resolvers 2020-01-16 10:09:34 -03:00
Eugenio Lacuesta 0f155b059a
Make Flake8 happy (remove unused import) 2020-01-16 04:27:13 -03:00
Eugenio Lacuesta e69cf415c8
Ability to choose name resolver 2020-01-16 03:58:07 -03:00
Mikhail Korobov 50310fc0f9
Merge pull request #4270 from wRAR/asyncio-pipelines 
async def support in pipelines
 2020-01-16 03:28:09 +05:00
Eugenio Lacuesta 8c3de288fa
Remove non-working DNS timeout code 2020-01-15 12:38:10 -03:00
Eugenio Lacuesta 55babf9acd
Cache resolution only if the DNS request was successful 2020-01-15 12:38:09 -03:00
Eugenio Lacuesta f1c184631e
Name resolver: timeout 2020-01-15 12:38:09 -03:00
Eugenio Lacuesta 735c0ceb78
Custom name resolver implementing twisted.internet.interfaces.IHostnameResolver 2020-01-15 12:38:08 -03:00
Andrey Rakhmatullin a91a13b443 Support for async def callbacks. 2020-01-09 15:59:19 +05:00
Andrey Rakhmatullin 7d85984880 Use get_from_asyncio_queue in the pipeline test. 2020-01-09 14:49:16 +05:00
Andrey Rakhmatullin 9d8c54c0f2 Fix/ignore flake8 problems. 2020-01-09 14:49:02 +05:00
Andrey Rakhmatullin bdef948aae Mark the asyncio pipelines test as only_asyncio. 2020-01-09 14:19:02 +05:00
Andrey Rakhmatullin bfdd552a32 Add a test for pipelines using asyncio. 2020-01-09 14:19:02 +05:00