RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 149 MiB
Commit Graph

10710 Commits

Author SHA1 Message Date
Mikhail Korobov 8d2705f23c
Merge pull request #4305 from Respawnz/patch-1 
fix a typo in devloper-tools.rst
 2020-02-06 23:17:28 +05:00
elacuesta 35dafef7f1
Specify Twisted reactor (TWISTED_REACTOR setting) (#4294) 
* Add the ability to install a specific reactor

* Add docs for the TWISTED_REACTOR setting

* Add tests for the TWISTED_REACTOR setting

* Update asyncio reactor test

* Ignore W503 globally

W503 is not PEP8-compliant:
c59c4376ad

* Line length adjustment

* Adjust asyncio reactor tests

* Merge ASYNCIO_ENABLED and TWISTED_REACTOR settings

* More docs about TWISTED_REACTOR

* Fix asyncio reactor test

* Docs: fix title

* Reword docs

* Check the TWISTED_REACTOR setting outside of the installing function

* Remove unrelated change

* Update scrapy/utils/log.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-06 22:42:34 +05:00
Andrey Rakhmatullin 489ffcda51 Add a test for an async item_scraped handler. 2020-02-06 22:40:11 +05:00
Vostretsov Nikita 4be19e443e name signla catcher in accord with signal name 2020-02-06 13:46:23 +00:00
Vostretsov Nikita 4bcc0933d9 Merge branch 'request_left_downloader_signal' of github.com:whalebot-helmsman/scrapy into request_left_downloader_signal 2020-02-06 13:45:00 +00:00
Vostretsov Nikita 4a91a5427d fix typo 2020-02-06 13:44:51 +00:00
Vostretsov Nikita 6733f4d976
Update docs/topics/signals.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-06 18:40:42 +05:00
Mikhail Korobov bbbb8f1418
Merge pull request #4304 from elacuesta/remove-six-from-tox-ini 
Remove elusive six occurrence from tox.ini
 2020-02-06 17:09:57 +05:00
Adrián Chaves 576663e5a7 Make METAREFRESH_IGNORE_TAGS an empty list by default 2020-02-06 10:43:20 +01:00
Respawnz c2cca36821
typo 2020-02-06 05:39:15 +08:00
Eugenio Lacuesta 11941c3244
Remove elusive six occurrence from tox.ini 2020-02-05 13:27:54 -03:00
Vostretsov Nikita 3769f75386 pep8 E302 2020-02-05 12:08:08 +00:00
Vostretsov Nikita aab39f6341 docummentation for new signal 2020-02-05 11:35:03 +00:00
Vostretsov Nikita 9916f6e556 tests for new signal 2020-02-05 11:32:54 +00:00
Vostretsov Nikita ae04174884 emit new signal 2020-02-05 11:32:31 +00:00
Vostretsov Nikita 898bc00811 new signal 2020-02-05 11:31:27 +00:00
Eugenio Lacuesta fbea370c58
Rename function parameter 2020-02-05 01:35:13 -03:00
Eugenio Lacuesta ad70497416
Remove unnecessary parentheses in class definition 2020-02-04 13:30:13 -03:00
Eugenio Lacuesta 13670f0397
Ignore tests/CrawlerRunner directory 2020-02-03 16:16:43 -03:00
Eugenio Lacuesta e0ef8ad2d6
CrawlerRunner test for Response.ip_address 2020-02-03 15:52:15 -03:00
Eugenio Lacuesta 4851efdfb0
Flake8 adjustments 2020-02-03 14:50:54 -03:00
Eugenio Lacuesta bb8f7dc609
Mock DNS server 2020-02-03 14:50:14 -03:00
Eugenio Lacuesta a2ae380efc
Remove unnecessary commas 2020-02-03 13:23:52 -03:00
Eugenio Lacuesta b9e3a6201b
Merge branch 'master' into response_ip_address 2020-02-03 04:07:44 -03:00
faizan2700 0641ba0faa SCRAPY_CHECK will be set while running contact 2020-02-02 16:54:22 +05:30
Mikhail Korobov 22f7934fcc
Merge pull request #4269 from wRAR/asyncio-parse 
Support for async def callbacks.
 2020-01-31 23:50:19 +05:00
Andrey Rakhmatullin 7687564c73 Merge remote-tracking branch 'origin/master' into asyncio-signals 2020-01-30 17:42:03 +05:00
Andrey Rakhmatullin cc825c21de Test returning items from an async def callback. 2020-01-30 16:17:06 +05:00
Mikhail Korobov 47b9de93a9
Merge pull request #4288 from petervandenabeele/patch-1 
[Docs] 2 typos + 1 clarification in docs
 2020-01-30 03:15:18 +05:00
Mikhail Korobov 534ce07b10
Merge pull request #4293 from edorofeev/fix-accept-encoding-delimiter 
[HttpCompressionMiddleware] fix delimiter for Accept-Encoding header
 2020-01-30 00:19:40 +05:00
Eugenio Lacuesta 6f02a8dccb
Add source parameter to bytes_received signal 2020-01-29 15:51:25 -03:00
Eugenio Lacuesta a499f38b14
Remove object parent class 2020-01-29 14:35:17 -03:00
Eugenio Lacuesta 2c9643d38c
Test: bytes_received signal fired multiple times 2020-01-29 14:17:06 -03:00
Evgeny Dorofeev 4e56571a19 [HttpCompressionMiddleware] fix delimiter for Accept-Encoding header 2020-01-29 15:49:22 +03:00
Daniel Kimsey 752e8f7018 FilesPipeline.file_path has optional arguments 
Documented signature doesn't match the actual interface in [files.py](https://github.com/scrapy/scrapy/blob/master/scrapy/pipelines/files.py#L520).

Specifically, it looks like it may be [called](https://github.com/scrapy/scrapy/blob/master/scrapy/pipelines/files.py#L422) without a response value.

I found this when I was implementing the pipeline with the signature `file_path(self, request, response, info)` and the following error was being return in my results :

    [(False, <twisted.python.failure.Failure builtins.TypeError: file_path() missing 1 required positional argument: 'response'>)]

Scrapy==1.8.0
 2020-01-28 15:21:15 -06:00
Kevin Lloyd Bernal a3b168948c Log an error when giving up requests after too many retries (#3566) 2020-01-28 21:53:25 +01:00
Eugenio Lacuesta ad4477d335
Remove unnecessary else 2020-01-27 14:16:43 -03:00
Eugenio Lacuesta 4ffd18fb11
[docs] Mention that signals.bytes_received could be fired multiple times 2020-01-27 13:29:33 -03:00
Eugenio Lacuesta 613fd41f44
bytes_received signal: improve test performance 2020-01-27 13:25:06 -03:00
Mikhail Korobov a6bf895474
Merge pull request #4286 from petervandenabeele/fix-documentation 
[Docs] Fix variable name `author_page_links`
 2020-01-27 20:57:22 +05:00
Eugenio Lacuesta dbe20a863f
bytes_received signal: send spider argument 2020-01-27 12:28:54 -03:00
Matsievskiy S.V c9d3652230 add zsh -h autocomplete option 2020-01-27 18:24:57 +03:00
Adrián Chaves 89483ce9f7
Fix Flake8 issues 2020-01-27 12:08:56 -03:00
Eugenio Lacuesta bda37e38bd
[Tests] bytes_received signal 2020-01-27 12:08:56 -03:00
Eugenio Lacuesta cab449b195
Typo fix 2020-01-27 12:08:34 -03:00
Eugenio Lacuesta 72b8613ee9
bytes_received signal (no tests) 2020-01-27 12:06:48 -03:00
Eugenio Lacuesta 8529dff41d
Update docs regarding Response.ip_address and IPv6 2020-01-26 18:00:56 -03:00
Eugenio Lacuesta e8da7e2966
Test DNS resolution using CrawlerProcess 2020-01-26 17:53:39 -03:00
Eugenio Lacuesta 80925ab845
Get server IP address for HTTP/1.1 responses 2020-01-26 16:57:04 -03:00
Peter Vandenabeele f72d4e93e6
[Docs] 2 typos + 1 clarification in docs 
Fixing 2 small typos and adding 1 word as clarification in the downloader-middlewares.

Also, I was confused with the entries like `ref:Reppy <reppy-parser>` and similar entries.
Are these supposed to be links to other parts of the doc, or is this the intended way of showing these references ?
 2020-01-26 10:48:28 +01:00