RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,472 Commits 49 Branches 132 Tags 153 MiB
Commit Graph

9472 Commits

Author SHA1 Message Date
Adrián Chaves 5ff9eb90ea Add a test for the copy of cb_kwargs from Request to Response 2020-02-13 22:36:18 +01:00
Adrián Chaves df937d8280 Implement Response.cb_kwargs 2020-02-13 22:33:36 +01:00
Adrián Chaves b4958358e8 Update tests to account for link extractors escaping spaces 2020-02-12 19:00:04 +01:00
Adrián Chaves e1be078eaa Fix Flake8-reported issues 2020-02-12 17:38:06 +01:00
Adrián Chaves 73e88d036c
Import mock from unittest 2020-02-12 17:17:38 +01:00
Drew Seibert 2d6d4fb233
Deprecate overriding settings with SCRAPY-prefixed environment variables (#4300) 2020-02-11 10:35:23 +01:00
Mikhail Korobov a6ef065eb5
Merge pull request #4271 from wRAR/asyncio-signals 
async def support for signal handlers that already supported Deferreds
 2020-02-11 02:05:45 +05:00
Adrián Chaves 61e74bac76 Extract links with safe_url_string 
canonicalize_url changes links in undesirable ways.
 2020-02-10 21:57:21 +01:00
Andrey Rakhmatullin 1f0f52cbf7 Improve async signal tests. 2020-02-11 01:05:45 +05:00
Andrey Rakhmatullin 36dcf90184 Also test non-default async callbacks. 2020-02-11 01:00:16 +05:00
Adrián Chaves 037ae5b22e
Explicitly indicate None as ip_address’s default value 2020-02-10 19:54:47 +01:00
Abhishek Pratap Singh 4626e90df8
Allow updating flags in follow and follow_all (#4279) 2020-02-10 19:48:31 +01:00
Adrián Chaves 96bb3b5144
Merge branch 'master' into feat-685 2020-02-10 19:45:09 +01:00
Adrián Chaves 7025c18b15
Clear line of spaces 2020-02-10 19:43:23 +01:00
Eugenio Lacuesta 13ba9bc629
Note about Response.ip_address 2020-02-10 12:29:39 -03:00
Eugenio Lacuesta 42b4e9b337
Reword signal docs 2020-02-10 11:23:38 -03:00
Eugenio Lacuesta 122ce6d6fb
Check bytes are received in order (bytes_received signal) 2020-02-10 10:20:26 -03:00
Eugenio Lacuesta a64fa2f086
Keyword arguments when creating a _ResponseReader 2020-02-10 10:16:05 -03:00
Adrián Chaves 35723d76c0 Use canonicalize_url in link extraction 2020-02-07 22:59:53 +01:00
Adrián Chaves c2f484de88
Merge branch 'master' into response_ip_address 2020-02-07 21:49:38 +01:00
Adrián Chaves 4c6993f264
Merge branch 'master' into asyncio-parse-asyncgen 2020-02-07 21:34:54 +01:00
Adrián Chaves aae4935605
Merge branch 'master' into spider.parse 2020-02-07 21:17:56 +01:00
Adrián Chaves 59653ebac6 Update installation instructions regarding Python 3 and virtual environments 2020-02-07 21:07:57 +01:00
Mikhail Korobov b0eaf114e5
Merge pull request #4197 from elacuesta/sphinx-twisted-api 
[Docs] Fix Twisted links
 2020-02-07 23:51:15 +05:00
Mikhail Korobov bd7780277c
Merge pull request #4275 from abhishekh2001/master 
Fixed artwork/README formatting
 2020-02-07 23:44:15 +05:00
Mikhail Korobov 7e341e0f6b
Merge pull request #4291 from seregaxvm/master 
add zsh -h autocomplete option
 2020-02-07 23:42:10 +05:00
Mikhail Korobov c3b690a5b5
Merge pull request #4290 from dekimsey/patch-1 
FilesPipeline.file_path has optional arguments
 2020-02-07 23:41:31 +05:00
Mikhail Korobov 957681bcfa
Merge pull request #4272 from elacuesta/spider-middleware 
Spider middleware: catch spider callback exceptions early
 2020-02-07 23:40:50 +05:00
Mikhail Korobov afbaf9d430
Merge pull request #4303 from whalebot-helmsman/request_left_downloader_signal 
request_left_downloader signal
 2020-02-07 23:33:51 +05:00
Mikhail Korobov 0f62e44def
Merge pull request #4316 from wRAR/asyncio-parse-request-tests 
Add a test for an async callbacks that returns requests.
 2020-02-07 23:22:19 +05:00
Andrey Rakhmatullin 7323780c97 Support yield in async def callbacks. 2020-02-07 21:32:45 +05:00
Andrey Rakhmatullin 31f6c7112f Add a test for an async callbacks that returns requests. 2020-02-07 17:14:52 +05:00
Andrey Rakhmatullin 4a7c7340a0 Merge remote-tracking branch 'origin/master' into asyncio-signals 2020-02-07 16:58:59 +05:00
Vostretsov Nikita 153b78e53f
Update docs/topics/signals.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-07 11:08:55 +05:00
Vostretsov Nikita 8817b9e8e9
Update docs/topics/signals.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-02-07 11:07:53 +05:00
Vostretsov Nikita 2f83f3e2cb
Update docs/topics/signals.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-02-07 11:07:43 +05:00
Vostretsov Nikita 84b55b7364
Update docs/topics/signals.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-02-07 11:07:35 +05:00
Eugenio Lacuesta 6c333491a8
Merge branch 'master' into response_ip_address 2020-02-06 18:53:11 -03:00
Joy Bhalla 4f31c3ce01
Document a backward incompatibility that may affect custom schedulers (#4274) 2020-02-06 22:21:33 +01:00
Eugenio Lacuesta 7f2d3051fe
Fix Flake8 issue 2020-02-06 18:19:40 -03:00
Lane Shaw 3263441fbc
Update RFPDupeFilter line separator for correct universal newlines mode usage (#4283) 2020-02-06 22:14:40 +01:00
Adrián Chaves 24bb9fd5f7
Merge branch 'master' into spider.parse 2020-02-06 21:39:09 +01:00
Mikhail Korobov 042e71e2b8
Merge pull request #4311 from Gallaecio/metarefresh-ignore-tags 
Make METAREFRESH_IGNORE_TAGS an empty list by default
 2020-02-06 23:40:45 +05:00
Mikhail Korobov 8d2705f23c
Merge pull request #4305 from Respawnz/patch-1 
fix a typo in devloper-tools.rst
 2020-02-06 23:17:28 +05:00
elacuesta 35dafef7f1
Specify Twisted reactor (TWISTED_REACTOR setting) (#4294) 
* Add the ability to install a specific reactor

* Add docs for the TWISTED_REACTOR setting

* Add tests for the TWISTED_REACTOR setting

* Update asyncio reactor test

* Ignore W503 globally

W503 is not PEP8-compliant:
c59c4376ad

* Line length adjustment

* Adjust asyncio reactor tests

* Merge ASYNCIO_ENABLED and TWISTED_REACTOR settings

* More docs about TWISTED_REACTOR

* Fix asyncio reactor test

* Docs: fix title

* Reword docs

* Check the TWISTED_REACTOR setting outside of the installing function

* Remove unrelated change

* Update scrapy/utils/log.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-06 22:42:34 +05:00
Andrey Rakhmatullin 489ffcda51 Add a test for an async item_scraped handler. 2020-02-06 22:40:11 +05:00
Vostretsov Nikita 4be19e443e name signla catcher in accord with signal name 2020-02-06 13:46:23 +00:00
Vostretsov Nikita 4bcc0933d9 Merge branch 'request_left_downloader_signal' of github.com:whalebot-helmsman/scrapy into request_left_downloader_signal 2020-02-06 13:45:00 +00:00
Vostretsov Nikita 4a91a5427d fix typo 2020-02-06 13:44:51 +00:00
Vostretsov Nikita 6733f4d976
Update docs/topics/signals.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-06 18:40:42 +05:00