RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,015 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

1379 Commits

Author SHA1 Message Date
Daniel Kimsey 752e8f7018 FilesPipeline.file_path has optional arguments 
Documented signature doesn't match the actual interface in [files.py](https://github.com/scrapy/scrapy/blob/master/scrapy/pipelines/files.py#L520).

Specifically, it looks like it may be [called](https://github.com/scrapy/scrapy/blob/master/scrapy/pipelines/files.py#L422) without a response value.

I found this when I was implementing the pipeline with the signature `file_path(self, request, response, info)` and the following error was being return in my results :

    [(False, <twisted.python.failure.Failure builtins.TypeError: file_path() missing 1 required positional argument: 'response'>)]

Scrapy==1.8.0
 2020-01-28 15:21:15 -06:00
Eugenio Lacuesta 8529dff41d
Update docs regarding Response.ip_address and IPv6 2020-01-26 18:00:56 -03:00
Eugenio Lacuesta 80925ab845
Get server IP address for HTTP/1.1 responses 2020-01-26 16:57:04 -03:00
Peter Vandenabeele f72d4e93e6
[Docs] 2 typos + 1 clarification in docs 
Fixing 2 small typos and adding 1 word as clarification in the downloader-middlewares.

Also, I was confused with the entries like `ref:Reppy <reppy-parser>` and similar entries.
Are these supposed to be links to other parts of the doc, or is this the intended way of showing these references ?
 2020-01-26 10:48:28 +01:00
Mikhail Korobov 8b8df31961
Merge pull request #4227 from elacuesta/name-resolver 
Name resolver with IPv6 support
 2020-01-25 05:03:13 +05:00
OmarFarrag fc98aa6b67
Merge branch 'master' into ftp_files#3928 2020-01-24 14:52:40 +02:00
Mikhail Korobov 6a98d660e5
Merge pull request #3551 from jpbalarini/change_scraper_slot 
[MRG+1] Add ability to change max_active_size by setting
 2020-01-23 23:40:05 +05:00
JP Balarini 0f2d871d88 Use PEP 515 style for SCRAPER_SLOT_MAX_ACTIVE_SIZE documentation 2020-01-20 11:28:28 -03:00
Juan Pablo Balarini eaa8ed02d0 Add ability to change max_active_size by settings 2020-01-20 11:27:58 -03:00
Eugenio Lacuesta 302d3f552b
[doc] DNS_RESOLVER setting 2020-01-18 01:46:24 -03:00
Eugenio Lacuesta 2e405d2d5c
Merge branch 'master' into response_follow_all 2020-01-05 00:33:19 -03:00
1um0s 14d4428e70 Rephrasing documentation for image and file pipelines (#4252) 
* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Revert image, file pipeline docs. Enhance custom media pipeline docs.

* scrapy#4034 rebase master

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Revert image, file pipeline docs. Enhance custom media pipeline docs.

* scrapy#4034 rebase master

* Rebase master

* Add class to media pipeline docs

Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>

Co-authored-by: elacuesta <elacuesta@users.noreply.github.com>
 2019-12-30 00:56:22 +05:00
Andrey Rakhmatullin dc1ee09481 Rename ASYNCIO_ENABLED to ASYNCIO_REACTOR, change the logic accordingly. 2019-12-27 21:56:28 +05:00
Andrey Rakhmatullin 20289be810 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-20 19:31:20 +05:00
Mikhail Korobov fb3fb17c28
Merge pull request #4045 from Gallaecio/documentation-coverage 
Deprecate FilteringLinkExtractor direct usage; add API docs for scrapy.linkextractors
 2019-12-20 02:01:45 +05:00
Mikhail Korobov 6a32a36e09
Merge pull request #4143 from Gallaecio/undent-rst-doctests 
Do not indent doctests from the documentation unnecessarily
 2019-12-20 01:59:07 +05:00
Adrián Chaves b4a2d985a7
Merge branch 'master' into undent-rst-doctests 2019-12-19 12:39:17 +01:00
Marc Hernandez Cabot 16b363de31 Merge branch 'master' into fix_typos 2019-12-19 10:02:21 +01:00
Marc Hernandez Cabot 23a67cec27 fix first letter capitalization for Raring and Scrapy 2019-12-19 09:57:17 +01:00
Mikhail Korobov 8f7faaa63d
Merge pull request #4190 from Gallaecio/doctest 
Make developer-tools doctests pass
 2019-12-18 23:40:21 +05:00
Marc Hernandez Cabot c0d84f0962 fix typos 2019-12-18 19:39:21 +01:00
elacuesta 916382e109 Add errback parameter to scrapy.spiders.crawl.Rule (#4000) 
* Add errback parameter to scrapy.spiders.crawl.Rule

* CrawlSpider: optimize by reducing iterations

* [test] Rule.errback

* [doc] Rule.errback

* [doc] Use autoclass in docs/topics/spiders.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Rule.process_links takes a list

* Fix aesthetic issue reported by Flake8
 2019-12-18 20:05:33 +05:00
Andrey Rakhmatullin 3560123090 Rename ASYNCIO_SUPPORT to ASYNCIO_ENABLED. 2019-12-13 18:07:47 +05:00
Eugenio Lacuesta 21a2f8c6f8
Merge branch 'master' into response_follow_all 2019-12-10 17:30:59 -03:00
Adrián Chaves 1fc2b140c1
Merge branch 'master' into documentation-coverage 2019-12-05 14:43:36 +01:00
Adrián Chaves 83b8046fdc Do not indent doctests from the documentation unnecessarily 2019-12-05 13:52:31 +01:00
Andrey Rakhmatullin ed34ce14c0 Add the ASYNCIO_SUPPORT setting, reshuffle other logic accordingly. 2019-12-04 21:32:56 +05:00
Mikhail Korobov 9b7452211a
Merge pull request #4099 from BurnzZ/itemloader-docs 
update docs of scrapy.loader.ItemLoader.item
 2019-12-03 13:14:45 +05:00
Adrián Chaves 54b056c4be Make developer-tools doctests pass 2019-11-25 12:32:35 +01:00
Eugenio Lacuesta 40b5cfc0a4
Item loaders: allow single-argument processors (unbound methods) 2019-11-22 20:47:22 -03:00
Eugenio Lacuesta 6fa8f06b64
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-22 13:05:06 -03:00
Adrián Chaves f251dda268 Make debug doctests pass 2019-11-21 11:59:10 +01:00
Mikhail Korobov f1fd7ec318
Merge pull request #4171 from Gallaecio/doctest 
Make command doctests pass
 2019-11-20 19:47:22 +05:00
Eugenio Lacuesta 48223c107a
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-20 09:41:42 -03:00
Adrián Chaves e84cb18ca0 Use InterSphinx to link to the Twisted documentation 2019-11-18 15:50:45 +01:00
Adrián Chaves 74589df02f Make command doctests pass 2019-11-18 14:51:44 +01:00
Andrey Rakhmatullin dd367438fa Improve the dbm module ref. 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin a9c891399d Fix a duplicate ref name in docs. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 1d7c8cb0b1 Remove six.PY2 and six.PY3 conditionals. 2019-11-14 13:45:37 +05:00
Andrey Rahmatullin c911e80209
Merge pull request #4088 from ammarnajjar/4086-constructor-initializer-docs 
docs: use __init__ method instead of constructor
 2019-11-12 21:48:16 +05:00
Andrey Rahmatullin d8098350f0
Merge pull request #3960 from thernstig/issue_2149 
Update documentation for logging manually
 2019-11-12 16:17:49 +05:00
Adrián Chaves aef98188fa Improve the details about request serialization requirements for JOBDIR 2019-11-07 18:06:55 +01:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
Adrián Chaves 94f060fcc8
Cover Scrapy 1.8.0 in the release notes (#3952) 2019-10-29 12:53:46 +01:00
Andrey Rahmatullin 18b808b2e9
Merge pull request #4092 from further-reading/master 
Add Python 3.8 official support
 2019-10-29 16:30:58 +05:00
Adrián Chaves 3d0df419c4 Mark the LevelDB storage backend as deprecated 2019-10-28 11:24:47 +01:00
elacuesta 7731814cc2 ItemLoader: improve handling of initial item (#4036) 2019-10-28 10:53:53 +01:00
Kevin Lloyd Bernal 84fe4011b0 update docs of scrapy.loader.ItemLoader.item 2019-10-23 20:39:53 +08:00
Adrián Chaves 5479e7ecc7 Indicate that lists of emails may be provided as a single string or as a list of strings 2019-10-22 15:24:44 +02:00
Ammar Najjar d21e1034f0
docs: correct point,comma and plural replacements 
Issue #4086
 2019-10-22 13:29:38 +02:00
WinterComes 5d75ed4cba Remove an old note about contracts (#4093) 2019-10-22 12:19:07 +02:00
Ammar Najjar da8cd9448d
docs: always surround __init__ with `` in docs 
Issue #4086
 2019-10-21 19:56:58 +02:00
Adrián Chaves ad6075440c Fix references to Python types in parameter type fields 2019-10-21 19:00:03 +02:00
Ammar Najjar 68a7d05ed8
docs: use __init__ method instead of constructor 
Issue #4086
 2019-10-21 15:42:24 +02:00
Adrián Chaves 0fbd1ff4a9 constructor → __init__ method 2019-10-21 14:06:45 +02:00
Mikhail Korobov 8cb53441b0
Merge pull request #4056 from Gallaecio/documentation-build 
Fix internal links in the tutorial and release notes
 2019-10-19 00:00:02 +02:00
Bulat 84be6a941e Refactor sentence. 2019-10-16 14:04:07 +03:00
Bulat c9614a5bdd Fixed BOT_NAME documentation 2019-10-16 12:07:19 +03:00
Adrián Chaves d72ed46fe0 Improve how extra Item API members are introduced in the documentation 2019-10-15 16:04:26 +02:00
Baron Hou 7b1e69dec4 reponse → response (#4079) 2019-10-15 14:51:15 +02:00
Eugenio Lacuesta 5f168cd459
Response.follow_all 2019-10-08 16:32:56 -03:00
Adrián Chaves a4aa5b8926 Fix internal links in the tutorial and release notes 2019-10-02 14:51:06 +02:00
elacuesta 6ad5a89cb0 [Doc] Use autoclass in topics/request-response.rst (#4055) 2019-10-02 12:18:36 +02:00
Mikhail Korobov 74b4a5c77c
Merge pull request #4006 from whalebot-helmsman/protego-default 
Use protego as a default robots.txt parser
 2019-10-02 00:03:50 +05:00
Kristobal Junta c232bbdc42 fix typo in docs/topics/spiders 2019-10-01 17:41:38 +03:00
OmarFarrag 175cd2ece5
Update docs/topics/media-pipeline.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-10-01 07:27:31 +02:00
Adrián Chaves 7f4f98fd38 Provide complete API documentation coverage of scrapy.linkextractors 2019-09-30 18:22:28 +02:00
OmarFarrag 28005b2872
Update media-pipeline.rst 2019-09-28 06:21:14 +02:00
Mikhail Korobov 31c631f9ee
Merge pull request #3999 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.item
 2019-09-25 16:13:13 +01:00
Adrián Chaves 1236e9e81e Provide complete API documentation coverage of scrapy.item 2019-09-25 13:24:56 +02:00
Mikhail Korobov 17c0d72d04
Merge pull request #4016 from elacuesta/rule_default_link_extractor 
Crawling rules: make link extractors optional
 2019-09-20 15:27:17 +01:00
OmarFarrag b14c3cb612
Add media pipelines FTP documentation 2019-09-19 23:33:57 +02:00
Adrián Chaves c26a9015ad Clarify the effects of dont_merge_cookies 2019-09-19 11:08:06 +02:00
Adrián Chaves 9b65f9aa5b
Fix the item exporter example (#4022) 2019-09-19 09:17:23 +02:00
Vostretsov Nikita 2438ac529a
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-09-17 12:27:22 +05:00
Vostretsov Nikita d1d0bf8491
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-09-17 12:27:12 +05:00
Vostretsov Nikita 57e6f4c750 add link to performance comparison 2019-09-17 07:18:37 +00:00
Vostretsov Nikita d39ef77e6b add link to google description of lenght-based rule 2019-09-17 06:34:33 +00:00
Vostretsov Nikita b792dba528 remove periods 2019-09-17 06:28:33 +00:00
Vostretsov Nikita 50c85e3114 Merge branch 'protego-default' of github.com:whalebot-helmsman/scrapy into protego-default 2019-09-16 14:24:31 +00:00
Vostretsov Nikita 5197b39e01 fix capitalization, remove commas 2019-09-16 14:24:25 +00:00
Eugenio Lacuesta 21ad8e20b9
Crawling rules: make link extractors optional 2019-09-13 17:32:42 -03:00
Vostretsov Nikita 66145b4eaf
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
 2019-09-12 18:51:00 +03:00
Vostretsov Nikita 171fa1cd10 documentation rework 2019-09-10 09:59:36 +00:00
Vostretsov Nikita 6bd88711f2 update documentation 2019-09-10 08:55:37 +00:00
Vostretsov Nikita 38828d3fd4
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2019-09-09 17:04:13 +03:00
Vostretsov Nikita 9578f49099 use protego as a default robots.txt parser 2019-09-09 07:36:55 +00:00
Mikhail Korobov 534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
Eugenio Lacuesta 2061f2a382
[doc] cb_kwargs contract 2019-08-31 02:10:18 -03:00
Adrián Chaves 2828cb769f Provide complete API documentation coverage of scrapy.extensions 2019-08-30 14:29:15 +02:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Mikhail Korobov 93d4b0b0d7
Merge pull request #3973 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.exporters
 2019-08-28 21:59:10 +05:00
Anubhav Patel 77c8ab2e62 makes suggested changes 2019-08-27 18:44:08 +05:30
Anubhav Patel ad824a264b fixes a link in doc 2019-08-27 18:30:11 +05:30
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
Adrián Chaves 0fa384e80d Provide complete API documentation coverage of scrapy.exporters 2019-08-22 20:10:42 +02:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Tobias Hernstig 2b0de0606c Fix merge conflicts 2019-08-15 18:54:28 +02:00
Tobias Hernstig 50c4cafe0c Update documentation for logging manually 
Usage of basicConfig() together with crawlerRunner is not recommended.
Update documentation to highlight this fact.

Closes #2149, #2352, #3146
 2019-08-15 18:46:00 +02:00