RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,947 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

1351 Commits

Author SHA1 Message Date
Andrey Rakhmatullin ed34ce14c0 Add the ASYNCIO_SUPPORT setting, reshuffle other logic accordingly. 2019-12-04 21:32:56 +05:00
Mikhail Korobov 9b7452211a
Merge pull request #4099 from BurnzZ/itemloader-docs 
update docs of scrapy.loader.ItemLoader.item
 2019-12-03 13:14:45 +05:00
Adrián Chaves 54b056c4be Make developer-tools doctests pass 2019-11-25 12:32:35 +01:00
Eugenio Lacuesta 40b5cfc0a4
Item loaders: allow single-argument processors (unbound methods) 2019-11-22 20:47:22 -03:00
Eugenio Lacuesta 6fa8f06b64
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-22 13:05:06 -03:00
Adrián Chaves f251dda268 Make debug doctests pass 2019-11-21 11:59:10 +01:00
Mikhail Korobov f1fd7ec318
Merge pull request #4171 from Gallaecio/doctest 
Make command doctests pass
 2019-11-20 19:47:22 +05:00
Eugenio Lacuesta 48223c107a
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-20 09:41:42 -03:00
Adrián Chaves e84cb18ca0 Use InterSphinx to link to the Twisted documentation 2019-11-18 15:50:45 +01:00
Adrián Chaves 74589df02f Make command doctests pass 2019-11-18 14:51:44 +01:00
Andrey Rakhmatullin dd367438fa Improve the dbm module ref. 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin a9c891399d Fix a duplicate ref name in docs. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 1d7c8cb0b1 Remove six.PY2 and six.PY3 conditionals. 2019-11-14 13:45:37 +05:00
Andrey Rahmatullin c911e80209
Merge pull request #4088 from ammarnajjar/4086-constructor-initializer-docs 
docs: use __init__ method instead of constructor
 2019-11-12 21:48:16 +05:00
Andrey Rahmatullin d8098350f0
Merge pull request #3960 from thernstig/issue_2149 
Update documentation for logging manually
 2019-11-12 16:17:49 +05:00
Adrián Chaves aef98188fa Improve the details about request serialization requirements for JOBDIR 2019-11-07 18:06:55 +01:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
Adrián Chaves 94f060fcc8
Cover Scrapy 1.8.0 in the release notes (#3952) 2019-10-29 12:53:46 +01:00
Andrey Rahmatullin 18b808b2e9
Merge pull request #4092 from further-reading/master 
Add Python 3.8 official support
 2019-10-29 16:30:58 +05:00
Adrián Chaves 3d0df419c4 Mark the LevelDB storage backend as deprecated 2019-10-28 11:24:47 +01:00
elacuesta 7731814cc2 ItemLoader: improve handling of initial item (#4036) 2019-10-28 10:53:53 +01:00
Kevin Lloyd Bernal 84fe4011b0 update docs of scrapy.loader.ItemLoader.item 2019-10-23 20:39:53 +08:00
Ammar Najjar d21e1034f0
docs: correct point,comma and plural replacements 
Issue #4086
 2019-10-22 13:29:38 +02:00
WinterComes 5d75ed4cba Remove an old note about contracts (#4093) 2019-10-22 12:19:07 +02:00
Ammar Najjar da8cd9448d
docs: always surround __init__ with `` in docs 
Issue #4086
 2019-10-21 19:56:58 +02:00
Ammar Najjar 68a7d05ed8
docs: use __init__ method instead of constructor 
Issue #4086
 2019-10-21 15:42:24 +02:00
Adrián Chaves 0fbd1ff4a9 constructor → __init__ method 2019-10-21 14:06:45 +02:00
Mikhail Korobov 8cb53441b0
Merge pull request #4056 from Gallaecio/documentation-build 
Fix internal links in the tutorial and release notes
 2019-10-19 00:00:02 +02:00
Bulat 84be6a941e Refactor sentence. 2019-10-16 14:04:07 +03:00
Bulat c9614a5bdd Fixed BOT_NAME documentation 2019-10-16 12:07:19 +03:00
Adrián Chaves d72ed46fe0 Improve how extra Item API members are introduced in the documentation 2019-10-15 16:04:26 +02:00
Baron Hou 7b1e69dec4 reponse → response (#4079) 2019-10-15 14:51:15 +02:00
Eugenio Lacuesta 5f168cd459
Response.follow_all 2019-10-08 16:32:56 -03:00
Adrián Chaves a4aa5b8926 Fix internal links in the tutorial and release notes 2019-10-02 14:51:06 +02:00
elacuesta 6ad5a89cb0 [Doc] Use autoclass in topics/request-response.rst (#4055) 2019-10-02 12:18:36 +02:00
Mikhail Korobov 74b4a5c77c
Merge pull request #4006 from whalebot-helmsman/protego-default 
Use protego as a default robots.txt parser
 2019-10-02 00:03:50 +05:00
Kristobal Junta c232bbdc42 fix typo in docs/topics/spiders 2019-10-01 17:41:38 +03:00
OmarFarrag 175cd2ece5
Update docs/topics/media-pipeline.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-10-01 07:27:31 +02:00
Adrián Chaves 7f4f98fd38 Provide complete API documentation coverage of scrapy.linkextractors 2019-09-30 18:22:28 +02:00
OmarFarrag 28005b2872
Update media-pipeline.rst 2019-09-28 06:21:14 +02:00
Mikhail Korobov 31c631f9ee
Merge pull request #3999 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.item
 2019-09-25 16:13:13 +01:00
Adrián Chaves 1236e9e81e Provide complete API documentation coverage of scrapy.item 2019-09-25 13:24:56 +02:00
Mikhail Korobov 17c0d72d04
Merge pull request #4016 from elacuesta/rule_default_link_extractor 
Crawling rules: make link extractors optional
 2019-09-20 15:27:17 +01:00
OmarFarrag b14c3cb612
Add media pipelines FTP documentation 2019-09-19 23:33:57 +02:00
Adrián Chaves c26a9015ad Clarify the effects of dont_merge_cookies 2019-09-19 11:08:06 +02:00
Adrián Chaves 9b65f9aa5b
Fix the item exporter example (#4022) 2019-09-19 09:17:23 +02:00
Vostretsov Nikita 2438ac529a
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-09-17 12:27:22 +05:00
Vostretsov Nikita d1d0bf8491
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-09-17 12:27:12 +05:00
Vostretsov Nikita 57e6f4c750 add link to performance comparison 2019-09-17 07:18:37 +00:00
Vostretsov Nikita d39ef77e6b add link to google description of lenght-based rule 2019-09-17 06:34:33 +00:00
Vostretsov Nikita b792dba528 remove periods 2019-09-17 06:28:33 +00:00
Vostretsov Nikita 50c85e3114 Merge branch 'protego-default' of github.com:whalebot-helmsman/scrapy into protego-default 2019-09-16 14:24:31 +00:00
Vostretsov Nikita 5197b39e01 fix capitalization, remove commas 2019-09-16 14:24:25 +00:00
Eugenio Lacuesta 21ad8e20b9
Crawling rules: make link extractors optional 2019-09-13 17:32:42 -03:00
Vostretsov Nikita 66145b4eaf
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
 2019-09-12 18:51:00 +03:00
Vostretsov Nikita 171fa1cd10 documentation rework 2019-09-10 09:59:36 +00:00
Vostretsov Nikita 6bd88711f2 update documentation 2019-09-10 08:55:37 +00:00
Vostretsov Nikita 38828d3fd4
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2019-09-09 17:04:13 +03:00
Vostretsov Nikita 9578f49099 use protego as a default robots.txt parser 2019-09-09 07:36:55 +00:00
Mikhail Korobov 534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
Eugenio Lacuesta 2061f2a382
[doc] cb_kwargs contract 2019-08-31 02:10:18 -03:00
Adrián Chaves 2828cb769f Provide complete API documentation coverage of scrapy.extensions 2019-08-30 14:29:15 +02:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Mikhail Korobov 93d4b0b0d7
Merge pull request #3973 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.exporters
 2019-08-28 21:59:10 +05:00
Anubhav Patel 77c8ab2e62 makes suggested changes 2019-08-27 18:44:08 +05:30
Anubhav Patel ad824a264b fixes a link in doc 2019-08-27 18:30:11 +05:30
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
Adrián Chaves 0fa384e80d Provide complete API documentation coverage of scrapy.exporters 2019-08-22 20:10:42 +02:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Tobias Hernstig 2b0de0606c Fix merge conflicts 2019-08-15 18:54:28 +02:00
Tobias Hernstig 50c4cafe0c Update documentation for logging manually 
Usage of basicConfig() together with crawlerRunner is not recommended.
Update documentation to highlight this fact.

Closes #2149, #2352, #3146
 2019-08-15 18:46:00 +02:00
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin ce281d890d Documentation for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:53:19 +05:00
Marc Hernández d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Adrián Chaves 04bca6af7c
Merge pull request #3894 from KristobalJunta/fix_retry_docs 
fix default RETRY_HTTP_CODES value in docs
 2019-07-29 18:20:55 +02:00
Mikhail Korobov 9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Mikhail Korobov c679aefe85
Merge pull request #3660 from anubhavp28/logFormatter-doc-patch 
[MRG+1] docs for scrapy.logformatter
 2019-07-24 01:40:56 +05:00
Kristobal Junta 43d5b5a524 fix default RETRY_HTTP_CODES value in docs 2019-07-22 10:24:14 +03:00
Andrey Rakhmatullin 42743fd9dd Move tls_verbose_logging extraction from __init__ to from_settings. 2019-07-18 20:49:25 +05:00
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Deschner, Magdalena c44d49b238 minor PEP8 style changes 2019-07-17 13:13:52 +02:00
Deschner, Magdalena 6660020ebb remove detailed description about individual settings 2019-07-17 11:30:02 +02:00
Deschner, Magdalena e892a484e8 add instructions about how to define output file when running scrapy from script instead of cmd 2019-07-16 13:53:56 +02:00
Adrián Chaves fa6a0d799b
Merge pull request #3684 from Gallaecio/remove-ubuntu-topic 
Remove docs/topics/ubuntu.rst
 2019-07-12 14:03:11 +02:00
Adrián Chaves ed487b7db3 broad-crawls.rst: Refactor the memory usage section 2019-07-11 11:14:22 +02:00
Andrey Rakhmatullin 98689b27a8 Improve the DOWNLOADER_CLIENTCONTEXTFACTORY doc. 2019-07-11 14:02:35 +05:00
Robin 5cdf277017 Update broad-crawls.rst 
Added section on how to treat memory consumption problems of broad crawls.
 2019-07-11 08:49:24 +02:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
Mikhail Korobov 050a62a955
Merge pull request #3861 from Gallaecio/documentation-coverage 
Increase the documentation coverage of exceptions
 2019-07-10 10:18:34 +05:00
Eugenio Lacuesta 9fb0f8454e
Extend docs about Crawling Rules 2019-07-09 15:30:22 -03:00
Eugenio Lacuesta d04e84c11d Merge remote-tracking branch 'upstream/master' into docs_crawlspider_link_text 2019-07-09 15:24:31 -03:00
Adrián Chaves a0bbbe8ffa Add the API documentation of ContractFail to the contracts topic 2019-07-09 12:33:57 +02:00
Andrey Rakhmatullin 0b9dce3a6c Add DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING setting. 2019-07-08 17:40:56 +05:00
Adrián Chaves 6ea62cacc7 Add a FAQ entry about splitting items in item pipelines 2019-07-08 13:58:55 +02:00
Mikhail Korobov 4d4bd0e823
Merge pull request #3609 from Gallaecio/2253 
Document FilesPipeline.file_path and ImagesPipeline.file_path
 2019-07-04 22:05:46 +05:00
Mikhail Korobov 4c755f1ddc
Merge pull request #3692 from anubhavp28/doc-patch 
[MRG+1] doc for creating custom cache storage backend.
 2019-07-02 20:18:14 +05:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00