RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,290 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

1847 Commits

Author SHA1 Message Date
Anubhav Patel ad824a264b fixes a link in doc 2019-08-27 18:30:11 +05:30
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin ce281d890d Documentation for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:53:19 +05:00
Marc Hernández d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Adrián Chaves a12e8251e0 Cover Scrapy 1.7.3 in the release notes 2019-08-01 17:10:31 +02:00
Adrián Chaves 04bca6af7c
Merge pull request #3894 from KristobalJunta/fix_retry_docs 
fix default RETRY_HTTP_CODES value in docs
 2019-07-29 18:20:55 +02:00
Mikhail Korobov 9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Mikhail Korobov c679aefe85
Merge pull request #3660 from anubhavp28/logFormatter-doc-patch 
[MRG+1] docs for scrapy.logformatter
 2019-07-24 01:40:56 +05:00
Adrián Chaves 7843101f9a Cover Scrapy 1.7.2 in the release notes 2019-07-23 13:35:27 +02:00
Kristobal Junta 43d5b5a524 fix default RETRY_HTTP_CODES value in docs 2019-07-22 10:24:14 +03:00
Adrián Chaves b8a43011e7 Cover Scrapy 1.7.1 in the release notes 2019-07-19 18:17:24 +02:00
Andrey Rakhmatullin 42743fd9dd Move tls_verbose_logging extraction from __init__ to from_settings. 2019-07-18 20:49:25 +05:00
Adrián Chaves ae4eab9843 Cover the 1.7.1 PyPI repackaging in the release notes 2019-07-18 17:02:48 +02:00
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Deschner, Magdalena c44d49b238 minor PEP8 style changes 2019-07-17 13:13:52 +02:00
Deschner, Magdalena 6660020ebb remove detailed description about individual settings 2019-07-17 11:30:02 +02:00
Deschner, Magdalena e892a484e8 add instructions about how to define output file when running scrapy from script instead of cmd 2019-07-16 13:53:56 +02:00
Adrián Chaves fa6a0d799b
Merge pull request #3684 from Gallaecio/remove-ubuntu-topic 
Remove docs/topics/ubuntu.rst
 2019-07-12 14:03:11 +02:00
Adrián Chaves ed487b7db3 broad-crawls.rst: Refactor the memory usage section 2019-07-11 11:14:22 +02:00
Andrey Rakhmatullin 98689b27a8 Improve the DOWNLOADER_CLIENTCONTEXTFACTORY doc. 2019-07-11 14:02:35 +05:00
Robin 5cdf277017 Update broad-crawls.rst 
Added section on how to treat memory consumption problems of broad crawls.
 2019-07-11 08:49:24 +02:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
Mikhail Korobov 050a62a955
Merge pull request #3861 from Gallaecio/documentation-coverage 
Increase the documentation coverage of exceptions
 2019-07-10 10:18:34 +05:00
Eugenio Lacuesta 9fb0f8454e
Extend docs about Crawling Rules 2019-07-09 15:30:22 -03:00
Eugenio Lacuesta d04e84c11d Merge remote-tracking branch 'upstream/master' into docs_crawlspider_link_text 2019-07-09 15:24:31 -03:00
Adrián Chaves a0bbbe8ffa Add the API documentation of ContractFail to the contracts topic 2019-07-09 12:33:57 +02:00
Adrián Chaves 578bccf3bb Skip scrapy.exceptions.UsageError in the documentation coverage report 2019-07-09 12:33:07 +02:00
Mikhail Korobov a97a9b57bf
Merge pull request #3621 from Gallaecio/bfo-for-few-requests 
Document that the crawl order is BFO for small numbers of start requests
 2019-07-08 21:26:04 +05:00
Mikhail Korobov dda12b6537
Merge pull request #3672 from Gallaecio/split-items 
Add a FAQ entry about splitting items in item pipelines
 2019-07-08 20:15:23 +05:00
Adrián Chaves a0b09e0193 Skip scrapy.downloadermiddlewares private APIs in the documentation coverage report 2019-07-08 14:45:17 +02:00
Andrey Rakhmatullin 0b9dce3a6c Add DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING setting. 2019-07-08 17:40:56 +05:00
Adrián Chaves 783de8a8d8 Document how request concurrency settings impact request order 2019-07-08 14:23:21 +02:00
Adrián Chaves 6ea62cacc7 Add a FAQ entry about splitting items in item pipelines 2019-07-08 13:58:55 +02:00
Mikhail Korobov 4d4bd0e823
Merge pull request #3609 from Gallaecio/2253 
Document FilesPipeline.file_path and ImagesPipeline.file_path
 2019-07-04 22:05:46 +05:00
Mikhail Korobov 4c755f1ddc
Merge pull request #3692 from anubhavp28/doc-patch 
[MRG+1] doc for creating custom cache storage backend.
 2019-07-02 20:18:14 +05:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Mikhail Korobov e5f12faf5f
Merge pull request #3703 from Gallaecio/ajax-docs 
Add a topic about reaching data that selectors cannot reach
 2019-06-25 23:11:59 +03:00
Adrián Chaves d7795484dc Document scrapy.crawler.Crawler.stop() 2019-06-25 17:29:49 +02:00
Adrián Chaves 8a3b15eb91 Document how to select dynamically-loaded content 2019-06-24 16:06:21 +02:00
Daniel Graña cdfb20aee8
Merge pull request #3739 from Matthijsy/feature/scrapy_check_env 
[MRG+1] Add SCRAPY_CHECK environment variable
 2019-06-13 19:43:01 -03:00
Daniel Graña 8a022ac6ea
Merge pull request #3810 from Gallaecio/documentation-coverage 
Skip scrapy.contracts private APIs in the documentation coverage report
 2019-06-13 17:16:07 -03:00
Adrián Chaves c7ba72b5dc Skip scrapy.contracts private APIs in the documentation coverage report 2019-06-07 13:14:33 +02:00
Artem Kuchumov c81e15ed6e
Tutorial: scrapy shell example should say "text" not "title" (#3807) 
Tutorial: scrapy shell example should say "text" not "title"
 2019-06-05 13:15:23 +05:00
Mikhail Korobov f755507d68
Merge pull request #3567 from Gallaecio/documentation-coverage 
Add a Sphinx extension to generate documentation coverage information
 2019-06-04 18:47:24 +05:00
Adrián Chaves c7b5ad0e20 Add a Sphinx extension to generate documentation coverage information 2019-06-04 15:03:25 +02:00
Adrián Chaves ea209a0ea7 Fix module double indexing issues in the documentation 2019-06-04 14:35:27 +02:00