RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,758 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

1634 Commits

Author SHA1 Message Date
Hugo 44623687ab Drop support for EOL Python 3.3 2017-12-19 17:59:05 +02:00
Mikhail Korobov 9aa9dd8d45 DOC mention an easier way to track pull requests locally. 
Thanks @eliasdorneles!
 2017-12-12 19:17:00 +05:00
Mikhail Korobov f716843a66 DOC update "Contributing" docs: 
* suggest Stack Overflow for Scrapy usage questions;
* encourage users to submit test-only pull requests with reproducable examples;
* encourage users to pick up stalled pull requests;
* we don't use AUTHORS file as a main acknowledgement source;
* suggest using Sphinx autodocs extension
 2017-12-12 16:19:43 +05:00
Daniel Graña 3cf0332ec3
Merge pull request #2957 from ScrapingLab/add_meta_json_to_parse_command 
[MRG+1] Scrapy Command: add --meta/-m to the "parse" command to pass additional meta data into the request
 2017-11-29 16:26:48 -03:00
IAlwaysBeCoding 6af323d7c8
Fix spelling mistake on scrapy parse command docs 
Fixed spelling mistake from "will be pass" to "will be passed"
 2017-11-26 00:24:52 +01:00
Jesse Bakker 0b14cb44aa Added from_crawler to middleware docs 2017-11-23 15:25:43 +01:00
IAlwaysBeCoding 846fd83512 removed commented out code, wrapped line to pep-8 and removed backlashes 2017-11-11 18:30:01 -05:00
Daniel Graña b8870ee8a1
Merge pull request #2989 from colinmorris/ItemExporterDocsExample 
[MRG+1] Revise/modernize item exporter example in docs
 2017-11-03 13:15:31 -02:00
colinmorris 23e571e860 fix issues identified in review 2017-10-31 18:08:47 -04:00
colinmorris 8a7552370d revise/modernize item exporter example in docs 2017-10-31 17:14:53 -04:00
Aditya df7e0a4315 Use https link in default user agent 2017-10-28 23:37:44 +05:30
Aditya 9d9d83a8c3 Use https links wherever possible 2017-10-28 16:24:40 +05:30
Aditya 9dd680d5c9 Use https for external links wherever possible in docs 2017-10-26 23:33:45 +05:30
Mikhail Korobov 98fb03e8bd Merge pull request #2958 from codeaditya/update-links 
Link "Debugging in Python" article to its new location
 2017-10-26 17:01:47 +05:00
Mikhail Korobov 496fc60b25 Merge pull request #2963 from djunzu/mention_request_meta_depth_on_depthmiddleware_doc 
Add note about request.meta['depth'] in DepthMiddleware
 2017-10-26 16:15:36 +05:00
Mikhail Korobov 73c40eb324 Merge pull request #2964 from weldon0405/patch-1 
Update tutorial.rst startproject files
 2017-10-26 16:05:31 +05:00
Weldon Malbrough 95815d27e8 updated file structure to include middlewares.py 2017-10-25 23:16:30 -04:00
Weldon Malbrough 169dc2860e Update tutorial.rst startproject files 
Added middlewares.py to accurately reflect the file structure created by "scrapy startproject tutorial"
 2017-10-16 22:46:32 -04:00
djunzu 8288f78a39 Add note about request.meta['depth'] in DepthMiddleware 2017-10-16 21:34:37 -02:00
Aditya 9cdf34b7c7 Link "Debugging in Python" article to its new location 
Reference: https://web.archive.org/web/20170203104051/http://www.ferg.org/papers/debugging_in_python.html
 2017-10-10 22:49:22 +05:30
Paul Tremberth 0d8a33fddc Update docs 2017-10-05 10:16:42 +02:00
Daniel Graña 5fac2d7b90 Merge pull request #2923 from rhoboro/fixes-685 
[MRG+2] Fixes #685 FilesPipeline support for Google Cloud Storage.
 2017-10-02 13:52:20 -03:00
Mikhail Korobov 92fa74fb43 Merge pull request #2922 from stav/doc.response-body 
[Doc] Update Response.body type
 2017-10-02 10:26:54 +02:00
Mikhail Korobov 3431870b53 Merge pull request #2924 from superyyrrzz/patch-1 
minor fix typo
 2017-10-02 10:25:40 +02:00
rhoboro d4555b2bcc update docs for supporting google cloud storage 2017-09-29 13:04:17 +09:00
Renze Yu 088b80d41a minor fix typo 2017-09-13 23:29:22 +08:00
Steven Almeroth 3637b75a67 [Doc] Update Response.body type 2017-09-12 17:41:47 -04:00
cclauss b8fabeed86 ur'string' not needed in Py 2, syntax error in Py3 
This instance was missed in #2909 --> ur'Scrapy developers' --> u'Scrapy developers'
 2017-09-04 14:29:23 +02:00
cclauss b702236082 ur'string' not needed in Py 2, syntax error in Py3 
Convert `u'(.*)\:\d+\:\s\[(.*)\]\s(?:(.*)\sto\s(.*)|(.*))'`--> `u'(.*)\:\d+\:\s\[(.*)\]\s(?:(.*)\sto\s(.*)|(.*))'`to be compatible with both Python 2 and Python 3.  See #2891
 2017-09-01 11:56:09 +02:00
cclauss 9f16f040b6 ur'string' not needed in Py 2, syntax error in Py3 
Convert `ur'Scrapy Documentation'`--> `u'Scrapy Documentation'`to be compatible with both Python 2 and Python 3.  See #2891
 2017-09-01 11:53:59 +02:00
Pablo Hoffman a429d78019 update scrapinghub.com urls to use https 2017-08-24 16:03:36 -03:00
david watson 71d5b7d75a fix typo (#2867) 2017-08-01 14:49:22 -03:00
simik-ru a65fec050a Small fix in description of startproject arguments 2017-07-31 13:03:49 +02:00
Daniel Graña b6d036ede9 Merge pull request #2837 from dguo/patch-2 
Fix a typo in the Items documentation
 2017-07-24 15:55:30 -03:00
Daniel Graña 6de106e7a8 Merge pull request #2763 from scrapy/dataloss-typo 
DOC fixed rst syntax in DOWNLOAD_FAIL_ON_DATALOSS docs
 2017-07-24 15:48:03 -03:00
Daniel Graña d1e948cef8 Merge pull request #2791 from starrify/doc-DontCloseSpider 
[MRG+1] Added doc for `scrapy.exceptions.DontCloseSpider`. Also fixes inaccurate doc for `scrapy.signals.spider_idle`.
 2017-07-24 15:29:38 -03:00
Daniel Graña 8e0b640d2b Merge pull request #2826 from dguo/patch-1 
[MRG+1] Tweak the CSVFeedSpider documentation
 2017-07-24 15:28:15 -03:00
Danny Guo 26c488970c Fix a typo in the Items documentation 2017-07-18 19:56:51 -05:00
Claus Conrad 18b96dd82a Spelling mistake 2017-07-15 11:31:09 +02:00
Danny Guo dedc4a8b8f Tweak the CSVFeedSpider documentation 2017-07-13 22:58:10 -05:00
Pengyu CHEN f712513ed7 Added doc for `scrapy.exceptions.DontCloseSpider`. Also fixes inaccurate doc for `scrapy.signals.spider_idle`. 2017-06-15 11:09:17 +08:00
Mikhail Korobov 5aebdac45d Merge pull request #2781 from crasker/patch-1 
use suggest method instead of DEPRECATED one
 2017-06-14 03:29:45 +05:00
Casker ae679f6499 Create item-pipeline.rst 2017-06-09 16:12:20 +08:00
Paul Tremberth 39ad0d0bdd Fix setting name reference 2017-06-06 10:48:30 +02:00
Paul Tremberth e7061f7a41 Reformat a bit 2017-06-06 10:47:43 +02:00
Chuan Jin 3f8542eb56 Update extensions.rst 
#2759
 2017-06-05 23:31:37 +02:00
Paul Tremberth fad6b70d92 Use https:// for readthedocs links 2017-06-01 16:41:52 +02:00
Mikhail Korobov 083880888b DOC fixed rst syntax in DOWNLOAD_FAIL_ON_DATALOSS docs 2017-05-30 00:44:11 +05:00
Valdir Stumm Junior 80b160d0d7 include references to scrapy subreddit in the docs 2017-05-29 14:56:49 -03:00
Kurt Peek af2963d0eb Update autothrottle.rst 
Added missing bullet point for the AUTOTHROTTLE_TARGET_CONCURRENCY setting.
 2017-05-24 19:59:36 +05:00
Paul Tremberth fc2846d637 Set release date for v1.4.0 2017-05-18 22:59:46 +02:00
Mikhail Korobov 76e5b0f65c DOC 1.4 deprecations and backwards incompatible changes, add recent commits to news. 2017-05-18 22:37:15 +02:00
Mikhail Korobov edcde7a2cf DOC tweak release notes: promote response.follow, mention logging/stats changes 2017-05-18 22:37:15 +02:00
Paul Tremberth a3d3cd4cb7 Update with latest merges 2017-05-18 22:37:15 +02:00
Paul Tremberth 432668acf7 Mention implementation of #667 2017-05-18 22:37:15 +02:00
Paul Tremberth 896c30a8eb Reference items pretty-printing issue number 2017-05-18 22:37:15 +02:00
Paul Tremberth 55d1082360 Reference recent fixes and commits 2017-05-18 22:37:15 +02:00
Paul Tremberth 7d72394794 Reword mention of new response.follow() shortcut 2017-05-18 22:37:15 +02:00
Paul Tremberth c6464cc4f5 Add verbose introduction to new features 2017-05-18 22:37:14 +02:00
Paul Tremberth e139d990fc Fix sphinx-build warning on deprecated latex_paper_size 2017-05-18 22:37:14 +02:00
Paul Tremberth cba55cd190 Rephrase other sections 2017-05-18 22:37:14 +02:00
Paul Tremberth 8729a91f7a Rephrase "New features" section 2017-05-18 22:37:14 +02:00
Paul Tremberth 851adcedf2 List merged pull requests since 1.3.3 2017-05-18 22:37:14 +02:00
Paul Tremberth 8aa2e4f997 Merge pull request #1829 from nyov/nyov/editor 
[MRG+1] Remove dependency on os.environ from default settings
 2017-05-18 21:32:51 +02:00
Daniel Graña b9d3b447a5 Merge pull request #2670 from qhuang872/master 
[MRG+1] Update spiders.rst
 2017-05-18 16:31:01 -03:00
Mikhail Korobov 532400f993 Merge pull request #2643 from harshasrinivas/set-retry-times-per-request 
[MRG+1] Add feature to set RETRY_TIMES per request (#2642)
 2017-05-17 15:46:33 +05:00
Mikhail Korobov 1acacab93b Merge pull request #2741 from eliat123/2576_cleanup_MEMUSAGE_REPORT 
[FIX #2576] cleanup: removed unused MEMUSAGE_REPORT
 2017-05-17 13:48:00 +05:00
Daniel Graña 73668ce407 Merge pull request #2721 from HarrisonGregg/feature-drop-from-response-field 
[MRG+1] Allow dropping field in from_response formdata
 2017-05-16 09:47:45 -03:00
Eli Atzaba b74b98fa3e cleanup: removed unused MEMUSAGE_REPORT 
Signed-off-by: Eli Atzaba <eliat123@gmail.com>
 2017-05-16 13:59:58 +03:00
Harrison Gregg 45a323024c Add documentation for dropping fields in from_response request body 2017-05-15 12:25:32 -04:00
Paul Tremberth dfe6d3d59a Merge pull request #2456 from elacuesta/feed_export_beautify 
[MRG+1] Feed exports: beautify JSON and XML
 2017-05-12 18:12:48 +02:00
Paul Tremberth 3a0a86ed31 Clarify FEED_EXPORT_INDENT section 2017-05-12 17:26:17 +02:00
Eugenio Lacuesta 25535dba9c Feed exports: edit note, fix typos 2017-05-10 16:45:15 -03:00
Kurt Peek 548a432951 Minor grammatical changes 2017-05-10 23:48:01 +05:00
yandongxu 4966dd7a7f Fix doc: open file with "wb" mode will get an error in python 3 2017-05-08 18:50:30 +08:00
Liu Siyuan 6c1cacb5d5 [MRG+1] doc: fix documentation error in link-extractor.rst (#2676) 
* fix doc error in link-extractor.rst

* remove the import clause

* update based on suggestion
 2017-05-06 02:47:06 +05:00
Martín Gaitán 4bc0c6b0f4 Update practices.rst 
fix a typo
 2017-04-26 14:12:57 +05:00
Paul Tremberth aa59cf4241 Merge pull request #2668 from harshasrinivas/docs-sphinx-rtd-theme 
[MRG+1] Add sphinx_rtd_theme to docs setup readme
 2017-04-24 12:35:40 +02:00
Tiago Cardoso 00ee9eaeaf Mention how to disable request filtering in documentation of DUPEFILTER_CLASS setting 2017-04-22 14:36:44 +01:00
Mikhail Korobov 9731077a54 Merge pull request #2683 from harshasrinivas/docs-SelectorList 
[MRG+1] Remove __nonzero__ from SelectorList docs
 2017-04-12 21:25:00 +05:00
Paul Tremberth 422b38f65c DOC Rearrange selector sections 2017-04-11 16:55:43 +02:00
Paul Tremberth 163618c9b7 FAQ Rewrite note on Python 3 support on Windows 2017-03-29 12:02:44 +02:00
harshasrinivas 2ff6b05723 Remove __nonzero__ from SelectorList docs 2017-03-24 20:43:28 +05:30
harshasrinivas 0298bcbe79 Update Makefile to open webbrowser in MacOS (#2661) 2017-03-24 18:13:08 +05:30
Qiwei Huang 8ecc307e8f Update spiders.rst 
Added note to allowed_domain attribute with an example explaining what goes in the list
 2017-03-20 19:37:07 -07:00
Qiwei Huang c5f74f7d1a Update spiders.rst 
Added a note to allowed_domains attribute, reminding users not to add urls into the list.
 2017-03-20 18:52:33 -07:00
harshasrinivas 83aa0c5e1a Clarify docs readme 2017-03-20 22:36:09 +05:30
harshasrinivas 4ec07ae764 Create docs/requirements.txt 2017-03-20 22:21:08 +05:30
harshasrinivas a57e49d55b Add sphinx_rtd_theme to docs setup readme 2017-03-20 19:49:31 +05:30
Oto Brglez 11cf6ad425 Comments for AWS_ENDPOINT_URL setting. 2017-03-19 12:48:06 +01:00
Oto Brglez 605691792f Updating media-pipeline docs for S3-like storage. 2017-03-19 12:35:39 +01:00
harshasrinivas 10741aca72 Update docs - improve clarity 2017-03-19 06:17:28 +05:30
Mikhail Korobov ec55799d5e Merge pull request #2616 from redapple/mediapipeline-redirect-fix-continued 
[MRG] Allow redirections in media files downloads
 2017-03-16 15:31:28 +05:00
harshasrinivas 9d97d788c0 Update docs for meta key 2017-03-15 04:13:47 +05:30
Paul Tremberth a7f5207e9f Update version added for SPIDER_LOADER_WARN_ONLY 2017-03-10 16:38:09 +01:00
Paul Tremberth b2c505d8ec Set release date in changelog for v1.3.3 2017-03-10 16:34:21 +01:00
Paul Tremberth d8865b3304 Update changelog for upcoming 1.3.3 2017-03-10 16:34:21 +01:00
Mikhail Korobov 9c69e90056 Merge pull request #2632 from redapple/spider-loader-warn-or-fail 
[MRG] Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure or warning
 2017-03-09 23:01:27 +05:00
Paul Tremberth 9628a73972 Update settings docs for new SPIDER_LOADER_WARN_ONLY 2017-03-09 17:40:34 +01:00
Paul Tremberth 36160f1b0a Merge pull request #2477 from scrapy/recommend-anaconda-for-win 
[MRG+2] docs: installation instructions, mention conda in the beginning (closes #2475)
 2017-03-09 12:56:50 +01:00