RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,004 Commits 42 Branches 132 Tags 165 MiB
Commit Graph

1414 Commits

Author SHA1 Message Date
Paul Tremberth bb8740a552 Update 1.2 release notes with latest changes 2016-09-19 12:26:36 +02:00
Wayne Lovely cc8497abb1 Fix a dict key in the tutorial 2016-09-17 11:09:28 +00:00
Valdir Stumm Junior 233b98d642 include section describing spider arguments 2016-09-16 18:08:10 -03:00
Elias Dorneles 31545a9f84 tutorial: updating extracting data section to introduce CSS and XPath equally 2016-09-16 17:13:24 -03:00
Elias Dorneles 147e75602d update after review comments (thanks @stummjr) 2016-09-16 16:47:24 -03:00
Elias Dorneles 31260cf02f mentions stackoverflow as help channel (fixes #2255) 2016-09-16 16:05:36 -03:00
Elias Dorneles 21de617c77 mention that spiders need to subclass scrapy.Spider 2016-09-16 15:55:14 -03:00
Elias Dorneles b2a5cddbb0 tutorial: update section about following links, expand examples 
adding an AuthorSpider to demonstrate further a different crawling
arrangement.
 2016-09-16 15:49:49 -03:00
Valdir Stumm Junior 0cd9dfcc85 small fixes on tutorial 2016-09-16 15:21:49 -03:00
Valdir Stumm Junior 0da497cf7a updates on the first section (our first spider) 2016-09-16 11:55:23 -03:00
Elias Dorneles c508f40689 use harcoded URLs, remove item reference on second spider 2016-09-15 18:05:09 -03:00
Elias Dorneles 2427791287 tutorial: remove item class definition and present start_requests first 
This changes the tutorial, removing the step of creating an item class
and also starts by presenting the start_requests method instead of
start_urls.
 2016-09-15 17:46:31 -03:00
Elias Dorneles 75531e409e use better condition in example spider 2016-09-15 16:56:13 -03:00
Elias Dorneles 1d159ae6f9 minor grammar fix 2016-09-15 15:37:03 -03:00
Elias Dorneles 18bd0b0886 docs: update overview spider code to use toscrape.com and minor changes 
So, this will replace the spider example code from the overview that
scrapes questions from StackOverflow by a spider scraping quotes (much
like the one in the tutorial), and upates the text around it to be
consistent.

There are also minor wording changes plus a small Sphinx/reST syntax fix
on the features list at the bottom (it was creating a definition list,
causing one line to be bold).
 2016-09-15 15:16:30 -03:00
Paul Tremberth 2f60f2a5a6 Merge pull request #2236 from stummjr/new-tutorial-toscrape 
[MRG+1] Update broken Scrapy tutorial to use quotes.toscrape.com
 2016-09-15 12:05:03 +02:00
Elias Dorneles a9a96bed8f updated tutorial as per review comments 2016-09-14 18:09:39 -03:00
Valdir Stumm Junior bc67cd9edd fix indentation issue 2016-09-14 12:39:29 -03:00
Valdir Stumm Junior 10f8c52f5d changed tutorial examples from dmoz to quotes.toscrape.com 2016-09-13 14:05:26 -03:00
Matti Remes 0ef570f6f0 Update exceptions.rst 
Added the missing dot. (+1 squashed commit)
Squashed commits:
[2198972] Update exceptions.rst

There are namely no constructors in classes in Python but an ``__init__`` method instead.
 2016-09-08 19:38:17 +05:00
Valdir Stumm Junior 9cea6f0730 Add Segment Analytics to Documentation 2016-09-02 14:51:07 -03:00
Paul Tremberth b188f61b95 Update release notes for upcoming 1.2.0 version 2016-09-01 17:38:38 +02:00
Paul Tremberth eedb6ce774 Merge pull request #2190 from stummjr/fix-docs 
[MRG+1] Fix RANDOMIZE_DOWNLOAD_DELAY description in the docs
 2016-08-31 11:51:47 +02:00
Mikhail Korobov 495d322691 DOC move Data Flow below the picture; add links to components 2016-08-26 20:16:22 +05:00
Thom Dixon f68dc3026d Fix indentation 2016-08-24 09:11:27 -07:00
Thom Dixon 633abfbea1 Correct documentation about Response parameters 
This fixes issue #2196
 2016-08-24 08:47:52 -07:00
Valdir Stumm Junior d61650d843 fix RANDOMIZE_DOWNLOAD_DELAY description in the docs 2016-08-19 18:24:32 -03:00
Paul Tremberth f18c3e5ce5 Add release notes for 1.1.2 version 2016-08-19 17:01:57 +02:00
Ashish Kulkarni bb3b806467 Use w3lib.url.canonicalize_url() from w3lib 1.15.0 
Also remove code/imports which are now unused due to this change.

fixes #2157
 2016-08-16 17:42:16 +05:30
Lorena 7d432872bf text updates to match graphic 2016-08-04 11:01:14 -07:00
Lorena 04f93e096c updated graph for organization/clarity 2016-08-04 10:04:47 -07:00
Paul Tremberth 5b1d98b8c8 Update 1.1.1 release date 2016-08-01 20:21:12 +02:00
Paul Tremberth 928e93f8f3 Update notes with latest 1.1 commits 2016-08-01 20:21:12 +02:00
Paul Tremberth e1d118d5ca Update release notes for upcoming 1.1.1 release 2016-08-01 20:21:12 +02:00
Elias Dorneles 34e7dadf38 Merge pull request #1610 from darshanime/scheduler_debug 
[MGR+1] Change, document `LOG_UNSERIALIZABLE_REQUESTS`
 2016-07-29 10:12:52 -03:00
darshanime d8e62e660d update log demo print 2016-07-26 20:46:12 +05:30
Paul Tremberth fe68a45c82 Merge pull request #2138 from jesuslosada/master 
[MRG+1] Apply the FILES_STORE_S3_ACL setting in ImagesPipeline
 2016-07-26 12:50:25 +02:00
Elias Dorneles b6375d3701 Merge pull request #1566 from darshanime/signal-example 
[MRG+1] Include example for signal docs
 2016-07-25 10:49:29 -03:00
Darshan Chaudhary a2e6452554 Include signal example 2016-07-25 18:41:25 +05:30
darshanime 0c77b6d033 update docs for settings 2016-07-25 18:05:51 +05:30
Darshan Chaudhary 472a8a47d0 Change name, log once 2016-07-25 18:03:43 +05:30
Jesús Losada 8221556089 Add documentation on Amazon S3 ACLs 2016-07-22 23:22:18 +02:00
Jesús Losada c6a2ca4e93 Document S3 capabilities in FilesPipeline and ImagesPipeline 2016-07-22 19:32:29 +02:00
Paul Tremberth ec1c61504a Merge pull request #2005 from feliperuhland/master 
[MRG+1] Included new optional parameter in startproject command line
 2016-07-19 12:31:06 +02:00
Kenneth Schnall 5fabed514e Update logging.rst 2016-07-15 23:14:11 -04:00
Mikhail Korobov 79639d0fec Merge pull request #1989 from pawelmhm/fix-images-pipeline-uppercase-other 
[MRG+1] [image_pipeline] bring back uppercase class attributes
 2016-07-13 14:44:00 +00:00
Kenneth Schnall 2489f84d39 Update practices.rst 2016-07-12 15:34:50 -04:00
Paul Tremberth c3109daa72 Merge pull request #2034 from dracony/master 
[MRG+1] Added option to turn off ensure_ascii for JSON exporters
 2016-07-12 17:01:09 +02:00
Dracony 33a39b368f added FEED_EXPORT_ENCODING setting to allow encoding specification 2016-07-12 16:20:17 +02:00
Pawel Miech c22cc1096b [image_pipeline] style edits 
* 80 characters line limit
* shortening some code
* removed dead code
* add doctest for _key_for_pipe function
 2016-07-12 13:58:36 +02:00
Mikhail Korobov 5f261cf295 Merge pull request #2026 from Digenis/contributing-full_subjects 
[MRG+1] Encourage complete titles in pull requests
 2016-07-08 10:45:22 +06:00
Valdir Stumm Junior 081595a2e4 document new genspider behavior 2016-07-06 15:10:48 -03:00
Raul Gallegos 49ac7de231 prioritize default headers over user agent 2016-07-05 15:38:17 -05:00
Νικόλαος-Διγενής Καραγιάννης 2a92ffb409 Encourage descriptive PR titles 2016-07-05 14:21:28 +03:00
Pawel Miech b67440dec0 docs on Scrapy on Windows + Python 3 2016-07-04 16:37:51 +02:00
Pawel Miech 07d1605586 [docs] warnings about windows + python 3 in faq and install 2016-06-17 13:28:51 +02:00
Pawel Miech 73cc066caa [docs] add note about windows + python3 2016-06-17 09:21:58 +02:00
Pawel Miech 10b79c9b3e [files-pipeline] update docs with note about settings 
for subclasses.
 2016-06-15 15:49:11 +02:00
Mikhail Korobov 80c296e091 Merge pull request #2048 from redapple/bs4-faq 
[MRG] Add FAQ entry on using BeautifulSoup in spider callbacks
 2016-06-14 23:45:04 +06:00
Paul Tremberth 1ff9a4828b Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-14 19:33:56 +02:00
Paul Tremberth 6cbd92fac1 Add note on how to choose parser with BeautifulSoup 2016-06-14 15:33:34 +02:00
Matt O'Connell 0bb1288969 Typo fixes 2016-06-13 16:21:02 -04:00
Paul Tremberth 7978237ed5 Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-10 17:42:18 +02:00
Pawel Miech a62d4b081c [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-06-10 12:48:02 +02:00
Konstantin Lopuhin da8d0eadd6 Use "url" variable in the example 
Instead of hardcoded http://www.example.com: without it url variable is unused and only one request will make it past dupefilter.
 2016-05-27 10:47:57 +03:00
Felipe Ruhland 089483aece
Updated docs for new option in command line 2016-05-23 23:16:15 -03:00
Paul Tremberth b3367c7acd DOC Add info and example on errbacks 2016-05-18 18:00:09 +02:00
Paul Tremberth c2c8036a97 DOC Update copyright notice 2016-05-18 16:52:09 +02:00
Pawel Miech 6c67db3917 [image_pipeline] tests for class attrs backward compatibility 
and docs about image pipeline settings.
 2016-05-18 12:04:52 +02:00
Paul Tremberth 149c4cd41e [DOC][Overview] Use idiomatic .extract_first() 2016-05-17 17:53:40 +02:00
Paul Tremberth 9b3c72cb37 DOC Place FEED_TEMPDIR setting at lexicographical position 2016-05-17 16:25:57 +02:00
Paul Tremberth 0700e437f2 Update release notes and date for 1.1.0 2016-05-11 20:39:35 +02:00
Paul Tremberth d62654dfda Merge 1.0.6 release notes 2016-05-11 14:39:55 +02:00
Panayiotis Lipiridis 3f8be374c3 Github >> GitHub 2016-05-09 16:15:18 +02:00
Paul Tremberth cb38f3896b Add notes on botocore support and JsonItemExporter opening/closing brackets 2016-04-28 22:34:17 +02:00
Paul Tremberth dc9de7524f Update release notes with recent features and bug fixes. 2016-04-28 22:34:07 +02:00
Paul Tremberth 2849ebf4c6 Reference StackOverflow's "minimal, complete, and verifiable example" guide 2016-04-12 14:07:33 +02:00
Konstantin Lopuhin 1ec49c2ada Fix FAQ entry about python versions support 2016-04-12 11:48:57 +03:00
nblock a3557dd34d Fix spelling mistake 2016-04-11 14:06:57 +02:00
Paul Tremberth 0ede017d2a Merge pull request #1891 from djunzu/update_files_images_pipelines 
[MRG+1] Change Files/ImagesPipelines class attributes to instance attributes
 2016-04-08 12:55:09 +02:00
djunzu 6988e9cd4b Update docs. 
modified:   docs/topics/media-pipeline.rst
 2016-04-01 21:51:15 -03:00
Paul Tremberth bf7f675493 Merge pull request #1847 from aron-bordin/add_blocking_storage_path_setting 
[MRG+2] added BLOCKING_FEED_STORAGE_PATH to settings
 2016-04-01 15:47:06 +02:00
Aron Bordin 9250a5bffa added FEED_TEMPDIR to settings 2016-04-01 00:05:21 -03:00
Paul Tremberth 9d8c368ce8 Merge pull request #1879 from scrapy/scrapy-arch-docs 
DOC improved Architecture overview
 2016-03-31 12:09:24 +02:00
Paul Tremberth 1075587dbd Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 14:40:52 +02:00
Lele 7082454f2a Changed sel. to response. for clarity 
Changed sel. to response. to comply with the rest of the examples in the same section, to avoid confusion.
 2016-03-28 05:27:15 +05:00
Mikhail Korobov 4f335b5a01 DOC clarify Architecture docs 2016-03-25 17:03:41 +05:00
Mikhail Korobov 3ca977a8cb DOC improved Architecture overview 
* spiders don't have to work on specific domains;
* explain what to use Downloader middleware for
  and what to use Spider middleware for;
* Engine no longer locates spiders based on domains;
* "Spider middleware output direction" step was missing.

See also: GH-1569.
 2016-03-25 07:11:33 +05:00
Aron Bordin 2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth 90c64a6d13 Update release notes about change of default S3 ACL policy to "private" 2016-02-29 12:27:25 +01:00
Paul Tremberth 709b4fa884 Update release notes about HTTPS downloader 2016-02-26 18:35:29 +01:00
Paul Tremberth 174f526709 Document DOWNLOADER_* settings for HTTP/1.0 and TLS 2016-02-26 18:23:57 +01:00
Paul Tremberth 84dea19455 Update release notes 2016-02-25 00:59:15 +01:00
Konstantin Lopuhin 6137dd96d9 Fix documentation for S3_STORE_ACL (now settings.FILES_STORE_S3_ACL) settings: it has nothing to do with feed exporters. 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin 164f300762 See #1778 - change default S3 ACL to "private" and allow customization via settings 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin e7c4806c5e Update feedstorage docs: add botocore, mention that boto is supported only on Python 2 2016-02-15 19:59:48 +03:00
Elias Dorneles 164493df2e add deprecation for pydispatch (thanks for the help @redapple) 2016-02-11 16:15:28 -02:00
Nicolas Pennequin 6efc7a9de0 Update the email doc for the charset argument to send(). 2016-02-04 19:33:44 +01:00
Elias Dorneles ca08314654 makes more explicit source of problems in Python 3 limitations 2016-02-03 21:38:25 -02:00
Elias Dorneles 2167573b32 fix comment about disabling robots.txt 2016-02-03 21:31:27 -02:00
Elias Dorneles 3180abc800 applying more review comments 2016-02-03 21:28:05 -02:00
Elias Dorneles d40add7b27 add note about robots.txt waiting and make it explicit builtin extensions only are ported 2016-02-03 21:18:25 -02:00
Elias Dorneles 9cfefd5277 favor beta over basic to describe PY3 support 2016-02-03 20:59:56 -02:00
Elias Dorneles 2f697ce59f added note about how to revert to old behavior for robotstxt 2016-02-03 20:51:26 -02:00
Elias Dorneles 89a088b3c3 applying review comments 2016-02-03 20:45:55 -02:00
Paul Tremberth 972f192568 Blank line before CoC 2016-02-03 22:58:49 +01:00
Paul Tremberth 728e505e53 Merge remote-tracking branch 'origin/master' into release-notes-1.1 2016-02-03 22:47:15 +01:00
Valdir Stumm Jr dd7b0eb359 Updates to release notes 2016-02-03 19:23:02 -02:00
Paul Tremberth 51dc741fa2 Add 1.0.5 release notes 2016-02-03 21:07:21 +01:00
Valdir Stumm Jr 0f816524e8 edit API changes, Deprecations, and Relocations sections. 2016-02-03 16:21:32 -02:00
Paul Tremberth fd3193da86 Reword some of "New Features and Enhancements" 2016-02-03 18:19:38 +01:00
Paul Tremberth 3b2a6abc0b Format "Code refactoring" subsection 2016-02-03 17:37:58 +01:00
Paul Tremberth ed02455010 Format "Logging" subsection 2016-02-03 17:27:43 +01:00
Valdir Stumm Jr 1fd95c9c9f fixed references to issues and fit text to 80 cols in the first two sections 2016-02-03 12:45:48 -02:00
Valdir Stumm Jr a21c90bee7 edits on Py3 Support and New features sections 2016-02-03 11:54:46 -02:00
Paul Tremberth 44d8df2060 Add versionadded note for MEMUSAGE_CHECK_INTERVAL_SECONDS 2016-02-03 12:35:26 +01:00
Paul Tremberth 2b033eebce Fix recently added HTTPCACHE_ settings versionadded notes 2016-02-03 12:34:44 +01:00
Paul Tremberth 142aa21737 Add AUTOTHROTTLE_TARGET_CONCURRENCY versionadded note 2016-02-03 12:33:10 +01:00
Paul Tremberth db0697bc06 Add 1.1 release notes (draft) 2016-02-03 12:32:40 +01:00
Paul Tremberth c6f374f2eb Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-28 14:02:48 +01:00
Paul Tremberth 27fb200e45 Merge pull request #1727 from redapple/priority-adjust 
[MRG+1] Clarify priority adjust settings docs
 2016-01-27 18:52:39 +01:00
Mikhail Korobov 7ca9ae1976 DOC typo fix 2016-01-27 17:54:28 +05:00
Paul Tremberth d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
Paul Tremberth 6ed08d2332 Add note for DEPTH_PRIORITY 2016-01-27 11:53:29 +01:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth 1c83108893 Clarify priority adjust settings docs 
Fixes #1593
 2016-01-26 19:24:11 +01:00
Paul Tremberth bb1f4013a3 Rewrite warning about shell with local files as note 2016-01-26 17:23:28 +01:00
Mikhail Korobov f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
Νικόλαος-Διγενής Καραγιάννης 1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Paul Tremberth 713e1eee9b Update docs about local files support for "scrapy shell" 2016-01-26 10:44:38 +01:00
Raul Gallegos a06a5f00f4 adding configurable encoding for httpproxy authentication 2016-01-21 09:35:04 -05:00
Elias Dorneles d4c4ca8062 fix version number to appear new feature 2016-01-21 09:42:15 -02:00
Capi Etheriel 659715ecd9 implements FormRequest.from_response CSS support 2016-01-21 01:05:20 -02:00
Daniel Graña b1e44436bc Merge pull request #1642 from orangain/doc-metarefresh 
[MRG+1] DOC: Update MetaRefreshMiddlware's setting variables
 2016-01-19 13:33:50 -03:00
carlosp420 e15f361b05 fixed typo You -> you 2016-01-19 11:12:43 -05:00
Ralph Gutkowski 79147a61a7 Update stats.rst 2016-01-15 19:25:56 +01:00
Ralph Gutkowski bb38400db5 Update Stats Collection documentation 
`pages_crawled` value doesn't exist. Replace it with `downloader/response_count` in order to avoid confusion.
 2016-01-15 19:00:58 +01:00
Valdir Stumm Jr 2abc9bc901 Update deprecated examples 
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
 2016-01-06 10:29:45 -02:00
Daniel Graña 1b435b2887 Add 1.0.4 release notes 2015-12-30 15:43:54 -03:00
orangain bcce8d3d80 DOC: Update MetaRefreshMiddlware's setting variables 
* `REDIRECT_MAX_METAREFRESH_DELAY` has been deprecated and was
  renamed to `METAREFRESH_MAXDELAY`.
* Merge duplicate documents about `METAREFRESH_MAXDELAY` appeared both
  in the settings page and the downloader-middlewares page.
 2015-12-17 15:25:02 +09:00
orangain 4be4ef038e DOC: Add captions to toctrees which appear in sidebar 2015-12-12 16:21:00 +09:00
Rolando Espinoza f8ae99d18f DOC Removed pywin32 from install instructions as it's already declared as dependency. 2015-12-05 09:48:17 -04:00
palego 2c251000d3 custom project templates 
allow override of TEMPLATES_DIR for startproject
copy full TEMPLATES_DIR/project tree
doc update
 2015-12-01 03:07:27 +01:00
Pablo Hoffman eb4daa34a2 Merge pull request #1544 from rolando/conda-docs 
Added installation notes about using Conda for Windows.
 2015-11-26 14:11:28 -03:00
Rolando Espinoza cc8ddb68b0 Added installation notes about using Conda for Windows and other OSes. 2015-11-26 13:08:56 -04:00
Julia Medina 48539af163 Merge pull request #1601 from mvj3/minor-documentation-fixes 
Fixed minor grammar issues.
 2015-11-16 14:58:07 -03:00
David Chen 0025d5a943 Fixed minor grammar issues. 2015-11-16 07:30:17 +08:00
Jakob de Maeyer e66f649894 Bring back _BASE settings 2015-11-11 17:39:56 +01:00
Julia Medina 72eeead6db Merge pull request #1524 from Digenis/relocate_telnet_console 
[MRG+1] Relocate telnetconsole to extensions/
 2015-10-30 12:10:38 -03:00