RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,809 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

6809 Commits

Author SHA1 Message Date
Elias Dorneles 063315258e Merge pull request #2202 from scrapy/doc-arch-overview2 
[MRG+1] DOC move Data Flow below the picture; add links to components
 2016-09-19 08:11:18 -03:00
Paul Tremberth bb8740a552 Update 1.2 release notes with latest changes 2016-09-19 12:26:36 +02:00
Mikhail Korobov 490f6e08f3 Merge pull request #2239 from redapple/streamlogger-flush 
[MRG+1] Add flush() method to StreamLogger
 2016-09-19 14:44:45 +06:00
Mikhail Korobov 5657f6b8ef Merge pull request #2258 from redapple/feed-export-started 
[MRG+1] Feed exporter: start exporting only on first item
 2016-09-19 14:40:30 +06:00
Mikhail Korobov 552368727a Merge pull request #2225 from Tethik/parse_command_rules_fix 
[MRG+1] Two small fixes for when using the parse command and the '-r' flag (rules).
 2016-09-19 14:39:09 +06:00
Joakim Uddholm 8c38dde4e8 Moved parse command tests to its own file. Added some checks to check for logged errors. 2016-09-19 05:33:05 +02:00
Joakim Uddholm 88cf86f5f2 Merge pull request #1 from redapple/tethik_parse_command_rules_fix 
Add tests for crawl command non-default cases
 2016-09-19 00:51:36 +02:00
Paul Tremberth 48f6a065b8 Flush StreamLogger handlers 2016-09-17 15:25:45 +02:00
Paul Tremberth 27f88ad9cb Merge pull request #2260 from waynelovely/tutorial-fix-20160917-1 
Fix a dict key in the tutorial
 2016-09-17 14:19:21 +02:00
Wayne Lovely cc8497abb1 Fix a dict key in the tutorial 2016-09-17 11:09:28 +00:00
Mikhail Korobov 992b2517b0 Merge pull request #2248 from redapple/scrapy-shell-import-scrapy 
[MRG+1] Make scrapy available in shell without explicit import statement
 2016-09-17 06:10:06 +06:00
Mikhail Korobov 91fcafde5e Merge pull request #2257 from scrapy/mention-stackoverflow 
Mentions stackoverflow as support channel (fixes #2255)
 2016-09-17 06:06:53 +06:00
Paul Tremberth 03ab077249 Feed exporter: start exporting only on first item 
Fixes GH-872
 2016-09-17 01:36:56 +02:00
Valdir Stumm Junior 233b98d642 include section describing spider arguments 2016-09-16 18:08:10 -03:00
Elias Dorneles 31545a9f84 tutorial: updating extracting data section to introduce CSS and XPath equally 2016-09-16 17:13:24 -03:00
Elias Dorneles 147e75602d update after review comments (thanks @stummjr) 2016-09-16 16:47:24 -03:00
Elias Dorneles 31260cf02f mentions stackoverflow as help channel (fixes #2255) 2016-09-16 16:05:36 -03:00
Elias Dorneles de1a6ac677 Merge pull request #2249 from scrapy/fix-overview-spider 
[MRG+1] docs: update overview spider code to use toscrape.com and minor changes
 2016-09-16 16:00:23 -03:00
Elias Dorneles 21de617c77 mention that spiders need to subclass scrapy.Spider 2016-09-16 15:55:14 -03:00
Elias Dorneles b2a5cddbb0 tutorial: update section about following links, expand examples 
adding an AuthorSpider to demonstrate further a different crawling
arrangement.
 2016-09-16 15:49:49 -03:00
Valdir Stumm Junior 0cd9dfcc85 small fixes on tutorial 2016-09-16 15:21:49 -03:00
Valdir Stumm Junior 0da497cf7a updates on the first section (our first spider) 2016-09-16 11:55:23 -03:00
Elias Dorneles c508f40689 use harcoded URLs, remove item reference on second spider 2016-09-15 18:05:09 -03:00
Elias Dorneles 2427791287 tutorial: remove item class definition and present start_requests first 
This changes the tutorial, removing the step of creating an item class
and also starts by presenting the start_requests method instead of
start_urls.
 2016-09-15 17:46:31 -03:00
Elias Dorneles 75531e409e use better condition in example spider 2016-09-15 16:56:13 -03:00
Paul Tremberth effaab867e Update shell help with availability of scrapy module 2016-09-15 21:37:15 +02:00
Elias Dorneles 1d159ae6f9 minor grammar fix 2016-09-15 15:37:03 -03:00
Elias Dorneles 18bd0b0886 docs: update overview spider code to use toscrape.com and minor changes 
So, this will replace the spider example code from the overview that
scrapes questions from StackOverflow by a spider scraping quotes (much
like the one in the tutorial), and upates the text around it to be
consistent.

There are also minor wording changes plus a small Sphinx/reST syntax fix
on the features list at the bottom (it was creating a definition list,
causing one line to be bold).
 2016-09-15 15:16:30 -03:00
Paul Tremberth 105163fece Make scrapy available in shell without explicit import statement 2016-09-15 19:26:53 +02:00
Paul Tremberth b828facff4 Add shell test for using scrapy.Request() directly without importing scrapy 2016-09-15 19:25:20 +02:00
Paul Tremberth 2f60f2a5a6 Merge pull request #2236 from stummjr/new-tutorial-toscrape 
[MRG+1] Update broken Scrapy tutorial to use quotes.toscrape.com
 2016-09-15 12:05:03 +02:00
pawelmhm 7d88209543 [image & file pipeline] loading setting for user classes 
if user has some custom subclass of Image pipeline and no setting for
this pipeline, he should get default settings defined for Image Pipeline.

Fixes #2198
 2016-09-15 09:39:16 +02:00
Elias Dorneles a9a96bed8f updated tutorial as per review comments 2016-09-14 18:09:39 -03:00
Valdir Stumm Junior bc67cd9edd fix indentation issue 2016-09-14 12:39:29 -03:00
Paul Tremberth 498a3725d1 Add flush() method to StreamLogger 
Fixes GH-2125
 2016-09-14 12:19:50 +02:00
Valdir Stumm Junior 10f8c52f5d changed tutorial examples from dmoz to quotes.toscrape.com 2016-09-13 14:05:26 -03:00
Elias Dorneles 129421c7e3 Merge pull request #1503 from demelziraptor/amazon-json-response 
[MRG+1] interpreting json-amazonui-streaming as TextResponse
 2016-09-12 13:21:16 -03:00
Paul Tremberth fbb5559299 Add tests for crawl command non-default cases 2016-09-12 13:35:14 +02:00
Andrew Hlynskyi 80260824c6 Fix completion in `scrapy shell` for new imports 2016-09-12 01:13:23 +03:00
Joakim Uddholm 743a0aa422 Two fixes for when using the parse command and the '-r' flag (rules). 
1. Use default "parse" as callback when the matching rule has no callback.
2. Log error and return when no rule matches the parsed url.
 2016-09-08 21:52:14 +02:00
Matti Remes 0ef570f6f0 Update exceptions.rst 
Added the missing dot. (+1 squashed commit)
Squashed commits:
[2198972] Update exceptions.rst

There are namely no constructors in classes in Python but an ``__init__`` method instead.
 2016-09-08 19:38:17 +05:00
Paul Tremberth ec4ab126b6 Merge pull request #2220 from scrapy/comment-typo-fix 
typo fix in HttpProxyMiddleware
 2016-09-07 10:15:12 +02:00
Mikhail Korobov 960b1bc8f0 typo fix in HttpProxyMiddleware 2016-09-07 04:54:32 +05:00
Paul Tremberth 9e6a72cc4b Merge pull request #2217 from stummjr/add-analytics-to-docs 
[MRG+1] Add Segment Analytics to Documentation
 2016-09-05 15:59:58 +02:00
Valdir Stumm Junior 9cea6f0730 Add Segment Analytics to Documentation 2016-09-02 14:51:07 -03:00
Paul Tremberth b188f61b95 Update release notes for upcoming 1.2.0 version 2016-09-01 17:38:38 +02:00
Paul Tremberth 58cd7bf895 Remove "precise" test env from Travis-CI config 2016-09-01 11:17:53 +02:00
Paul Tremberth 2b2bfcea88 Add "jessie" build to Travis-CI config 2016-09-01 10:20:49 +02:00
Paul Tremberth 22e870e955 Add Debian Jessie test env 2016-09-01 10:19:49 +02:00
Paul Tremberth eedb6ce774 Merge pull request #2190 from stummjr/fix-docs 
[MRG+1] Fix RANDOMIZE_DOWNLOAD_DELAY description in the docs
 2016-08-31 11:51:47 +02:00