RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,396 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

6396 Commits

Author SHA1 Message Date
Paul Tremberth 624284e851 Fix indent 2016-11-29 18:18:59 +01:00
Pawel Miech f98ffb53b6 add docs for some scheduler settings 2016-11-29 16:52:54 +01:00
Paul Tremberth 059085b5b4 Remove docs for deprecated ChunkedTransfer middleware 2016-11-24 18:23:34 +01:00
Paul Tremberth 8cffb4bbef Update warning wording 2016-11-24 17:50:21 +01:00
Paul Tremberth e6f174b015 Add deprecation warning for ChunkedTransfer middleware 2016-11-24 17:33:27 +01:00
Paul Tremberth 03cf5f1bd2 Remove ChunkedTransferMiddleware from default settings 2016-11-24 17:18:57 +01:00
Paul Tremberth 9aefc0a886 Add test for fetch command with redirections disabled 2016-11-24 13:41:51 +01:00
Paul Tremberth 35b655d2f8 Handle redirects transparently by default in shell and fetch 
Adds --no-status-aware command line option to have previous behaviour
 2016-11-24 12:23:22 +01:00
Paul Tremberth 01142e2ae5 Print more dependencies versions in "scrapy version" verbose output 2016-11-22 14:48:33 +01:00
Mikhail Korobov a07400ce0a Merge pull request #2396 from redapple/ubuntu-packages-toc 
[MRG] DOC Remove "Ubuntu" section from sidebar/ToC
 2016-11-22 18:21:38 +05:00
Pablo Hoffman d62776a858 mention scrapoxy in best practices doc 2016-11-16 12:19:32 -03:00
Paul Tremberth 8db8545393 Add "orphan" metadata for Ubuntu packages page 
As described in http://www.sphinx-doc.org/en/latest/markup/misc.html#metadata
 2016-11-16 13:56:58 +01:00
Paul Tremberth 11fe3751cf DOC Remove "Ubuntu" section from sidebar/ToC 2016-11-16 11:55:09 +01:00
Paul Tremberth 6cd35c77da Pass user-agent as native str when checking URLs against robots.txt 2016-11-15 17:38:32 +01:00
Elias Dorneles 1cbc57faa5 Merge pull request #2337 from muriloviana/add-middleware-template 
[MRG+1] Add middleware to template project (closes #2335)
 2016-11-15 12:58:40 -02:00
Mikhail Korobov 570e12b5db Merge pull request #2328 from scrapy/download-latency-meta-docs 
[MRG+1] Document `download_latency` meta key
 2016-11-14 21:24:14 +05:00
Valdir Stumm Junior 7025d6656a document download_latency meta key 2016-11-14 13:06:18 -02:00
Paul Tremberth 201a16f6d5 Merge pull request #2394 from nyov/fix-le 
[MRG+1] LinkExtractor PY3 'unicode' type fix
 2016-11-14 12:10:14 +01:00
nyov e8205f6733 LinkExtractor PY3 'unicode' type fix 2016-11-10 17:50:06 +00:00
Paul Tremberth de89b1b562 Merge pull request #2275 from scrapy/response-css-xpath-message 
[MRG+1] Add better messages for when response content isn't text (closes #2264)
 2016-11-10 11:38:22 +01:00
Mikhail Korobov 6d6dc7138b Merge pull request #2387 from rolando-contribute/conda-forge-updates 
[MRG+1] Update conda channel to conda-forge and show conda version badge.
 2016-11-10 00:35:43 +06:00
Paul Tremberth 28155dfccc Parse robots.txt content as native str 
Fixes #2373
 2016-11-09 12:20:06 +01:00
Rolando Espinoza 76459e1969 Update conda channel to conda-forge and show conda version badge. 2016-11-08 21:30:27 -03:00
Rolando Espinoza 3cd56da0cc Ignore explicitly compiled python files. 
This avoids to include compiled files in the templates directory.
 2016-11-08 20:52:32 -03:00
Mikhail Korobov ea83e67796 Merge pull request #2382 from redapple/slot-heartbeat-state-test 
Test Slot's heartbeat state before stopping it
 2016-11-08 18:32:07 +06:00
Paul Tremberth af2280e695 Update docstring 2016-11-08 13:30:51 +01:00
Paul Tremberth 27456996a9 Add assertion on crawler not running 2016-11-08 11:46:16 +01:00
Paul Tremberth 61efacdd1f Add testcase for catching exception from open_spider() from pipeline 2016-11-08 11:35:42 +01:00
Paul Tremberth 7727d87f64 Test Slot's heartbeat state before stopping it 
Also add a test on state of looping task in LogStats extension

Fixes #2011 and #2362
 2016-11-07 16:44:57 +01:00
Mikhail Korobov 0f5eb4cf88 Merge pull request #2380 from rahulkant13may/master 
Document update in "Using Firefox for scraping"
 2016-11-07 19:02:53 +06:00
Rahul Kant f56aef99c2 Add closing tag in <tbody> 2016-11-07 17:49:22 +05:30
Gustavo de Andrade f2e49bc23c Update selectors.rst 
Decode instead escape, exceptions &lt; and &amp; (kmike)
Second sentence droped (Digenis)
 2016-11-01 21:32:17 -02:00
Mikhail Korobov 9755ef933a Merge pull request #2369 from jc0n/media-pipeline-docs-typo 
Fix typo in media pipeline docs
 2016-10-29 12:46:48 +06:00
John O'Connor d85da273be Fix typo in media pipeline docs 2016-10-28 19:44:46 -07:00
Elias Dorneles a9c74dbe42 Merge pull request #2339 from visued/master 
[MRG+1] Update documentation to explain the use of double quotes on Windows (fixes #2325)
 2016-10-25 09:53:24 -02:00
Paul Tremberth bd4f156d16 Merge pull request #2354 from stav/doc-spider-arguments 
[MRG+1] doc: wording
 2016-10-24 12:42:59 +02:00
Steven Almeroth 99daea495b Doc: wording 2016-10-21 16:16:42 -07:00
Steven Almeroth a958e54954 Doc: remove trailing spaces 2016-10-21 16:16:37 -07:00
Mikhail Korobov 1be2447af9 Merge pull request #2346 from redapple/master-pr2313 
Fix tutorial AuthorSpider
 2016-10-21 11:08:17 +02:00
Randy Pen c7d245b90b update 
Thx for your advice.
 2016-10-21 10:52:50 +02:00
Randy Pen eacc5937e4 fix example code 
In the AuthorSpider, original css selector failed to get links of author pages
 2016-10-21 10:52:50 +02:00
Paul Tremberth 6df48d57e0 Bump version: 1.2.0 → 1.2.1 2016-10-21 10:37:45 +02:00
Paul Tremberth f0e935735f Merge pull request #2341 from redapple/release-notes-1.2.1 
Update release notes for upcoming 1.2.1
 2016-10-21 10:34:47 +02:00
Paul Tremberth c559a64c20 Set date for 1.2.1 release 2016-10-21 10:17:46 +02:00
Paul Tremberth 22772a61c8 Update release notes for upcoming 1.2.1 2016-10-21 10:17:04 +02:00
gustavodeandrade c7dfb5eb88 Fix issue 1704 2016-10-21 00:08:08 -02:00
Paul Tremberth eb5d396527 Merge pull request #2344 from jaympatel/master 
Typo (through was misspelled)
 2016-10-20 22:24:14 +02:00
Jay Patel f357ccd0d7 Typo (through was misspelled) 2016-10-20 15:52:28 -04:00
Mikhail Korobov 871eec9827 Merge pull request #2327 from bopace/http-header-docs 
[MRG+1] Added documentation about accessing header values
 2016-10-20 09:05:14 +02:00
Mikhail Korobov 71c8278f57 Merge pull request #2329 from josericardo/scrapy-2262 
[MRG+1] Better explain middleware orders and processing directions
 2016-10-20 09:04:19 +02:00