RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,222 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

6222 Commits

Author SHA1 Message Date
Paul Tremberth f3d5995329 Bump version: 1.2.1 → 1.2.2 2016-12-06 15:21:00 +01:00
Paul Tremberth ea968c6c3d Merge pull request #2423 from redapple/release-notes-1.2.2 
Update changelog for upcoming 1.2.2 release
 2016-12-06 15:18:31 +01:00
Paul Tremberth 09e310d0b7 Set release date for 1.2.2 2016-12-06 15:03:38 +01:00
Paul Tremberth aa2e1b030d Add reference to fixed scheduler settings documentation 2016-12-06 14:59:11 +01:00
Paul Tremberth 89d5f5acd3 Update changelog for upcoming 1.2.2 release 2016-12-06 14:59:11 +01:00
Mikhail Korobov 97df139119 Merge pull request #2417 from pawelmhm/scheduler-docs 
[MRG+1] Add docs for some scheduler settings
 2016-12-06 18:42:07 +05:00
Moisés Guimarães c08d278c0c removes note from docs. 2016-12-05 16:47:24 -03:00
Moisés Guimarães a4178f99da fixes params types in docs. 2016-12-05 15:24:26 -03:00
Moisés Guimarães c58ea021b8 fixes docs 2016-12-04 11:56:14 -03:00
nyov 67bc2e0b18 Wipe scrapy.xlib.tx 2016-12-02 21:22:35 +00:00
nyov 67cf64edbe Deprecate scrapy.xlib.tx 2016-12-02 21:22:35 +00:00
nyov 534772f6ea Import xlib.tx code from twisted proper 2016-12-02 21:21:51 +00:00
nyov 985755d1fe Remove obsolete xlib code for Twisted 13.1.0 2016-12-01 23:44:15 +00:00
nyov c8cf1a303d Bump Twisted dependency to 13.1.0 (released June 2013) 2016-12-01 22:02:11 +00:00
nyov e1ea0c433a Strip xlib.tx code of Twisted 10 2016-12-01 22:02:10 +00:00
Elias Dorneles c4e67c0696 Merge pull request #2421 from rolando-contrib/tests-bug 
[MRG+1] TST: Fix duplicated test name.
 2016-12-01 18:43:48 -02:00
Elias Dorneles f3a4420750 Merge pull request #2388 from redapple/robotparser-native-str 
[MRG+1] Parse robots.txt content as native str
 2016-12-01 18:43:23 -02:00
Mikhail Korobov 826b7e92b6 Merge pull request #2386 from rolando-contrib/manifest-tweak 
[MRG+1] Ignore explicitly compiled python files.
 2016-12-02 00:20:27 +05:00
Rolando Espinoza 923b974f0a TST Include nested a nested spider in spider loader test. 2016-12-01 13:26:19 -03:00
Rolando Espinoza 6431e7a138 DOC State explicitly that spiders are loaded recursively. 2016-12-01 13:26:19 -03:00
Rolando Espinoza d9f43e21ba TST: Fix duplicated test name. 2016-12-01 11:56:33 -03:00
Elias Dorneles 5f0af90674 Merge pull request #2395 from elacuesta/robots_txt_relative_sitemap_url 
[MRG+1] Handle relative sitemap URLs in robots.txt
 2016-12-01 11:24:54 -02:00
Eugenio Lacuesta 5ff64ad015 handle relative sitemap urls in robots.txt 2016-12-01 09:53:40 -03:00
Elias Dorneles 2086ff4065 Merge pull request #2418 from ahlinc/fix_retriggered_issue_396 
[MRG+1] Fix #396 re-triggered issue
 2016-11-30 10:49:41 -02:00
Mikhail Korobov bba4645f21 Merge pull request #2404 from redapple/scrapy-version 
[MRG+1] Print more dependencies versions in "scrapy version" verbose output
 2016-11-30 17:01:37 +05:00
Andrew Hlynskyi 27606aad11 Fix #396 re-triggered issue 
The InteractiveShellEmbed class is a singleton
and we need to drop the instance with its clear_instance() method
to rebuild the instance from scratch with fresh environment
for each subsequent Scrapy's shell drop in.
 2016-11-30 09:52:44 +02:00
Paul Tremberth 624284e851 Fix indent 2016-11-29 18:18:59 +01:00
Pawel Miech f98ffb53b6 add docs for some scheduler settings 2016-11-29 16:52:54 +01:00
Paul Tremberth 059085b5b4 Remove docs for deprecated ChunkedTransfer middleware 2016-11-24 18:23:34 +01:00
Paul Tremberth 8cffb4bbef Update warning wording 2016-11-24 17:50:21 +01:00
Paul Tremberth e6f174b015 Add deprecation warning for ChunkedTransfer middleware 2016-11-24 17:33:27 +01:00
Paul Tremberth 03cf5f1bd2 Remove ChunkedTransferMiddleware from default settings 2016-11-24 17:18:57 +01:00
Paul Tremberth 9aefc0a886 Add test for fetch command with redirections disabled 2016-11-24 13:41:51 +01:00
Paul Tremberth 35b655d2f8 Handle redirects transparently by default in shell and fetch 
Adds --no-status-aware command line option to have previous behaviour
 2016-11-24 12:23:22 +01:00
Paul Tremberth 01142e2ae5 Print more dependencies versions in "scrapy version" verbose output 2016-11-22 14:48:33 +01:00
Mikhail Korobov a07400ce0a Merge pull request #2396 from redapple/ubuntu-packages-toc 
[MRG] DOC Remove "Ubuntu" section from sidebar/ToC
 2016-11-22 18:21:38 +05:00
Pablo Hoffman d62776a858 mention scrapoxy in best practices doc 2016-11-16 12:19:32 -03:00
Paul Tremberth 8db8545393 Add "orphan" metadata for Ubuntu packages page 
As described in http://www.sphinx-doc.org/en/latest/markup/misc.html#metadata
 2016-11-16 13:56:58 +01:00
Paul Tremberth 11fe3751cf DOC Remove "Ubuntu" section from sidebar/ToC 2016-11-16 11:55:09 +01:00
Paul Tremberth 6cd35c77da Pass user-agent as native str when checking URLs against robots.txt 2016-11-15 17:38:32 +01:00
Elias Dorneles 1cbc57faa5 Merge pull request #2337 from muriloviana/add-middleware-template 
[MRG+1] Add middleware to template project (closes #2335)
 2016-11-15 12:58:40 -02:00
Mikhail Korobov 570e12b5db Merge pull request #2328 from scrapy/download-latency-meta-docs 
[MRG+1] Document `download_latency` meta key
 2016-11-14 21:24:14 +05:00
Valdir Stumm Junior 7025d6656a document download_latency meta key 2016-11-14 13:06:18 -02:00
Paul Tremberth 201a16f6d5 Merge pull request #2394 from nyov/fix-le 
[MRG+1] LinkExtractor PY3 'unicode' type fix
 2016-11-14 12:10:14 +01:00
nyov e8205f6733 LinkExtractor PY3 'unicode' type fix 2016-11-10 17:50:06 +00:00
Paul Tremberth de89b1b562 Merge pull request #2275 from scrapy/response-css-xpath-message 
[MRG+1] Add better messages for when response content isn't text (closes #2264)
 2016-11-10 11:38:22 +01:00
Mikhail Korobov 6d6dc7138b Merge pull request #2387 from rolando-contribute/conda-forge-updates 
[MRG+1] Update conda channel to conda-forge and show conda version badge.
 2016-11-10 00:35:43 +06:00
Paul Tremberth 28155dfccc Parse robots.txt content as native str 
Fixes #2373
 2016-11-09 12:20:06 +01:00
Rolando Espinoza 76459e1969 Update conda channel to conda-forge and show conda version badge. 2016-11-08 21:30:27 -03:00
Rolando Espinoza 3cd56da0cc Ignore explicitly compiled python files. 
This avoids to include compiled files in the templates directory.
 2016-11-08 20:52:32 -03:00