RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,956 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

5956 Commits

Author SHA1 Message Date
Paul Tremberth bfd23f4260 Bump version: 1.2.2 → 1.2.3 2017-03-03 19:40:54 +01:00
Paul Tremberth 15abc2a7fb Merge pull request #2620 from scrapy/release-notes-1.2.3 
[1.2.x] Update release notes for 1.2.3
 2017-03-03 19:40:20 +01:00
Paul Tremberth a8d8271805 Update release notes for 1.2.3 2017-03-03 19:29:19 +01:00
Mikhail Korobov 9b8dbb0cef Merge pull request #2615 from redapple/1.2-stricter-requirements 
[1.2.x] Stricter requirements on Twisted (<17) and lxml (>=2.3)
 2017-03-03 02:46:09 +05:00
Paul Tremberth f45618d2b0 Stricter requirements on Twisted (<17) and lxml (>=2.3) 2017-03-02 17:33:09 +01:00
Mikhail Korobov 10954fc74b Merge pull request #2529 from scrapy/backport-1.2-pr2496-dns-float 
[backport][1.2] Enforce DNS resolution timeout
 2017-02-03 02:30:12 +05:00
Paul Tremberth 23905611b3 Enforce DNS resolution timeout 2017-02-02 17:58:05 +01:00
Paul Tremberth f3d5995329 Bump version: 1.2.1 → 1.2.2 2016-12-06 15:21:00 +01:00
Paul Tremberth ea968c6c3d Merge pull request #2423 from redapple/release-notes-1.2.2 
Update changelog for upcoming 1.2.2 release
 2016-12-06 15:18:31 +01:00
Paul Tremberth 09e310d0b7 Set release date for 1.2.2 2016-12-06 15:03:38 +01:00
Paul Tremberth aa2e1b030d Add reference to fixed scheduler settings documentation 2016-12-06 14:59:11 +01:00
Paul Tremberth 89d5f5acd3 Update changelog for upcoming 1.2.2 release 2016-12-06 14:59:11 +01:00
Mikhail Korobov 97df139119 Merge pull request #2417 from pawelmhm/scheduler-docs 
[MRG+1] Add docs for some scheduler settings
 2016-12-06 18:42:07 +05:00
Elias Dorneles c4e67c0696 Merge pull request #2421 from rolando-contrib/tests-bug 
[MRG+1] TST: Fix duplicated test name.
 2016-12-01 18:43:48 -02:00
Elias Dorneles f3a4420750 Merge pull request #2388 from redapple/robotparser-native-str 
[MRG+1] Parse robots.txt content as native str
 2016-12-01 18:43:23 -02:00
Mikhail Korobov 826b7e92b6 Merge pull request #2386 from rolando-contrib/manifest-tweak 
[MRG+1] Ignore explicitly compiled python files.
 2016-12-02 00:20:27 +05:00
Rolando Espinoza d9f43e21ba TST: Fix duplicated test name. 2016-12-01 11:56:33 -03:00
Elias Dorneles 5f0af90674 Merge pull request #2395 from elacuesta/robots_txt_relative_sitemap_url 
[MRG+1] Handle relative sitemap URLs in robots.txt
 2016-12-01 11:24:54 -02:00
Eugenio Lacuesta 5ff64ad015 handle relative sitemap urls in robots.txt 2016-12-01 09:53:40 -03:00
Elias Dorneles 2086ff4065 Merge pull request #2418 from ahlinc/fix_retriggered_issue_396 
[MRG+1] Fix #396 re-triggered issue
 2016-11-30 10:49:41 -02:00
Mikhail Korobov bba4645f21 Merge pull request #2404 from redapple/scrapy-version 
[MRG+1] Print more dependencies versions in "scrapy version" verbose output
 2016-11-30 17:01:37 +05:00
Andrew Hlynskyi 27606aad11 Fix #396 re-triggered issue 
The InteractiveShellEmbed class is a singleton
and we need to drop the instance with its clear_instance() method
to rebuild the instance from scratch with fresh environment
for each subsequent Scrapy's shell drop in.
 2016-11-30 09:52:44 +02:00
Paul Tremberth 624284e851 Fix indent 2016-11-29 18:18:59 +01:00
Pawel Miech f98ffb53b6 add docs for some scheduler settings 2016-11-29 16:52:54 +01:00
Paul Tremberth 01142e2ae5 Print more dependencies versions in "scrapy version" verbose output 2016-11-22 14:48:33 +01:00
Mikhail Korobov a07400ce0a Merge pull request #2396 from redapple/ubuntu-packages-toc 
[MRG] DOC Remove "Ubuntu" section from sidebar/ToC
 2016-11-22 18:21:38 +05:00
Pablo Hoffman d62776a858 mention scrapoxy in best practices doc 2016-11-16 12:19:32 -03:00
Paul Tremberth 8db8545393 Add "orphan" metadata for Ubuntu packages page 
As described in http://www.sphinx-doc.org/en/latest/markup/misc.html#metadata
 2016-11-16 13:56:58 +01:00
Paul Tremberth 11fe3751cf DOC Remove "Ubuntu" section from sidebar/ToC 2016-11-16 11:55:09 +01:00
Paul Tremberth 6cd35c77da Pass user-agent as native str when checking URLs against robots.txt 2016-11-15 17:38:32 +01:00
Elias Dorneles 1cbc57faa5 Merge pull request #2337 from muriloviana/add-middleware-template 
[MRG+1] Add middleware to template project (closes #2335)
 2016-11-15 12:58:40 -02:00
Mikhail Korobov 570e12b5db Merge pull request #2328 from scrapy/download-latency-meta-docs 
[MRG+1] Document `download_latency` meta key
 2016-11-14 21:24:14 +05:00
Valdir Stumm Junior 7025d6656a document download_latency meta key 2016-11-14 13:06:18 -02:00
Paul Tremberth 201a16f6d5 Merge pull request #2394 from nyov/fix-le 
[MRG+1] LinkExtractor PY3 'unicode' type fix
 2016-11-14 12:10:14 +01:00
nyov e8205f6733 LinkExtractor PY3 'unicode' type fix 2016-11-10 17:50:06 +00:00
Paul Tremberth de89b1b562 Merge pull request #2275 from scrapy/response-css-xpath-message 
[MRG+1] Add better messages for when response content isn't text (closes #2264)
 2016-11-10 11:38:22 +01:00
Mikhail Korobov 6d6dc7138b Merge pull request #2387 from rolando-contribute/conda-forge-updates 
[MRG+1] Update conda channel to conda-forge and show conda version badge.
 2016-11-10 00:35:43 +06:00
Paul Tremberth 28155dfccc Parse robots.txt content as native str 
Fixes #2373
 2016-11-09 12:20:06 +01:00
Rolando Espinoza 76459e1969 Update conda channel to conda-forge and show conda version badge. 2016-11-08 21:30:27 -03:00
Rolando Espinoza 3cd56da0cc Ignore explicitly compiled python files. 
This avoids to include compiled files in the templates directory.
 2016-11-08 20:52:32 -03:00
Mikhail Korobov ea83e67796 Merge pull request #2382 from redapple/slot-heartbeat-state-test 
Test Slot's heartbeat state before stopping it
 2016-11-08 18:32:07 +06:00
Paul Tremberth af2280e695 Update docstring 2016-11-08 13:30:51 +01:00
Paul Tremberth 27456996a9 Add assertion on crawler not running 2016-11-08 11:46:16 +01:00
Paul Tremberth 61efacdd1f Add testcase for catching exception from open_spider() from pipeline 2016-11-08 11:35:42 +01:00
Paul Tremberth 7727d87f64 Test Slot's heartbeat state before stopping it 
Also add a test on state of looping task in LogStats extension

Fixes #2011 and #2362
 2016-11-07 16:44:57 +01:00
Mikhail Korobov 0f5eb4cf88 Merge pull request #2380 from rahulkant13may/master 
Document update in "Using Firefox for scraping"
 2016-11-07 19:02:53 +06:00
Rahul Kant f56aef99c2 Add closing tag in <tbody> 2016-11-07 17:49:22 +05:30
Mikhail Korobov 9755ef933a Merge pull request #2369 from jc0n/media-pipeline-docs-typo 
Fix typo in media pipeline docs
 2016-10-29 12:46:48 +06:00
John O'Connor d85da273be Fix typo in media pipeline docs 2016-10-28 19:44:46 -07:00
Elias Dorneles a9c74dbe42 Merge pull request #2339 from visued/master 
[MRG+1] Update documentation to explain the use of double quotes on Windows (fixes #2325)
 2016-10-25 09:53:24 -02:00