RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,543 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

3543 Commits

Author SHA1 Message Date
Alex Cepoi b0845fa4ec remove redundant kwarg 2013-09-03 14:18:16 -03:00
Alex Cepoi eab547ce85 fix log intermixing in scrapy shell command 2013-09-03 14:18:13 -03:00
Alex Cepoi d311fefa7b fix some missing logs with new crawlerprocess 2013-09-03 14:18:11 -03:00
Alex Cepoi f9b18f9a59 port all scrapy commands to new CrawlerProcess 2013-09-03 14:18:06 -03:00
Alex Cepoi a5ffdf9bcb default to multi crawler commands 
compatibility for old single crawler process
deprecated command's `crawler` property
logs about settings only shown once
 2013-09-03 14:18:04 -03:00
Alex Cepoi 97d197267e disable logging for scrapy check 2013-09-03 14:18:00 -03:00
Alex Cepoi c09d414cac command with multiple crawlers, fix check command 2013-09-03 14:17:57 -03:00
Alex Cepoi 9c1585196f fix a possible error in contracts post hook 2013-09-03 14:17:51 -03:00
Daniel Graña ab74482315 Add 0,18.1 release notes 2013-08-27 18:45:21 -03:00
Daniel Graña 53bdf1c9e4 bumped version to 0.18.1 2013-08-27 18:42:10 -03:00
Daniel Graña d20304e588 remove extra import added by cherry picked changes 2013-08-27 18:31:06 -03:00
Daniel Graña 1994f38f4d fix crawling tests under twisted pre 11.0.0 2013-08-27 18:30:17 -03:00
Daniel Graña abf756fc5c py26 can not format zero length fields {} 2013-08-27 18:30:05 -03:00
Daniel Graña b15470dd20 test PotentiaDataLoss errors on unbound responses 2013-08-27 18:29:44 -03:00
Daniel Graña c4bf3249bc Treat responses without content-length or Transfer-Encoding as good responses 
There is not a way to determine if responses without Content-Length or Transfer-Encoding are complete, this change treat them as good responses but flags them as "partial".
This is backout change only for this functionality of 3c64a989
 2013-08-27 18:29:40 -03:00
Daniel Graña 6cbe68477c do no include ResponseFailed if http11 handler is not enabled 2013-08-27 18:28:50 -03:00
Daniel Graña 1a20bba54e New HTTP client wraps connection losts in ResponseFailed exception. fix #373 2013-08-27 18:28:42 -03:00
Daniel Graña 3b01bb8f03 limit travis-ci build matrix 2013-08-27 18:22:45 -03:00
Pablo Hoffman fa766d7eb6 Merge pull request #375 from peterarenot/patch-1 
Changed name of directory to save file to
 2013-08-27 06:46:34 -07:00
Peter 3283809f89 Fixed so it refers to the correct folder 2013-08-27 17:39:25 +10:00
Pablo Hoffman 1411923dd4 added quantal & raring to support ubuntu releases 2013-08-22 21:49:29 -03:00
Pablo Hoffman bb35ed017c fix retry middleware which didn't retry certain connection errors after the upgrade to http1 client, closes GH-373 2013-08-21 14:32:22 -03:00
Mikhail Korobov de3e451564 fix XmlItemExporter in Python 2.7.4 and 2.7.5 2013-08-21 13:57:02 -03:00
Pablo Hoffman c45e5f14ee minor updates to 0.18 release notes 2013-08-14 01:39:54 -03:00
Daniel Graña 0b6003148d fix contributters list format 2013-08-11 21:49:56 -03:00
Daniel Graña a6693c9a5c updated release notes and bumped version to 0.18.0 2013-08-09 19:02:28 -03:00
Pablo Hoffman 4dc76e7cca fixed scrapy.utils.gz.gunzip() broken after changes from Python 2.7.3 to 2.7.4 2013-08-09 18:20:04 -03:00
Daniel Graña 29a12af1b0 Merge pull request #361 from hmsimha/master 
Doc fixes: Minor typo (was fixed on 0.16 branch so I figured it could be on master as well) and an error in the overview doc.
 2013-08-05 09:10:22 -07:00
Hart c00c4d7148 correction to description of example XPath retrieval in overview doc 2013-08-03 17:08:58 -07:00
Hart 0ad01c34d4 fixed typo to parallel fix on 0.16 branch 2013-08-03 17:06:10 -07:00
Nicolás Alejandro Ramírez Quiros c2a4046f14 Merge pull request #359 from rocioar/master 
Added COMPRESSION_ENABLED setting
 2013-08-01 10:43:47 -07:00
Rocio Aramberri d227d530f6 Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware 
Added COMPRESSION_ENABLE setting to docs

Added COMPRESSION_ENABLED setting to default settings
 2013-08-01 11:31:28 -03:00
arijitchakraborty fb770852e8 Skipping cookie retrieval for non http requests 2013-07-22 11:33:26 -03:00
Mikhail Korobov e1f0d0d45d Merge pull request #352 from arijitchakraborty/bugfix/cookie_retrieval_for_hosts_with_port 
Fixes for bug - Cookie retrieval for hosts with port + unittests for the...
 2013-07-18 09:21:13 -07:00
arijitchakraborty 66ff34cf05 improving hostname extraction 2013-07-18 20:51:56 +05:30
arijitchakraborty ebc136ddb0 Fixes for bug - Cookie retrieval for hosts with port + unittests for the fix 2013-07-18 19:42:58 +05:30
Daniel Graña d7e11082f2 it is not possible to enforce an upper limit when latency is out of control 2013-07-18 11:07:42 -03:00
Daniel Graña 3eb5d5e931 Merge pull request #330 from nramirezuy/spider-from_crawler 
use from_crawler method if available in spiders
 2013-07-16 12:42:31 -07:00
Daniel Graña d14541c5f1 Merge pull request #205 from joehillen/master 
PyPy support
 2013-07-16 12:34:17 -07:00
Daniel Graña d51f1ff33d Merge pull request #350 from dellis23/master 
This pull request fixes #349
 2013-07-16 12:33:21 -07:00
Daniel Graña d74b60051d take in count response latencies when testing download delays 2013-07-16 16:22:29 -03:00
Dan 1ca31244b0 Fixed ordering of super argument call. 2013-07-16 14:50:10 -04:00
Dan e12b689c4f Updated documentation of spider arguments to include required super call. 2013-07-16 14:26:53 -04:00
Daniel Graña 891fa980d2 Merge pull request #341 from kmike/test-running 
Improved tox test running & fixed django tests
 2013-07-16 11:16:43 -07:00
Daniel Graña 66aa1331fc Twisted pre 10.2 (lucid) can not setup the required ftp server for tests 2013-07-16 00:49:35 -03:00
Daniel Graña 26c7ec7d9f Merge pull request #329 from kalessin/ftphandler 
added ftp handler
 2013-07-15 16:39:46 -07:00
Daniel Graña f8c105d5c5 Merge branch 'gh-347'. thanks @chrisboo 2013-07-15 12:28:24 -03:00
Daniel Graña 9ff82d6695 use lxml parser with recover option to parse invalid sitemaps. closes #347 
thanks chrissboo for reporting the issue and proposing a fix.
 2013-07-15 12:26:16 -03:00
Mikhail Korobov 1a1c93fafe tiny FormRequest doc fix 2013-07-15 15:47:34 +06:00
Daniel Graña aa7b529975 do not log Starting/Stopping factory messages 2013-07-10 12:35:15 -03:00