RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,529 Commits 49 Branches 132 Tags 176 MiB
Commit Graph

3529 Commits

Author SHA1 Message Date
Daniel Graña c4bf3249bc Treat responses without content-length or Transfer-Encoding as good responses 
There is not a way to determine if responses without Content-Length or Transfer-Encoding are complete, this change treat them as good responses but flags them as "partial".
This is backout change only for this functionality of 3c64a989
 2013-08-27 18:29:40 -03:00
Daniel Graña 6cbe68477c do no include ResponseFailed if http11 handler is not enabled 2013-08-27 18:28:50 -03:00
Daniel Graña 1a20bba54e New HTTP client wraps connection losts in ResponseFailed exception. fix #373 2013-08-27 18:28:42 -03:00
Daniel Graña 3b01bb8f03 limit travis-ci build matrix 2013-08-27 18:22:45 -03:00
Pablo Hoffman fa766d7eb6 Merge pull request #375 from peterarenot/patch-1 
Changed name of directory to save file to
 2013-08-27 06:46:34 -07:00
Peter 3283809f89 Fixed so it refers to the correct folder 2013-08-27 17:39:25 +10:00
Pablo Hoffman 1411923dd4 added quantal & raring to support ubuntu releases 2013-08-22 21:49:29 -03:00
Pablo Hoffman bb35ed017c fix retry middleware which didn't retry certain connection errors after the upgrade to http1 client, closes GH-373 2013-08-21 14:32:22 -03:00
Mikhail Korobov de3e451564 fix XmlItemExporter in Python 2.7.4 and 2.7.5 2013-08-21 13:57:02 -03:00
Pablo Hoffman c45e5f14ee minor updates to 0.18 release notes 2013-08-14 01:39:54 -03:00
Daniel Graña 0b6003148d fix contributters list format 2013-08-11 21:49:56 -03:00
Daniel Graña a6693c9a5c updated release notes and bumped version to 0.18.0 2013-08-09 19:02:28 -03:00
Pablo Hoffman 4dc76e7cca fixed scrapy.utils.gz.gunzip() broken after changes from Python 2.7.3 to 2.7.4 2013-08-09 18:20:04 -03:00
Daniel Graña 29a12af1b0 Merge pull request #361 from hmsimha/master 
Doc fixes: Minor typo (was fixed on 0.16 branch so I figured it could be on master as well) and an error in the overview doc.
 2013-08-05 09:10:22 -07:00
Hart c00c4d7148 correction to description of example XPath retrieval in overview doc 2013-08-03 17:08:58 -07:00
Hart 0ad01c34d4 fixed typo to parallel fix on 0.16 branch 2013-08-03 17:06:10 -07:00
Nicolás Alejandro Ramírez Quiros c2a4046f14 Merge pull request #359 from rocioar/master 
Added COMPRESSION_ENABLED setting
 2013-08-01 10:43:47 -07:00
Rocio Aramberri d227d530f6 Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware 
Added COMPRESSION_ENABLE setting to docs

Added COMPRESSION_ENABLED setting to default settings
 2013-08-01 11:31:28 -03:00
arijitchakraborty fb770852e8 Skipping cookie retrieval for non http requests 2013-07-22 11:33:26 -03:00
Mikhail Korobov e1f0d0d45d Merge pull request #352 from arijitchakraborty/bugfix/cookie_retrieval_for_hosts_with_port 
Fixes for bug - Cookie retrieval for hosts with port + unittests for the...
 2013-07-18 09:21:13 -07:00
arijitchakraborty 66ff34cf05 improving hostname extraction 2013-07-18 20:51:56 +05:30
arijitchakraborty ebc136ddb0 Fixes for bug - Cookie retrieval for hosts with port + unittests for the fix 2013-07-18 19:42:58 +05:30
Daniel Graña d7e11082f2 it is not possible to enforce an upper limit when latency is out of control 2013-07-18 11:07:42 -03:00
Daniel Graña 3eb5d5e931 Merge pull request #330 from nramirezuy/spider-from_crawler 
use from_crawler method if available in spiders
 2013-07-16 12:42:31 -07:00
Daniel Graña d14541c5f1 Merge pull request #205 from joehillen/master 
PyPy support
 2013-07-16 12:34:17 -07:00
Daniel Graña d51f1ff33d Merge pull request #350 from dellis23/master 
This pull request fixes #349
 2013-07-16 12:33:21 -07:00
Daniel Graña d74b60051d take in count response latencies when testing download delays 2013-07-16 16:22:29 -03:00
Dan 1ca31244b0 Fixed ordering of super argument call. 2013-07-16 14:50:10 -04:00
Dan e12b689c4f Updated documentation of spider arguments to include required super call. 2013-07-16 14:26:53 -04:00
Daniel Graña 891fa980d2 Merge pull request #341 from kmike/test-running 
Improved tox test running & fixed django tests
 2013-07-16 11:16:43 -07:00
Daniel Graña 66aa1331fc Twisted pre 10.2 (lucid) can not setup the required ftp server for tests 2013-07-16 00:49:35 -03:00
Daniel Graña 26c7ec7d9f Merge pull request #329 from kalessin/ftphandler 
added ftp handler
 2013-07-15 16:39:46 -07:00
Daniel Graña f8c105d5c5 Merge branch 'gh-347'. thanks @chrisboo 2013-07-15 12:28:24 -03:00
Daniel Graña 9ff82d6695 use lxml parser with recover option to parse invalid sitemaps. closes #347 
thanks chrissboo for reporting the issue and proposing a fix.
 2013-07-15 12:26:16 -03:00
Mikhail Korobov 1a1c93fafe tiny FormRequest doc fix 2013-07-15 15:47:34 +06:00
Daniel Graña aa7b529975 do not log Starting/Stopping factory messages 2013-07-10 12:35:15 -03:00
Mikhail Korobov 3fe2a32683 handle GET parameters for AJAX crawlable URLs: 2013-07-09 01:58:50 +06:00
Mikhail Korobov 8c555fd467 Enable Travis CI for pull requests and feature branches. Fix GH-340. 2013-07-09 01:50:52 +06:00
Mikhail Korobov 0d9f7843ba allow passing arguments to runtests script via tox 2013-07-09 00:41:13 +06:00
Mikhail Korobov f29e5d0ca1 reuse runtests.sh and runtests.bat scripts in tox.ini 2013-07-09 00:38:00 +06:00
Mikhail Korobov 3a6dff3090 make tox test running more consistent with bin/runtests.sh script 2013-07-09 00:30:39 +06:00
Mikhail Korobov 2105ec58eb add django to test requirements and fix test_djangoitem for modern django versions 2013-07-09 00:28:39 +06:00
Mikhail Korobov b98e80cbb8 use tox deps instead of "manual" pip install; add lucid and precise environments to tox.ini 2013-07-09 00:26:59 +06:00
Mikhail Korobov 35fbec2e63 allow passing arguments to tox 
e.g. "tox -- scrapy.tests.test_contrib_loader" runs test_contrib_loader for Python 2.6 and 2.7
 2013-07-08 22:51:03 +06:00
Daniel Graña 6db259c075 Merge pull request #338 from kmike/patch-3 
DownloaderMiddleware docs fix
 2013-07-08 07:12:42 -07:00
Mikhail Korobov ac2fadf3ab DownloaderMiddleware.process_response docs fix 
"returns an exception" -> "raises an exception"
 2013-07-08 19:41:58 +06:00
Pablo Hoffman 9a13a0fe74 Merge pull request #337 from kmike/patch-2 
improve docs for DownloaderMiddleware.process_response
 2013-07-08 06:21:49 -07:00
Mikhail Korobov 39e5da5f66 improve docs for DownloaderMiddleware.process_response 2013-07-08 19:17:29 +06:00
Pablo Hoffman 163592ee40 Merge pull request #334 from nramirezuy/feedexport-remove_multispider 
feedexport removed multispider support
 2013-07-03 14:26:49 -07:00
nramirezuy 1fc2bc3489 feedexport removed multispider support 2013-07-03 18:24:18 -03:00