RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,543 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

3543 Commits

Author SHA1 Message Date
Daniel Graña 7394969236 test pypy with travis 2013-08-27 15:31:08 -03:00
Daniel Graña 401e888ced test lucid and precise python packages under their respective python versions 2013-08-27 15:30:10 -03:00
Daniel Graña 588a262b73 fix crawling tests under twisted pre 11.0.0 2013-08-27 14:05:37 -03:00
Daniel Graña 92826586d5 py26 can not format zero length fields {} 2013-08-27 12:12:50 -03:00
Daniel Graña caa0f90263 test PotentiaDataLoss errors on unbound responses 2013-08-27 11:53:18 -03:00
Daniel Graña 6720acc1af Treat responses without content-length or Transfer-Encoding as good responses 
There is not a way to determine if responses without Content-Length or Transfer-Encoding are complete, this change treat them as good responses but flags them as "partial".
This is backout change only for this functionality of 3c64a989
 2013-08-27 11:10:20 -03:00
Pablo Hoffman da1f6d3144 remove unused imports and some assorted pylint-ing 2013-08-23 13:03:34 -03:00
Pablo Hoffman 19ff9ac4f9 url/body attributes of Request/Response objects are now immutable 2013-08-23 12:43:22 -03:00
Pablo Hoffman 86230c0ab8 added quantal & raring to support ubuntu releases 2013-08-22 21:49:55 -03:00
Daniel Graña efb4f32d80 do no include ResponseFailed if http11 handler is not enabled 2013-08-21 19:04:12 -03:00
Daniel Graña 3c64a989ca New HTTP client wraps connection losts in ResponseFailed exception. fix #373 2013-08-21 17:28:26 -03:00
Pablo Hoffman 071172cbd6 fix retry middleware which didn't retry certain connection errors after the upgrade to http1 client, closes GH-373 2013-08-21 14:32:33 -03:00
Pablo Hoffman 6e4ed8b004 Merge pull request #372 from kmike/xml-exporter-fix-for-2.7.4 
fix XmlItemExporter in Python 2.7.4 and 2.7.5
 2013-08-21 09:55:51 -07:00
Mikhail Korobov a6e6ca06b9 fix XmlItemExporter in Python 2.7.4 and 2.7.5 2013-08-21 06:05:40 +06:00
Mikhail Korobov 034ffae60f Recommend Pillow instead of PIL. Closes GH-317. 2013-08-18 00:44:01 +06:00
Pablo Hoffman f95b164d06 Merge pull request #367 from berendiwema/master 
#327 - Support STARTTLS / SSL option in email sender
 2013-08-14 07:17:30 -07:00
Berend Iwema 32b6364bcd #327 - Support STARTTLS / SSL option in email sender 2013-08-14 12:59:01 +02:00
Pablo Hoffman 49952a45ae add docstring to PythonItemExporter 2013-08-14 02:49:14 -03:00
Pablo Hoffman c0b26e3d49 minor updates to 0.18 release notes 2013-08-14 01:39:44 -03:00
Daniel Graña c6157d37e9 Merge pull request #366 from kalessin/exporter 
added a python native classes item exporter
 2013-08-13 06:55:27 -07:00
olveyra 456b6f2ef5 added a python native classes item exporter 2013-08-13 13:48:28 +00:00
Daniel Graña b43b5f575e adjust http11 pool size to per-domain concurrency 2013-08-12 22:43:40 -03:00
Pablo Hoffman 892386ee01 tox.ini: disable sitepackages on windows, as a compiler is often not available 2013-08-12 18:59:41 -03:00
Daniel Graña ed5b9068d2 fix contributters list format 2013-08-12 11:26:16 -03:00
Daniel Graña 80e25c5980 bumped version to 0.19.0 2013-08-09 19:09:47 -03:00
Daniel Graña a6693c9a5c updated release notes and bumped version to 0.18.0 2013-08-09 19:02:28 -03:00
Pablo Hoffman 4dc76e7cca fixed scrapy.utils.gz.gunzip() broken after changes from Python 2.7.3 to 2.7.4 2013-08-09 18:20:04 -03:00
Daniel Graña 29a12af1b0 Merge pull request #361 from hmsimha/master 
Doc fixes: Minor typo (was fixed on 0.16 branch so I figured it could be on master as well) and an error in the overview doc.
 2013-08-05 09:10:22 -07:00
Hart c00c4d7148 correction to description of example XPath retrieval in overview doc 2013-08-03 17:08:58 -07:00
Hart 0ad01c34d4 fixed typo to parallel fix on 0.16 branch 2013-08-03 17:06:10 -07:00
Nicolás Alejandro Ramírez Quiros c2a4046f14 Merge pull request #359 from rocioar/master 
Added COMPRESSION_ENABLED setting
 2013-08-01 10:43:47 -07:00
Rocio Aramberri d227d530f6 Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware 
Added COMPRESSION_ENABLE setting to docs

Added COMPRESSION_ENABLED setting to default settings
 2013-08-01 11:31:28 -03:00
arijitchakraborty fb770852e8 Skipping cookie retrieval for non http requests 2013-07-22 11:33:26 -03:00
Mikhail Korobov e1f0d0d45d Merge pull request #352 from arijitchakraborty/bugfix/cookie_retrieval_for_hosts_with_port 
Fixes for bug - Cookie retrieval for hosts with port + unittests for the...
 2013-07-18 09:21:13 -07:00
arijitchakraborty 66ff34cf05 improving hostname extraction 2013-07-18 20:51:56 +05:30
arijitchakraborty ebc136ddb0 Fixes for bug - Cookie retrieval for hosts with port + unittests for the fix 2013-07-18 19:42:58 +05:30
Daniel Graña d7e11082f2 it is not possible to enforce an upper limit when latency is out of control 2013-07-18 11:07:42 -03:00
Daniel Graña 3eb5d5e931 Merge pull request #330 from nramirezuy/spider-from_crawler 
use from_crawler method if available in spiders
 2013-07-16 12:42:31 -07:00
Daniel Graña d14541c5f1 Merge pull request #205 from joehillen/master 
PyPy support
 2013-07-16 12:34:17 -07:00
Daniel Graña d51f1ff33d Merge pull request #350 from dellis23/master 
This pull request fixes #349
 2013-07-16 12:33:21 -07:00
Daniel Graña d74b60051d take in count response latencies when testing download delays 2013-07-16 16:22:29 -03:00
Dan 1ca31244b0 Fixed ordering of super argument call. 2013-07-16 14:50:10 -04:00
Dan e12b689c4f Updated documentation of spider arguments to include required super call. 2013-07-16 14:26:53 -04:00
Daniel Graña 891fa980d2 Merge pull request #341 from kmike/test-running 
Improved tox test running & fixed django tests
 2013-07-16 11:16:43 -07:00
Daniel Graña 66aa1331fc Twisted pre 10.2 (lucid) can not setup the required ftp server for tests 2013-07-16 00:49:35 -03:00
Daniel Graña 26c7ec7d9f Merge pull request #329 from kalessin/ftphandler 
added ftp handler
 2013-07-15 16:39:46 -07:00
Daniel Graña f8c105d5c5 Merge branch 'gh-347'. thanks @chrisboo 2013-07-15 12:28:24 -03:00
Daniel Graña 9ff82d6695 use lxml parser with recover option to parse invalid sitemaps. closes #347 
thanks chrissboo for reporting the issue and proposing a fix.
 2013-07-15 12:26:16 -03:00
Mikhail Korobov 1a1c93fafe tiny FormRequest doc fix 2013-07-15 15:47:34 +06:00
Daniel Graña aa7b529975 do not log Starting/Stopping factory messages 2013-07-10 12:35:15 -03:00