RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,560 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

3560 Commits

Author SHA1 Message Date
Daniel Graña dc43890b59 Django 1.6 form validation errors now include ValidationError exception instances instead of just strings 2013-11-07 02:32:30 -02:00
Daniel Graña 24d071e42f shutdown the active crawler on SIGINT. fixes #450 2013-11-04 23:58:23 -02:00
Daniel Graña a288ced5f5 Add 0.18.4 release notes 2013-10-10 01:03:04 -02:00
Daniel Graña fde072a857 bumped version to 0.18.4 2013-10-10 01:01:40 -02:00
Daniel Graña 3d32c4f3b8 IPython refuses to update the namespace. fix #396 
IPython embedding code borrowed from https://github.com/mitsuhiko/werkzeug/pull/85
 2013-10-10 00:56:54 -02:00
Daniel Graña b1d891968c Fix AlreadyCalledError replacing a request in shell command. closes #407 2013-10-09 23:50:24 -02:00
Daniel Graña 89faf5294c Fix start_requests lazyness and early hangs 
- Removes new public methods added by #330 to Crawler and CrawlerProcess
- Add test for start_requests lazy evaluation
- Fix and test hangs when start_requests erroed before returning the generator
- Add test when start_requests fails while generating requests
- Simplify Crawler and CrawlerProcess implementation taking in count
  that only one spider can be attached per Crawler. As required by SEP-019
- "scrapy settings" command do not require starting a Crawler anymore
 2013-10-09 22:59:20 -02:00
Daniel Graña 614627b0b2 Add 0.18.3 release notes 2013-10-03 12:53:54 -03:00
Daniel Graña e3a7a25a0a bumped version to 0.18.3 2013-10-03 12:52:27 -03:00
Daniel Graña 12693a50cb fix regression on lazy evaluation of start requests 2013-10-03 12:37:06 -03:00
Daniel Graña e429f630b3 forms: do not submit reset inputs 2013-09-07 21:56:21 -03:00
Daniel Graña 912202e385 increase unittest timeouts to decrease travis false positive failures 2013-09-05 21:52:54 -03:00
Daniel Graña cfc2d46d99 backport master fixes to json exporter 2013-09-05 21:36:18 -03:00
Daniel Graña 06149e0543 Fix permission and set umask before generating sdist tarball 2013-09-03 23:21:29 -03:00
Daniel Graña 7811b4c01c Add 0.18.2 release notes 2013-09-03 14:27:22 -03:00
Daniel Graña 67cd04566f Bumped versiont to 0.18.2 2013-09-03 14:22:50 -03:00
Alex Cepoi 38528bc168 don't start reactor if no crawler is scheduled 2013-09-03 14:18:19 -03:00
Alex Cepoi b0845fa4ec remove redundant kwarg 2013-09-03 14:18:16 -03:00
Alex Cepoi eab547ce85 fix log intermixing in scrapy shell command 2013-09-03 14:18:13 -03:00
Alex Cepoi d311fefa7b fix some missing logs with new crawlerprocess 2013-09-03 14:18:11 -03:00
Alex Cepoi f9b18f9a59 port all scrapy commands to new CrawlerProcess 2013-09-03 14:18:06 -03:00
Alex Cepoi a5ffdf9bcb default to multi crawler commands 
compatibility for old single crawler process
deprecated command's `crawler` property
logs about settings only shown once
 2013-09-03 14:18:04 -03:00
Alex Cepoi 97d197267e disable logging for scrapy check 2013-09-03 14:18:00 -03:00
Alex Cepoi c09d414cac command with multiple crawlers, fix check command 2013-09-03 14:17:57 -03:00
Alex Cepoi 9c1585196f fix a possible error in contracts post hook 2013-09-03 14:17:51 -03:00
Daniel Graña ab74482315 Add 0,18.1 release notes 2013-08-27 18:45:21 -03:00
Daniel Graña 53bdf1c9e4 bumped version to 0.18.1 2013-08-27 18:42:10 -03:00
Daniel Graña d20304e588 remove extra import added by cherry picked changes 2013-08-27 18:31:06 -03:00
Daniel Graña 1994f38f4d fix crawling tests under twisted pre 11.0.0 2013-08-27 18:30:17 -03:00
Daniel Graña abf756fc5c py26 can not format zero length fields {} 2013-08-27 18:30:05 -03:00
Daniel Graña b15470dd20 test PotentiaDataLoss errors on unbound responses 2013-08-27 18:29:44 -03:00
Daniel Graña c4bf3249bc Treat responses without content-length or Transfer-Encoding as good responses 
There is not a way to determine if responses without Content-Length or Transfer-Encoding are complete, this change treat them as good responses but flags them as "partial".
This is backout change only for this functionality of 3c64a989
 2013-08-27 18:29:40 -03:00
Daniel Graña 6cbe68477c do no include ResponseFailed if http11 handler is not enabled 2013-08-27 18:28:50 -03:00
Daniel Graña 1a20bba54e New HTTP client wraps connection losts in ResponseFailed exception. fix #373 2013-08-27 18:28:42 -03:00
Daniel Graña 3b01bb8f03 limit travis-ci build matrix 2013-08-27 18:22:45 -03:00
Pablo Hoffman fa766d7eb6 Merge pull request #375 from peterarenot/patch-1 
Changed name of directory to save file to
 2013-08-27 06:46:34 -07:00
Peter 3283809f89 Fixed so it refers to the correct folder 2013-08-27 17:39:25 +10:00
Pablo Hoffman 1411923dd4 added quantal & raring to support ubuntu releases 2013-08-22 21:49:29 -03:00
Pablo Hoffman bb35ed017c fix retry middleware which didn't retry certain connection errors after the upgrade to http1 client, closes GH-373 2013-08-21 14:32:22 -03:00
Mikhail Korobov de3e451564 fix XmlItemExporter in Python 2.7.4 and 2.7.5 2013-08-21 13:57:02 -03:00
Pablo Hoffman c45e5f14ee minor updates to 0.18 release notes 2013-08-14 01:39:54 -03:00
Daniel Graña 0b6003148d fix contributters list format 2013-08-11 21:49:56 -03:00
Daniel Graña a6693c9a5c updated release notes and bumped version to 0.18.0 2013-08-09 19:02:28 -03:00
Pablo Hoffman 4dc76e7cca fixed scrapy.utils.gz.gunzip() broken after changes from Python 2.7.3 to 2.7.4 2013-08-09 18:20:04 -03:00
Daniel Graña 29a12af1b0 Merge pull request #361 from hmsimha/master 
Doc fixes: Minor typo (was fixed on 0.16 branch so I figured it could be on master as well) and an error in the overview doc.
 2013-08-05 09:10:22 -07:00
Hart c00c4d7148 correction to description of example XPath retrieval in overview doc 2013-08-03 17:08:58 -07:00
Hart 0ad01c34d4 fixed typo to parallel fix on 0.16 branch 2013-08-03 17:06:10 -07:00
Nicolás Alejandro Ramírez Quiros c2a4046f14 Merge pull request #359 from rocioar/master 
Added COMPRESSION_ENABLED setting
 2013-08-01 10:43:47 -07:00
Rocio Aramberri d227d530f6 Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware 
Added COMPRESSION_ENABLE setting to docs

Added COMPRESSION_ENABLED setting to default settings
 2013-08-01 11:31:28 -03:00
arijitchakraborty fb770852e8 Skipping cookie retrieval for non http requests 2013-07-22 11:33:26 -03:00