RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,078 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

3078 Commits

Author SHA1 Message Date
Alex Cepoi 99b76eaa2c SEP-017 contracts: first draft 2012-08-21 02:47:35 +02:00
Daniel Graña 19bcb44c25 Merge pull request #165 from andrix/master 
Just a simple refactoring on *MemoryQueue classes that improve the performance
 2012-08-10 15:04:30 -07:00
Andrés Moreira ed8721059d Small improvement on *MemoryQueue performance (mainly on push time) 2012-08-10 10:38:20 -03:00
Daniel Graña 483a70cb5b Merge pull request #160 from midiotthimble/patch-1 
Pass into the model only existing fields

test passing OK and changes makes sense
 2012-08-09 05:41:50 -07:00
Pablo Hoffman 3bdd9b7b89 added more extensions to ignore on link extractors 2012-08-07 14:23:50 -03:00
Pablo Hoffman 16bee46c70 added ppt to scrapy.linkextractor.IGNORED_EXTENSIONS 2012-08-07 14:18:55 -03:00
Daniel Graña abcc8c9f63 Recommend pypi as single way to install on Windows 2012-08-06 10:21:13 -03:00
Daniel Graña b7b0c49520 append parse command to example code sections in docs. closes #162 2012-08-06 09:10:16 -03:00
Vladislav Poluhin 0497ac2fc8 Simple test for default values of model in DjangoItem 2012-07-26 20:55:55 +08:00
Vladislav e14c41636f Pass into the model only existing fields 
Model fields has default values and when field doesn't exists in item container, added `None` instead default value. My patch solves this problem.
 2012-07-25 10:10:46 +08:00
Pablo Hoffman 832e45073b fixed typo in stats documentation. closes #159 2012-07-20 17:13:06 -03:00
Pablo Hoffman 831a45009a Merge pull request #155 from warvariuc/import_settings_module_verbose 
Spider not found - confusion
 2012-07-09 12:39:01 -07:00
Victor Varvariuc eb91cad0da if you put in settings.py something like import local_settings and local_settings doesn't exist or contains errors you get: 
/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/utils/project.py:17: UserWarning: Cannot import scrapy settings module settings
warnings.warn("Cannot import scrapy settings module %s" % scrapy_module)
...
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/cmdline.py", line 117, in _run_command
cmd.run(args, opts)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/commands/crawl.py", line 43, in run
spider = self.crawler.spiders.create(spname, **opts.spargs)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/spidermanager.py", line 43, in create
raise KeyError("Spider not found: %s" % spider_name)
KeyError: 'Spider not found: fb_spider'

Which is not very descriptive.

Now showing details about the exception when ImportError was raises
 2012-07-09 08:39:14 +04:00
Pablo Hoffman 99bfd1e590 Merge pull request #154 from vially/patch-1 
Fix typo in tutorial
 2012-07-04 04:36:16 -07:00
Valentin-Costel Hăloiu 00bfb37e79 Update master 2012-07-04 06:55:01 +03:00
Daniel Graña a40090a6a7 Merge pull request #151 from msabramo/tox 
Add support for tox (http://tox.testrun.org/)
 2012-06-29 05:37:24 -07:00
Marc Abramowitz 61952c3be6 Add .tox to .gitignore 2012-06-28 12:43:33 -07:00
Marc Abramowitz 9ae8ea96c4 Add tox.ini for tox (http://tox.testrun.org/) 2012-06-28 12:43:12 -07:00
Daniel Graña 277ed0ae23 Merge pull request #145 from alexcepoi/cookies-changes 
domain and path support for request cookies
 2012-06-25 11:29:04 -07:00
Pablo Hoffman 8b30575dd3 Merge pull request #147 from tonal/cfg-jsons 
Configure json services members
 2012-06-25 11:27:13 -07:00
Alexandru Cepoi 177c81745d domain and path support for request cookies 2012-06-25 20:17:59 +02:00
Alexandr N Zamaraev (aka tonal) cf968c328f Configure json services members 2012-06-25 12:50:48 +07:00
Pablo Hoffman 179e3810dc fixed links to doc. closes #150 2012-06-24 01:00:33 -03:00
Pablo Hoffman 700f20b28f Merge pull request #149 from alexcepoi/parse-changes 
documentation for `parse` command, debugging spiders section
 2012-06-22 17:50:40 -07:00
Alexandru Cepoi 2e05cf5685 fix small bug with parse command 2012-06-21 20:06:50 +02:00
Alexandru Cepoi f4faa19e31 added docs topic debugging spiders 2012-06-21 20:03:33 +02:00
Pablo Hoffman 4eeda53b0d Merge pull request #148 from tonal/cfg-launcher 
Add launcher class to config
 2012-06-19 08:14:58 -07:00
Alexandr N Zamaraev (aka tonal) 3bddedfc6d Add launcher class to config 2012-06-19 22:01:53 +07:00
Daniel Graña 27689009f1 fix urlparse monkeypatches for python 2.7.4. closes #144 2012-06-18 11:50:43 -03:00
Alexandru Cepoi 3e05a2ecf6 update docs for parse command 2012-06-12 18:28:10 +02:00
Pablo Hoffman e1be9c01bc updated FAQ about bot bans 2012-06-08 18:33:53 -03:00
Daniel Graña 3d6edc2f40 Merge pull request #135 from alexcepoi/parse-changes 
Add --depth option to parse
 2012-06-07 07:52:27 -07:00
Alexandru Cepoi 3b5cf31198 add --verbose option to parse command 2012-05-24 16:35:09 +02:00
Alexandru Cepoi 8e3c5f1bf7 add --depth field to parse command 2012-05-24 15:10:03 +02:00
Pablo Hoffman 8d77005047 scrapy shell: start shell in main thread and crawler in secondary thread, instead of the other way around. fixes #100 2012-05-22 19:15:54 -03:00
Pablo Hoffman b33303779a scrapyd.launcher: make SCRAPY_LOG_FILE and SCRAPY_FEED_URI optional 2012-05-21 14:29:15 -03:00
Daniel Graña 35ef7de546 add travis-ci build status to README 2012-05-17 09:07:36 -03:00
Daniel Graña e77e4b5f6e add precise to travis-ci build enviroments 2012-05-17 09:03:28 -03:00
Daniel Graña 3096e46401 add requirements file per travis env 2012-05-17 08:53:18 -03:00
Daniel Graña 7740581f88 Merge remote-tracking branch 'upstream/master' 2012-05-17 08:50:28 -03:00
Daniel Graña c3a3108799 multiple build enviroments for travis-ci 2012-05-17 08:49:45 -03:00
Pablo Hoffman 1bc18434f5 removed obsolete entries from MANIFEST.in 2012-05-17 01:09:48 -03:00
Daniel Graña 7fc573a230 fix libxml2 test 2012-05-16 18:54:02 -03:00
Daniel Graña f530b0b3eb make libxml2 optional now that lxml is the default 2012-05-16 18:17:51 -03:00
Daniel Graña 8376d95ce8 add travis-ci build configuration file 2012-05-16 16:54:12 -03:00
Pablo Hoffman b4f368c37e warn if Link objects are instantiated with unicode urls 2012-05-16 13:12:25 -03:00
Pablo Hoffman 30b6c77ce5 fixed typo in previous commit 2012-05-16 09:16:46 -03:00
Pablo Hoffman b53bc66c76 added lxml/libxml2 versions to 'scrapy version' output 2012-05-16 09:12:02 -03:00
Daniel Graña d74a067227 require w3lib 1.2 or greater 2012-05-15 17:11:51 -03:00
Daniel Graña ae2ff4d33a update news file with 0.14.4 release notes 2012-05-15 16:16:02 -03:00