RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,069 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

3069 Commits

Author SHA1 Message Date
Vladislav e14c41636f Pass into the model only existing fields 
Model fields has default values and when field doesn't exists in item container, added `None` instead default value. My patch solves this problem.
 2012-07-25 10:10:46 +08:00
Pablo Hoffman 832e45073b fixed typo in stats documentation. closes #159 2012-07-20 17:13:06 -03:00
Pablo Hoffman 831a45009a Merge pull request #155 from warvariuc/import_settings_module_verbose 
Spider not found - confusion
 2012-07-09 12:39:01 -07:00
Victor Varvariuc eb91cad0da if you put in settings.py something like import local_settings and local_settings doesn't exist or contains errors you get: 
/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/utils/project.py:17: UserWarning: Cannot import scrapy settings module settings
warnings.warn("Cannot import scrapy settings module %s" % scrapy_module)
...
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/cmdline.py", line 117, in _run_command
cmd.run(args, opts)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/commands/crawl.py", line 43, in run
spider = self.crawler.spiders.create(spname, **opts.spargs)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/spidermanager.py", line 43, in create
raise KeyError("Spider not found: %s" % spider_name)
KeyError: 'Spider not found: fb_spider'

Which is not very descriptive.

Now showing details about the exception when ImportError was raises
 2012-07-09 08:39:14 +04:00
Pablo Hoffman 99bfd1e590 Merge pull request #154 from vially/patch-1 
Fix typo in tutorial
 2012-07-04 04:36:16 -07:00
Valentin-Costel Hăloiu 00bfb37e79 Update master 2012-07-04 06:55:01 +03:00
Daniel Graña a40090a6a7 Merge pull request #151 from msabramo/tox 
Add support for tox (http://tox.testrun.org/)
 2012-06-29 05:37:24 -07:00
Marc Abramowitz 61952c3be6 Add .tox to .gitignore 2012-06-28 12:43:33 -07:00
Marc Abramowitz 9ae8ea96c4 Add tox.ini for tox (http://tox.testrun.org/) 2012-06-28 12:43:12 -07:00
Daniel Graña 277ed0ae23 Merge pull request #145 from alexcepoi/cookies-changes 
domain and path support for request cookies
 2012-06-25 11:29:04 -07:00
Pablo Hoffman 8b30575dd3 Merge pull request #147 from tonal/cfg-jsons 
Configure json services members
 2012-06-25 11:27:13 -07:00
Alexandru Cepoi 177c81745d domain and path support for request cookies 2012-06-25 20:17:59 +02:00
Alexandr N Zamaraev (aka tonal) cf968c328f Configure json services members 2012-06-25 12:50:48 +07:00
Pablo Hoffman 179e3810dc fixed links to doc. closes #150 2012-06-24 01:00:33 -03:00
Pablo Hoffman 700f20b28f Merge pull request #149 from alexcepoi/parse-changes 
documentation for `parse` command, debugging spiders section
 2012-06-22 17:50:40 -07:00
Alexandru Cepoi 2e05cf5685 fix small bug with parse command 2012-06-21 20:06:50 +02:00
Alexandru Cepoi f4faa19e31 added docs topic debugging spiders 2012-06-21 20:03:33 +02:00
Pablo Hoffman 4eeda53b0d Merge pull request #148 from tonal/cfg-launcher 
Add launcher class to config
 2012-06-19 08:14:58 -07:00
Alexandr N Zamaraev (aka tonal) 3bddedfc6d Add launcher class to config 2012-06-19 22:01:53 +07:00
Daniel Graña 27689009f1 fix urlparse monkeypatches for python 2.7.4. closes #144 2012-06-18 11:50:43 -03:00
Alexandru Cepoi 3e05a2ecf6 update docs for parse command 2012-06-12 18:28:10 +02:00
Pablo Hoffman e1be9c01bc updated FAQ about bot bans 2012-06-08 18:33:53 -03:00
Daniel Graña 3d6edc2f40 Merge pull request #135 from alexcepoi/parse-changes 
Add --depth option to parse
 2012-06-07 07:52:27 -07:00
Alexandru Cepoi 3b5cf31198 add --verbose option to parse command 2012-05-24 16:35:09 +02:00
Alexandru Cepoi 8e3c5f1bf7 add --depth field to parse command 2012-05-24 15:10:03 +02:00
Pablo Hoffman 8d77005047 scrapy shell: start shell in main thread and crawler in secondary thread, instead of the other way around. fixes #100 2012-05-22 19:15:54 -03:00
Pablo Hoffman b33303779a scrapyd.launcher: make SCRAPY_LOG_FILE and SCRAPY_FEED_URI optional 2012-05-21 14:29:15 -03:00
Daniel Graña 35ef7de546 add travis-ci build status to README 2012-05-17 09:07:36 -03:00
Daniel Graña e77e4b5f6e add precise to travis-ci build enviroments 2012-05-17 09:03:28 -03:00
Daniel Graña 3096e46401 add requirements file per travis env 2012-05-17 08:53:18 -03:00
Daniel Graña 7740581f88 Merge remote-tracking branch 'upstream/master' 2012-05-17 08:50:28 -03:00
Daniel Graña c3a3108799 multiple build enviroments for travis-ci 2012-05-17 08:49:45 -03:00
Pablo Hoffman 1bc18434f5 removed obsolete entries from MANIFEST.in 2012-05-17 01:09:48 -03:00
Daniel Graña 7fc573a230 fix libxml2 test 2012-05-16 18:54:02 -03:00
Daniel Graña f530b0b3eb make libxml2 optional now that lxml is the default 2012-05-16 18:17:51 -03:00
Daniel Graña 8376d95ce8 add travis-ci build configuration file 2012-05-16 16:54:12 -03:00
Pablo Hoffman b4f368c37e warn if Link objects are instantiated with unicode urls 2012-05-16 13:12:25 -03:00
Pablo Hoffman 30b6c77ce5 fixed typo in previous commit 2012-05-16 09:16:46 -03:00
Pablo Hoffman b53bc66c76 added lxml/libxml2 versions to 'scrapy version' output 2012-05-16 09:12:02 -03:00
Daniel Graña d74a067227 require w3lib 1.2 or greater 2012-05-15 17:11:51 -03:00
Daniel Graña ae2ff4d33a update news file with 0.14.4 release notes 2012-05-15 16:16:02 -03:00
Pablo Hoffman 9686f97242 added precise to supported ubuntu distros 2012-05-12 19:54:36 -03:00
Pablo Hoffman 58e88ed246 scrapyd: do not set SCRAPY_FEED_URI/SCRAPY_LOG_FILE if items_dir/logs_dir settings are not set 2012-05-08 17:43:00 -03:00
Daniel Graña 43732e5042 Merge pull request #129 from saxicek/master 
ImagesPipeline should not fail if Item['images'] is not defined.
 2012-05-07 10:38:49 -07:00
Daniel Graña 69078368ab Merge pull request #130 from alexcepoi/lxml-fixes 
fix `tail` issue when extracting nodes [lxml]
 2012-05-07 08:37:36 -07:00
Alexandru Cepoi 045ff8e5a7 fix `tail` issue when extracting nodes [lxml] 2012-05-07 17:23:27 +02:00
Libor Nenadl 2b93b0a93c Do not try to set item['images'] if it is not defined in the Item. 2012-05-05 14:53:02 +02:00
Daniel Graña 43028876b5 Merge pull request #128 from dangra/cannonicalize-missing-url-path 
handle missing paths in urls as /
 2012-05-03 10:27:11 -07:00
Daniel Graña 72b1c2e88b handle missing paths in urls as / 2012-05-03 13:56:45 -03:00
Pablo Hoffman 9c3b9f2968 fixed bug in json-rpc webservice reported in https://groups.google.com/d/topic/scrapy-users/qgVBmFybNAQ/discussion. also removed no longer supported 'run' command from extras/scrapy-ws.py 2012-05-03 12:05:40 -03:00