RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,110 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

2422 Commits

Author SHA1 Message Date
Pablo Hoffman caa64908d2 added tests for get_func_args() and support for more cases 2012-09-04 18:07:44 -03:00
Pablo Hoffman b901e64044 replaced memory usage acounting with (more portable) resource module, removed scrapy.utils.memory module. closes #161 2012-09-03 19:28:16 -03:00
Pablo Hoffman dd1398b280 removed another instance of scrapy.conf.settings singleton, this time from download handlers 2012-09-03 16:44:18 -03:00
Daniel Graña 3b768ae989 oops. remove santas debug line 2012-08-31 20:10:32 -03:00
Daniel Graña dcef7b03c1 format log lines lazily in case they are dropped by loglevels 2012-08-31 17:14:35 -03:00
Pablo Hoffman be206ca5ab added process_start_requests method to spider middlewares 2012-08-31 16:41:50 -03:00
Alex Cepoi 0350bed8fd parse command: fix crash in case of bad callback 2012-08-31 16:43:04 +02:00
Pablo Hoffman 94b40162a9 fixed tests to work on windows 2012-08-30 11:24:29 -03:00
Pablo Hoffman 3904f5f55a more reliable way to set subprocess PYTHONPATH in tests 2012-08-30 10:09:20 -03:00
Pablo Hoffman 6217f108cc avoid random "Interrupted system call" errors 2012-08-29 11:44:00 -03:00
Pablo Hoffman 8aa46a4b5d removed another usage of scrapy.conf.settings singleton 2012-08-29 11:43:01 -03:00
Pablo Hoffman 280d9d4094 added file missed in previous commit 2012-08-29 11:26:52 -03:00
Pablo Hoffman 70f8e517a1 promoted DjangoItem to main contrib 2012-08-29 11:23:11 -03:00
Pablo Hoffman 52151e8703 fixed bug in FeedExports extension, introduced in previous commit 2012-08-28 19:32:43 -03:00
Pablo Hoffman da234af456 Merge branch 'singleton_removal' 2012-08-28 18:42:33 -03:00
Pablo Hoffman babfc6e79b Updated documentation after singleton removal changes. 
Also removed some unused code and made some minor additional
refactoring.
 2012-08-28 18:35:57 -03:00
Pablo Hoffman c95f717ef2 fixed typo in tests 2012-08-22 13:50:23 -03:00
Pablo Hoffman 5fa5c4545b use ScrapyJSONEncoder JsonItemExporter & JsonLinesItemExporter, to support nested items properly 2012-08-22 13:46:50 -03:00
Pablo Hoffman 1e12c92b8f Removed signals/stats singletons 
This change removes singletons for stats collection and signal
dispatching facilities, by making them a member of the Crawler class.

Here are some examples to illustrates the old and new API:

Signals - before:

    from scrapy import signals
    from scrapy.xlib.pydispatch import dispatcher
    dispatcher.connect(self.spider_opened, signals.spider_opened)

Signals - now:

    from scrapy import signals
    crawler.signals.connect(self.spider.opened, signals.spider_opened)

Stats collection - before:

    from scrapy.stats import stats
    stats.inc_value('foo')

Stats collection - now:

    crawler.stats.inc_value('foo')

Backwards compatibility was retained as much as possible and the old API
has been properly flagged with deprecation warnings.
 2012-08-21 18:32:30 -03:00
Pablo Hoffman 36f47a4aec Removed per-spider settings concept, and scrapy.conf.settings singleton from many extensions and middlewares. There are some still remaining, that will be removed in future commits 2012-08-21 17:27:45 -03:00
Andrés Moreira ed8721059d Small improvement on *MemoryQueue performance (mainly on push time) 2012-08-10 10:38:20 -03:00
Daniel Graña 483a70cb5b Merge pull request #160 from midiotthimble/patch-1 
Pass into the model only existing fields

test passing OK and changes makes sense
 2012-08-09 05:41:50 -07:00
Pablo Hoffman 3bdd9b7b89 added more extensions to ignore on link extractors 2012-08-07 14:23:50 -03:00
Pablo Hoffman 16bee46c70 added ppt to scrapy.linkextractor.IGNORED_EXTENSIONS 2012-08-07 14:18:55 -03:00
Vladislav Poluhin 0497ac2fc8 Simple test for default values of model in DjangoItem 2012-07-26 20:55:55 +08:00
Vladislav e14c41636f Pass into the model only existing fields 
Model fields has default values and when field doesn't exists in item container, added `None` instead default value. My patch solves this problem.
 2012-07-25 10:10:46 +08:00
Victor Varvariuc eb91cad0da if you put in settings.py something like import local_settings and local_settings doesn't exist or contains errors you get: 
/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/utils/project.py:17: UserWarning: Cannot import scrapy settings module settings
warnings.warn("Cannot import scrapy settings module %s" % scrapy_module)
...
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/cmdline.py", line 117, in _run_command
cmd.run(args, opts)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/commands/crawl.py", line 43, in run
spider = self.crawler.spiders.create(spname, **opts.spargs)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/spidermanager.py", line 43, in create
raise KeyError("Spider not found: %s" % spider_name)
KeyError: 'Spider not found: fb_spider'

Which is not very descriptive.

Now showing details about the exception when ImportError was raises
 2012-07-09 08:39:14 +04:00
Daniel Graña 277ed0ae23 Merge pull request #145 from alexcepoi/cookies-changes 
domain and path support for request cookies
 2012-06-25 11:29:04 -07:00
Alexandru Cepoi 177c81745d domain and path support for request cookies 2012-06-25 20:17:59 +02:00
Pablo Hoffman 179e3810dc fixed links to doc. closes #150 2012-06-24 01:00:33 -03:00
Pablo Hoffman 700f20b28f Merge pull request #149 from alexcepoi/parse-changes 
documentation for `parse` command, debugging spiders section
 2012-06-22 17:50:40 -07:00
Alexandru Cepoi 2e05cf5685 fix small bug with parse command 2012-06-21 20:06:50 +02:00
Daniel Graña 27689009f1 fix urlparse monkeypatches for python 2.7.4. closes #144 2012-06-18 11:50:43 -03:00
Alexandru Cepoi 3b5cf31198 add --verbose option to parse command 2012-05-24 16:35:09 +02:00
Alexandru Cepoi 8e3c5f1bf7 add --depth field to parse command 2012-05-24 15:10:03 +02:00
Pablo Hoffman 8d77005047 scrapy shell: start shell in main thread and crawler in secondary thread, instead of the other way around. fixes #100 2012-05-22 19:15:54 -03:00
Daniel Graña 7fc573a230 fix libxml2 test 2012-05-16 18:54:02 -03:00
Daniel Graña f530b0b3eb make libxml2 optional now that lxml is the default 2012-05-16 18:17:51 -03:00
Pablo Hoffman b4f368c37e warn if Link objects are instantiated with unicode urls 2012-05-16 13:12:25 -03:00
Pablo Hoffman 30b6c77ce5 fixed typo in previous commit 2012-05-16 09:16:46 -03:00
Pablo Hoffman b53bc66c76 added lxml/libxml2 versions to 'scrapy version' output 2012-05-16 09:12:02 -03:00
Daniel Graña 43732e5042 Merge pull request #129 from saxicek/master 
ImagesPipeline should not fail if Item['images'] is not defined.
 2012-05-07 10:38:49 -07:00
Alexandru Cepoi 045ff8e5a7 fix `tail` issue when extracting nodes [lxml] 2012-05-07 17:23:27 +02:00
Libor Nenadl 2b93b0a93c Do not try to set item['images'] if it is not defined in the Item. 2012-05-05 14:53:02 +02:00
Daniel Graña 72b1c2e88b handle missing paths in urls as / 2012-05-03 13:56:45 -03:00
Pablo Hoffman 9c3b9f2968 fixed bug in json-rpc webservice reported in https://groups.google.com/d/topic/scrapy-users/qgVBmFybNAQ/discussion. also removed no longer supported 'run' command from extras/scrapy-ws.py 2012-05-03 12:05:40 -03:00
Daniel Graña 9d66d7cdf9 be consistent removing BOM from decoded bodies. #123 2012-04-27 00:31:03 -03:00
Daniel Graña 8cae228df1 do not treat input type "image" as form input. #111 2012-04-25 16:12:48 -03:00
Daniel Graña 5b9a7814a5 support TextResponse in open_in_browser util 2012-04-25 16:04:37 -03:00
Pablo Hoffman 7865fbf05a replace "import Image" by more standard "from PIL import Image". closes #88 2012-04-20 19:05:55 -03:00