cb8e0d9bdd
added scrapy.utils.defer, moved deferred functions from scrapy.utils.misc to that module
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40271
2008-09-23 21:48:48 +00:00
6637d8cf68
removed location_str function incorrectly added to this module
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40270
2008-09-23 21:34:05 +00:00
dbfc6341c2
removed duplicated function convert_entity from scrapy.utils.misc
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40269
2008-09-23 20:36:32 +00:00
4842fe0679
removed unquote_html
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40268
2008-09-22 18:04:30 +00:00
2b6ed80a66
Change site-media with static in css file
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40267
2008-09-19 20:00:08 +00:00
e985b6c368
Comment blog templatetags calls
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40266
2008-09-19 19:57:45 +00:00
d3f66b7d39
Remove blog url an installed app setting
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40265
2008-09-19 19:53:57 +00:00
bd28045b99
Remove blog application which is not django 1.0
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40264
2008-09-19 19:19:28 +00:00
4d388c946e
Django 1.0 support over download application
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40263
2008-09-19 19:17:41 +00:00
62975817af
remove decobot reference and fix test_plugin
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40262
2008-09-19 16:57:57 +00:00
baabdb004d
remove decobot reference
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40261
2008-09-19 16:57:21 +00:00
d87f979d2a
grrr.. stupid bugfix
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40260
2008-09-19 15:21:47 +00:00
c05dcdab1d
removed references to python2.5
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40259
2008-09-19 14:11:14 +00:00
ddd7393a0a
reverted changeset 254
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40258
2008-09-19 12:50:19 +00:00
332460851b
ouch! reverted a failed commit and now yes, removed an unneeded None
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40257
2008-09-18 20:19:26 +00:00
133e9242d3
removed unneeded None
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40256
2008-09-18 20:13:31 +00:00
6ce7ae4df8
allow to set download delay as a scrapy setting
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40255
2008-09-18 20:11:30 +00:00
cfb56e39bb
bump version
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40254
2008-09-18 19:21:40 +00:00
c6d4a37a98
include templates
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40253
2008-09-18 17:48:35 +00:00
818d9baf69
new setup.py
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40252
2008-09-18 16:08:33 +00:00
a85e53053d
in the way of setuptooling
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40251
2008-09-18 14:58:43 +00:00
93c313853d
use request.deferred for cached valued too
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40250
2008-09-17 12:34:48 +00:00
6f2d36199b
media: use deferred of request to allow adding callback from get_media_requests
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40249
2008-09-16 23:40:48 +00:00
7932a8a5b6
add image pipeline that stores and can thumbs images in different sizes
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40248
2008-09-16 23:40:06 +00:00
4807f2eb82
move media_to_download hook to allow caching and prevent downloading
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40247
2008-09-16 23:39:26 +00:00
5048491f8d
mediapipeline: change cache variable by domaininfo
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40246
2008-09-16 19:31:25 +00:00
41a4f22c27
messy typo adding errback
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40245
2008-09-16 14:03:39 +00:00
e0d90f9d87
Scrapy architecture diagram 1st version, dia source
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40244
2008-09-16 11:56:15 +00:00
0a7a5e5b3d
added docs dir, guess for what? :)
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40243
2008-09-16 11:30:22 +00:00
b57148dfad
renamed scrapy.lib to scrapy.xlib
...
--HG--
rename : scrapy/trunk/scrapy/lib/BeautifulSoup.py => scrapy/trunk/scrapy/xlib/BeautifulSoup.py
rename : scrapy/trunk/scrapy/lib/__init__.py => scrapy/trunk/scrapy/xlib/__init__.py
rename : scrapy/trunk/scrapy/lib/lrucache.py => scrapy/trunk/scrapy/xlib/lrucache.py
rename : scrapy/trunk/scrapy/lib/lsprofcalltree.py => scrapy/trunk/scrapy/xlib/lsprofcalltree.py
rename : scrapy/trunk/scrapy/lib/pydispatch/__init__.py => scrapy/trunk/scrapy/xlib/pydispatch/__init__.py
rename : scrapy/trunk/scrapy/lib/pydispatch/dispatcher.py => scrapy/trunk/scrapy/xlib/pydispatch/dispatcher.py
rename : scrapy/trunk/scrapy/lib/pydispatch/errors.py => scrapy/trunk/scrapy/xlib/pydispatch/errors.py
rename : scrapy/trunk/scrapy/lib/pydispatch/license.txt => scrapy/trunk/scrapy/xlib/pydispatch/license.txt
rename : scrapy/trunk/scrapy/lib/pydispatch/robust.py => scrapy/trunk/scrapy/xlib/pydispatch/robust.py
rename : scrapy/trunk/scrapy/lib/pydispatch/robustapply.py => scrapy/trunk/scrapy/xlib/pydispatch/robustapply.py
rename : scrapy/trunk/scrapy/lib/pydispatch/saferef.py => scrapy/trunk/scrapy/xlib/pydispatch/saferef.py
rename : scrapy/trunk/scrapy/lib/spidermonkey/INSTALL.scrapy => scrapy/trunk/scrapy/xlib/spidermonkey/INSTALL.scrapy
rename : scrapy/trunk/scrapy/lib/spidermonkey/__init__.py => scrapy/trunk/scrapy/xlib/spidermonkey/__init__.py
rename : scrapy/trunk/scrapy/lib/spidermonkey/sm_settings.py => scrapy/trunk/scrapy/xlib/spidermonkey/sm_settings.py
rename : scrapy/trunk/scrapy/lib/spidermonkey/spidermonkey.py => scrapy/trunk/scrapy/xlib/spidermonkey/spidermonkey.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40242
2008-09-15 15:37:44 +00:00
2cb300b926
removed simplejson from scrapy code
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40241
2008-09-15 15:29:12 +00:00
08feb4df1c
ername methods and enforce extraction of Request objects as item media to download
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40240
2008-09-15 13:55:23 +00:00
0090fcb0f5
improved fix to support IE7 (thanks bill\!)
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40239
2008-09-15 13:08:44 +00:00
d7a88e8d77
fixed hack for IE6
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40238
2008-09-15 12:58:19 +00:00
31d3b2cf5a
added hack for IE6
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40237
2008-09-15 12:56:31 +00:00
4944762341
move bugtraps to avoid silence of bugs in media_downloaded or media_failed
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40236
2008-09-15 12:46:34 +00:00
ca695c8c79
small bugfix and add bugtraps
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40235
2008-09-15 08:57:51 +00:00
6407a744e3
remove item.image_urls from get_urls_from_item
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40234
2008-09-15 08:27:15 +00:00
af0808be95
add info object to public methods
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40233
2008-09-15 07:08:27 +00:00
4263c3a2e1
Adds docstrings to public methods, remove unused imports, normalize urls to requests.
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40232
2008-09-15 06:01:40 +00:00
31e497e5eb
add media_to_download hook support, and return cached result if available
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40231
2008-09-15 04:33:40 +00:00
df2f4b21b8
removed reference to guid (decobot specific) attribute in item
...
pipeline logs
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40230
2008-09-15 01:23:10 +00:00
3803f67645
mediamiddleware: bugfixes and remove prints
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40229
2008-09-12 21:55:14 +00:00
2fd5cf644e
- Support for not output in cluster status
...
- schedule option by default does not have output
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40228
2008-09-12 20:48:48 +00:00
d612701b66
add media pipeline skeleton
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40227
2008-09-12 19:58:45 +00:00
2dc20bb56e
Fixed condition of new option --quiet.
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40226
2008-09-11 16:35:50 +00:00
4cffc039ff
- leave in adaptors.py only the elemental adaptors pipeline code.
...
- moved to contrib/adaptorpipeline the advanced adaptors pipeline code.
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40225
2008-09-11 15:33:35 +00:00
23814537b3
Fixed bug in the total items count. Added new option --quiet to the action diff. This option doesn't show the report if there aren't differences.
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40224
2008-09-11 14:59:16 +00:00
2b6bcd61df
Small piece of code moved for not complying with conventions
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40223
2008-09-11 14:50:15 +00:00
06964cf36b
some small semantics errors fixed in xpath test
...
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40222
2008-09-11 12:13:33 +00:00
Pablo Hoffman
samus_
Matias Aguirre
Daniel Grana
Andres Moreira