301 Commits

Author	SHA1	Message	Date
Ismael Carnales	ada46a2dbb	styled imagesp doc	2009-09-11 15:30:46 -03:00
Pablo Hoffman	be0f2beef0	more cleanup to scheduler middelware doc, and permanentely moved to experimental doc	2009-09-11 13:27:31 -03:00
Pablo Hoffman	0af052b68f	removed confusing title	2009-09-11 12:19:18 -03:00
Pablo Hoffman	f3240748cb	changed link to scheduler middleware doc, now in experimental	2009-09-11 12:03:23 -03:00
Ismael Carnales	3998a0cb58	added more scheduler middleware documentation, and moved it to experimental ... --HG-- rename : docs/topics/scheduler-middleware.rst => docs/experimental/scheduler-middleware.rst	2009-09-11 11:58:53 -03:00
Pablo Hoffman	d242a20573	updated images pipeline doc	2009-09-11 11:47:12 -03:00
Pablo Hoffman	f1bb8dc2a3	first cleanup of spider manager api ... - removed asdict() and reload() methods - added list() method - removed default spider	2009-09-10 19:06:46 -03:00
Pablo Hoffman	f85813cd94	added FAQ entry about scrapy recipes and community spiders	2009-09-10 18:32:50 -03:00
Pablo Hoffman	269724a2b7	added Debugger extension, removed StackTraceDump from extensions available by default	2009-09-08 22:32:17 -03:00
Ismael Carnales	4ddfa9a2a3	stlyed downloaded middleware doc	2009-09-07 12:18:57 -03:00
Ismael Carnales	e3df11e5bb	added module directive to spidermw documentation	2009-09-07 12:03:24 -03:00
Pablo Hoffman	827aa19c6e	removed obsolete scrapy.utils.db module	2009-09-04 17:38:14 -03:00
Pablo Hoffman	861a803cc3	removed obsolete RestrictMiddleware	2009-09-04 17:22:56 -03:00
Ismael Carnales	7e2587169b	added missing middleware docs	2009-09-04 12:39:02 -03:00
Pablo Hoffman	aefb94063a	more updates to spider middleware doc	2009-09-04 13:46:04 -03:00
Pablo Hoffman	d04640be5c	some improvements to spider middleware doc	2009-09-04 13:29:16 -03:00
Pablo Hoffman	96bb223c13	removed (pretty useless) DebugMiddleware	2009-09-04 12:59:58 -03:00
Pablo Hoffman	8a715701ec	fixed another doc typo	2009-09-03 14:31:00 -03:00
Ismael Carnales	3c1bb7bc40	fixed typo in djangoitems doc (thanks anibal)	2009-09-03 11:23:25 -03:00
Daniel Grana	0e7b2a6da5	write header line by default when using csv exporter ... --HG-- extra : rebase_source : 2d2d7153dde5e3f77e682e16d2e4408f732f234e	2009-09-03 13:58:39 -03:00
Pablo Hoffman	596d2c4479	moved CoreStats extension to scrapy.contrib.corestats ... --HG-- rename : scrapy/stats/corestats.py => scrapy/contrib/corestats.py	2009-09-01 23:00:49 -03:00
Pablo Hoffman	6a50af05d7	removed useless SpiderReloader extension	2009-09-01 22:49:15 -03:00
Pablo Hoffman	79851aefa6	moved SpiderProfiler extension to scrapy.contrib_exp and removed references from documentation ... --HG-- rename : scrapy/contrib/spider/profiler.py => scrapy/contrib_exp/spiderprofiler.py	2009-09-01 22:38:37 -03:00
Pablo Hoffman	d3c51fd6f2	improved images pipeline documentation	2009-09-01 21:07:47 -03:00
Pablo Hoffman	18fd635124	another doc typo	2009-09-01 12:52:40 -03:00
Pablo Hoffman	538cc9803a	fixed doc typo	2009-09-01 12:47:53 -03:00
Pablo Hoffman	df0e1f005f	exporters doc: fixed example and some typos	2009-09-01 08:56:54 -03:00
Pablo Hoffman	ac8f46ce9e	added File Export Pipeline reference to Exporters doc	2009-08-31 21:01:35 -03:00
Pablo Hoffman	8d006e9ea1	moved item exporters doc to stable doc ... --HG-- rename : docs/experimental/exporters.rst => docs/topics/exporters.rst	2009-08-31 20:47:12 -03:00
Pablo Hoffman	0b152c99b5	added File Export Pipeline, a wrapper to use Item Exporters as Item Pipelines	2009-08-31 20:40:41 -03:00
Pablo Hoffman	8fab524978	moved engine.getstatus() method to scrapy.utils.engine function, to leave reporting logic out of engine code. added est() shortcut to telnet console	2009-08-31 12:44:32 -03:00
Pablo Hoffman	884f0c878f	Stats collectin: fixed race condition between stats persistance and population of stats on domain close	2009-08-29 19:44:13 -03:00
Pablo Hoffman	895c70e036	doc: fixed some links to scrapy-ctl topic	2009-08-29 18:23:55 -03:00
Pablo Hoffman	64572124e0	added doc about SCRAPY_SETTINGS_MODULE	2009-08-29 18:20:13 -03:00
Pablo Hoffman	9dc7a3a373	some minor adjustments to logging doc	2009-08-29 18:10:28 -03:00
Pablo Hoffman	60cbf24c89	more cleanups to startproject and project templates ... --HG-- rename : scrapy/templates/project/root/scrapy-ctl.py => scrapy/templates/project/scrapy-ctl.py	2009-08-29 04:29:47 -03:00
Pablo Hoffman	c87216f02b	doc: added missing :synopsis: to some modules	2009-08-29 03:46:58 -03:00
Pablo Hoffman	2f55d7932b	replaced :ref: by :doc: links in doc index	2009-08-29 03:37:59 -03:00
Pablo Hoffman	924219dd5a	- added reference documentation about scrapy-ctl.py script ... - yet another refactor to cmdline module - removed help command (use -h to help now) --HG-- rename : docs/experimental/scripts.rst => docs/topics/scrapy-ctl.rst	2009-08-28 20:32:55 -03:00
Pablo Hoffman	609aed4553	added inspect_response() function for inspecting responses from spiders	2009-08-27 19:33:33 -03:00
Pablo Hoffman	ea4f16bae6	refactored scrapy shell implementation, dropping IPython dependency, and adding a new 'view' shortcut	2009-08-27 18:24:08 -03:00
Ismael Carnales	fb39bca24f	added djangoitem doc	2009-08-26 08:44:22 -03:00
Pablo Hoffman	65ab611fb0	more updates to HttpErrorMiddleware doc	2009-08-26 08:30:18 -03:00
Pablo Hoffman	44783a3a06	minor improvements to FAQ entry	2009-08-26 00:18:58 -03:00
Pablo Hoffman	4dba75a73f	updated HttpErrorMiddleware doc	2009-08-25 20:13:58 -03:00
Pablo Hoffman	8a074c9cb5	removed scrapy-admin.py command, and left only scrapy-ctl as the only scrapy command	2009-08-24 15:43:36 -03:00
Ismael Carnales	39540b188a	changed torrent in overview doc	2009-08-24 15:11:04 -03:00
Ismael Carnales	4b5aa30867	minor update to tutorial	2009-08-24 14:34:17 -03:00
Pablo Hoffman	0363040884	doc: added FAQ entry about Accept-Language	2009-08-24 13:56:44 -03:00
Ismael Carnales	85282a4b76	added scrapy commandline scripts doc	2009-08-24 12:02:44 -03:00
Pablo Hoffman	ead8cb9fac	removed documentation about ugly DontCloseDomain exception (which will be removed in the future)	2009-08-24 11:57:21 -03:00
Pablo Hoffman	74f706b356	renamed "parse_item" method of XMLFeedSpider to "parse_node", keeping backwards compatibility	2009-08-24 10:54:34 -03:00
Pablo Hoffman	31693eb90f	dropped "cache" attribute of Request and Response objects	2009-08-24 10:34:05 -03:00
Pablo Hoffman	0186c6937a	HTTP auth middleware: added doc and unittest	2009-08-24 08:07:20 -03:00
Pablo Hoffman	46aaea9f32	doc: improved documentation about debugging leaks	2009-08-23 20:36:00 -03:00
Pablo Hoffman	0ed849248f	some improvements to item exporters ... - passed previous class attributes to instances attributes - better handling of constructor arguments - better coverage on unittets (including encoding) - updated documentation with new changes	2009-08-23 05:48:35 -03:00
Pablo Hoffman	7461710ef1	added some missing dots	2009-08-21 21:54:10 -03:00
Pablo Hoffman	9635a7839c	rearranged documentation into a better organization ... --HG-- rename : docs/topics/index.rst => docs/index.rst	2009-08-21 21:49:54 -03:00
Pablo Hoffman	4761e0c88d	minor doc correction	2009-08-21 19:11:59 -03:00
Pablo Hoffman	012fd1cc2f	moved api-stability.rst doc to root and updated it ... --HG-- rename : docs/misc/api-stability.rst => docs/api-stability.rst	2009-08-21 16:29:23 -03:00
Pablo Hoffman	ef6f04eb06	moved doc about debugging memory leaks to its own topic and added doc about trackref module	2009-08-21 16:07:16 -03:00
Pablo Hoffman	4980eb4968	added titles to signals doc	2009-08-21 15:07:52 -03:00
Pablo Hoffman	daea3f8a0c	sphinx docs: replaced custom :exception: xref by standard :exc:	2009-08-21 15:05:06 -03:00
Ismael Carnales	c08d3aa9cc	updated tutorial to use new items api	2009-08-21 14:16:27 -03:00
Pablo Hoffman	37760eb13f	improved consistency of logging settings to use LOG_*	2009-08-21 08:54:12 -03:00
Ismael Carnales	f0689a9ee8	fixed error in link extractors doc, thanks tarasm	2009-08-21 08:34:53 -03:00
Pablo Hoffman	41f1972e47	removed unused TRACE log level and improved logging documentation	2009-08-20 18:17:48 -03:00
Pablo Hoffman	20e82335e2	rename some exporter methods and complete exporter tests refactoring	2009-08-20 14:09:24 -03:00
Pablo Hoffman	3152e66fec	deprecate domain_open signal and handle stats domain open/close directly from the engine	2009-08-20 10:54:58 -03:00
Pablo Hoffman	33b53c59d5	moved scrapy.xpath to scrapy.selector ... --HG-- rename : scrapy/xpath/__init__.py => scrapy/selector/__init__.py rename : scrapy/xpath/document.py => scrapy/selector/document.py rename : scrapy/xpath/factories.py => scrapy/selector/factories.py	2009-08-19 21:50:52 -03:00
Pablo Hoffman	dcc90fc196	declared loaders api stable and updated example project to use them ... --HG-- rename : docs/experimental/loaders.rst => docs/topics/loaders.rst	2009-08-19 21:39:59 -03:00
Pablo Hoffman	e8504a054c	moved scrapy.newitem to scrapy.item and declared newitem api officially stable. updated docs and example project. deprecated old ScrapedItem	2009-08-19 21:39:58 -03:00
Ismael Carnales	314e8dea43	added new item exporter tests, introduced some api changes	2009-08-19 19:05:33 -03:00
Pablo Hoffman	907fad6da8	make sure input processors always receive iterables as input	2009-08-19 16:49:49 -03:00
Pablo Hoffman	741a6d7863	item exporters refactoring ... --HG-- rename : scrapy/contrib/exporter/jsonexporter.py => scrapy/contrib/exporter/jsonlines.py	2009-08-19 13:09:39 -03:00
Pablo Hoffman	82e4b6adcf	merge with ismael repo	2009-08-18 15:38:20 -03:00
Pablo Hoffman	e01d31c498	another improvement to doc navbar	2009-08-18 15:35:53 -03:00
Ismael Carnales	be4226cec5	merge	2009-08-18 15:21:39 -03:00
Ismael Carnales	f88fb27851	fixed error in xpath selectors doc	2009-08-18 15:18:49 -03:00
Ismael Carnales	67c0c6a9e4	corrected indentation in xpath selectors doc	2009-08-18 15:13:23 -03:00
Pablo Hoffman	ff837e5a45	doc: improved top navbar	2009-08-18 15:12:44 -03:00
Ismael Carnales	428dfe0d4a	corrected the style of spiders documentation	2009-08-18 15:06:33 -03:00
Pablo Hoffman	0192282d07	reorganized doc and moved robotstxt doc inside downloader middlewares doc	2009-08-18 14:36:18 -03:00
Ismael Carnales	33089d287d	merged topics and reference doc	2009-08-18 14:05:15 -03:00
Pablo Hoffman	7f30461410	added support for defining EXTENSIONS setting using dicts, like middleware settings ... --HG-- rename : scrapy/tests/test_utils_middleware.py => scrapy/tests/test_utils_conf.py rename : scrapy/utils/middleware.py => scrapy/utils/conf.py	2009-08-18 11:05:36 -03:00
Ismael Carnales	1bbe7991dc	added documentation for ImagesPipeline	2009-08-18 09:35:32 -03:00
Ismael Carnales	48b40bd620	renamed x method of selectors to select	2009-08-17 15:58:06 -03:00
Pablo Hoffman	59e0a83ad4	removed more obsolete adaptors code	2009-08-17 14:48:11 -03:00
Ismael Carnales	8c2f62ba9c	added Item Exporters documentation	2009-08-14 09:16:29 -03:00
Pablo Hoffman	02b01c3d2b	loaders doc: fixed outdated line	2009-08-13 23:24:24 -03:00
Daniel Grana	6a74b51371	remove dupes words in loaders doc, and unused import in example ... --HG-- extra : rebase_source : ea7886725af7a54bd1031cb28f92efbdfe921d9e	2009-08-13 23:23:08 -03:00
Ismael Carnales	ecca7f6478	fixes to Item Loader doc	2009-08-13 13:30:50 -03:00
Pablo Hoffman	8131d674d4	fixed outdated documentation (refs #97)	2009-08-13 09:24:41 -03:00
Pablo Hoffman	5aeab5b291	converted scrapy.item package to module ... --HG-- rename : scrapy/item/models.py => scrapy/item.py	2009-08-12 21:31:50 -03:00
Pablo Hoffman	991afdf6b3	some minor fixes to loaders doc	2009-08-12 19:23:04 -03:00
Pablo Hoffman	e6dd4d0955	renamed ApplyConcat processor to MapCompose	2009-08-12 18:43:08 -03:00
Pablo Hoffman	4e74f32401	renamed Pipe processor to Compose and documented it	2009-08-12 18:09:36 -03:00
Pablo Hoffman	958c3d9940	fixed some links to item loaders doc	2009-08-12 17:42:01 -03:00
Pablo Hoffman	d94b633565	renamed ItemLoader method populate_item() to load_item()	2009-08-12 17:40:27 -03:00
Pablo Hoffman	1dc592882b	Moved Item Loader to its final location in scrapy.contrib.loader, and updated doc/tests ... --HG-- rename : docs/experimental/itemparser.rst => docs/experimental/loaders.rst rename : scrapy/contrib/itemparser/__init__.py => scrapy/contrib/loader/__init__.py rename : scrapy/contrib/itemparser/common.py => scrapy/contrib/loader/common.py rename : scrapy/contrib/itemparser/parsers.py => scrapy/contrib/loader/processor.py rename : scrapy/tests/test_itemparser.py => scrapy/tests/test_contrib_loader.py	2009-08-12 16:49:07 -03:00
Pablo Hoffman	7cbbc3ffb0	Renamed Loader to ItemParser (SEP-8 proposal). Documentation and unittests also updated. ... --HG-- rename : docs/experimental/loaders.rst => docs/experimental/itemparser.rst rename : scrapy/newitem/loader/__init__.py => scrapy/contrib/itemparser/__init__.py rename : scrapy/newitem/loader/expanders.py => scrapy/contrib/itemparser/parsers.py rename : scrapy/tests/test_itemloader.py => scrapy/tests/test_itemparser.py	2009-08-12 16:49:05 -03:00
Pablo Hoffman	d4aa72d2bb	removed obsolete documentation about Robust Scraped Item and Adaptors	2009-08-12 13:50:15 -03:00
Pablo Hoffman	e69311ef38	added missing text to new stats collector methods	2009-08-11 16:37:52 -03:00
Daniel Grana	45cbe1333f	fix typo in stats docs	2009-08-11 16:30:04 -03:00
Daniel Grana	8f16e7f9fa	remove `default` parameter from max_value/min_value stats methods, update docs, and add new methods to dummy collector	2009-08-11 16:23:23 -03:00
Daniel Grana	cd8eed77c5	stats collector gains two new methods to store values only if greater/lower than previous value	2009-08-11 15:46:44 -03:00
Ismael Carnales	3614998053	fixed error in doc	2009-08-11 09:23:45 -03:00
Pablo Hoffman	7ac7597950	removed obsolete scrapy.contrib.cluster	2009-08-10 21:02:46 -03:00
Pablo Hoffman	c0532712f9	removed unnecesary response ResponseSoup extension, and replaced by a utils function with cache support	2009-08-10 21:02:24 -03:00
Pablo Hoffman	d334c035c0	removed unnecesary ResponseLibxml2 extension and moved libxml2 document caching functionality to Libxml2Document using weak references	2009-08-10 20:52:43 -03:00
Pablo Hoffman	49bab4777b	XPathSelector: added 're' argument to add_xpath method, exposed selector attribute	2009-08-10 19:42:20 -03:00
Pablo Hoffman	50fcc67278	improved reducers examples	2009-08-10 10:13:30 -03:00
Pablo Hoffman	5be0cee577	added TreeExpander example	2009-08-09 20:54:22 -03:00
Pablo Hoffman	38f82e3993	loaders doc: added information about expanders/reducers declaration precendece, and other minor improvements	2009-08-09 18:06:12 -03:00
Pablo Hoffman	b296d4169e	minor doc update for making it more windows-friendly	2009-08-09 17:08:42 -03:00
Pablo Hoffman	91eea82eef	added XPathLoader for working with XPath Selectors more conveniently	2009-08-08 07:26:20 -03:00
Pablo Hoffman	db90e26a8b	renamed ItemLoader class to Loader	2009-08-07 14:39:30 -03:00
Pablo Hoffman	e585c6cac4	relocated experimental newitems/loaders doc, and added example for extending fields metadata ... --HG-- rename : docs/experimental/newitem-loader.rst => docs/experimental/loaders.rst rename : docs/experimental/newitem.rst => docs/experimental/newitems.rst	2009-08-07 14:28:58 -03:00
Pablo Hoffman	d95e99f585	Added documentation for Items and Loaders, removed obsolete Item Adaptors documentation ... --HG-- rename : docs/experimental/topics/newitem/index.rst => docs/experimental/newitem.rst	2009-08-07 03:50:09 -03:00
Ismael Carnales	32894643a0	added ListField documentation, ordered field reference alphabetically	2009-08-03 15:00:04 -03:00
Pablo Hoffman	c3427e075c	added domain_stats parameter to stats_domain_closed signal	2009-07-31 16:36:35 -03:00
Pablo Hoffman	73172b244d	added from_unicode_list() method to Field objects	2009-07-30 16:58:24 -03:00
Pablo Hoffman	8ec7c9e01c	WEBCONSOLE_PORT setting now defaults to 6080	2009-07-29 18:59:34 -03:00
Pablo Hoffman	fcc91901eb	finished cleaning up closedomain documentation, and updated default settings	2009-07-27 15:42:35 -03:00
Pablo Hoffman	09ba6927d7	Some changes to CloseDomain extension: ... - added support for closing by item passed count (CLOSEDOMAIN_ITEMPASSED) - removed support for sending notification emails (since that's the job of another extension)	2009-07-27 15:23:50 -03:00
Pablo Hoffman	9da66698f3	moved httprepr() method (from Request and Response objects) to scrapy.utils functions	2009-07-25 18:56:12 -03:00
Pablo Hoffman	c615ace6bd	doc: updated google directory links in firebug guide	2009-07-24 13:14:36 -03:00
Daniel Grana	96c3cdbec2	improve OffsiteMiddleware reference docs ... --HG-- extra : rebase_source : 3ed3f23fc1ec63b521ead029c5749898f3ab05d7	2009-07-24 12:59:38 -03:00
Pablo Hoffman	38c3f7d0b4	Some changes to logging of scraped items: ... 1. "Scraped Item" log level changed to DEBUG 2. "Dropped Item" log level changed to WARNING 3. added "Passed Item" log message with INFO level	2009-07-23 11:49:48 -03:00
Pablo Hoffman	e43e28bf1d	minimal doc improvement	2009-07-23 09:12:49 -03:00
Ismael Carnales	6d24ae5920	added reference to working with relative xpaths in the tutorial	2009-07-23 09:05:14 -03:00
Pablo Hoffman	9baa6bb2a8	minor selectors doc fix	2009-07-23 01:56:56 -03:00
Ismael Carnales	3c4afb23be	moved newitem from scrapy.contrib_exp to scrapy.newitem	2009-07-22 15:13:36 -03:00
Ismael Carnales	202894dd8f	fixes to newitem doc	2009-07-22 10:25:22 -03:00
Pablo Hoffman	7d8ba0542c	fixed typo in doc	2009-07-21 17:38:46 -03:00
Pablo Hoffman	aa345e116a	Added spider middleware documentation	2009-07-21 17:19:19 -03:00
Ismael Carnales	90f1d9e489	Added Scheduler middleware reference documentation	2009-07-21 16:52:27 -03:00
Ismael Carnales	00d9ce6608	fixes to newitem doc	2009-07-21 12:22:49 -03:00
Ismael Carnales	894a5d81d3	changed the newitem API to a dict-like interface	2009-07-21 12:20:49 -03:00
Pablo Hoffman	9867e7b2d6	removed references to obsolete ENABLED_SPIDERS_FILE setting	2009-07-21 01:11:48 -03:00
Pablo Hoffman	92b746d866	SimpledbStatsCollector: moved domain creation to constructor	2009-07-17 12:49:53 -03:00
Pablo Hoffman	292757f312	added link to architecture overview and fixed old link	2009-07-16 19:15:19 -03:00
Pablo Hoffman	136014d718	Added section about relative xpaths to XPathSelectors doc	2009-07-16 17:29:29 -03:00
Pablo Hoffman	0fd603e27e	Automated merge with http://hg.scrapy.org/users/ismael/scrapy-newitem/	2009-07-16 11:16:46 -03:00
Pablo Hoffman	c2f12327d6	added dumping of global scrapy stats at engine shutdown	2009-07-16 10:48:10 -03:00
Ismael Carnales	a9069e8eaa	reorganized experimental doc in topics and ref, slitted long documents, introduced minor changes	2009-07-16 09:58:44 -03:00
Pablo Hoffman	1125825996	doc: minor updates to tutorial	2009-07-15 22:10:00 -03:00
Pablo Hoffman	5582305648	renamed old STATS_DEBUG setting to STATS_DUMP	2009-07-15 21:02:51 -03:00
Pablo Hoffman	7f281cf295	some minor updates to doc	2009-07-15 01:40:44 -03:00
Pablo Hoffman	8480f49cf6	Complete stats refactoring and documentation ... --HG-- rename : scrapy/stats/statscollector.py => scrapy/stats/collector/__init__.py	2009-07-15 00:30:40 -03:00