RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,814 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

57 Commits

Author SHA1 Message Date
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Adrián Chaves 08756cd87a Fix the description of the ScreenshotPipeline example item pipeline 2020-05-22 18:06:54 +02:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
Adrián Chaves 6aa0ba4553
Write release notes for Scrapy 2.0.0 (#4329) 2020-03-03 09:11:11 +01:00
nyov a34c366fa4 DOC linkcheck run; https and 301 link updates. 
Closes #4359
 2020-02-25 16:51:33 +00:00
Adrián Chaves e84cb18ca0 Use InterSphinx to link to the Twisted documentation 2019-11-18 15:50:45 +01:00
Joaquin Garmendia Cabrera e1f8b55ba0
Improve syntax for readability 2018-12-28 16:53:12 -05:00
Joaquin Garmendia Cabrera f85c915872
Update item-pipeline example 2018-12-23 00:26:58 -05:00
Casker ae679f6499 Create item-pipeline.rst 2017-06-09 16:12:20 +08:00
Paul Tremberth fad6b70d92 Use https:// for readthedocs links 2017-06-01 16:41:52 +02:00
yandongxu 4966dd7a7f Fix doc: open file with "wb" mode will get an error in python 3 2017-05-08 18:50:30 +08:00
Bernardas dfba151f59 Remove unnecessary note for the JsonWriterPipeline example 2016-10-05 16:36:02 +00:00
Bernardas eb91cb8ea2 fix JsonWriterPipeline example 2016-10-03 20:31:41 +00:00
pawelmhm 39dbd890b0 [docs/item_pipeline] process_item returning Deferred docs 
* quote url
* use hash of url as filename
 2016-09-24 08:36:09 +02:00
Pawel Miech b2bfd1e5c5 [docs] document that process_item can return Deferred 2016-09-23 10:36:03 +02:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
David Chen 0025d5a943 Fixed minor grammar issues. 2015-11-16 07:30:17 +08:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
bagratte 1312bcd007 minor corrections in documentation. 2015-04-19 18:58:15 +04:00
Mikhail Korobov 817dbc6cbd DOC mention dicts in documentation; explain better what are Items for 2015-03-19 05:16:14 +05:00
Pablo Hoffman 0dce283459 Merge pull request #893 from kmike/less-ads 
[MRG] DOC simplify extension docs
 2014-10-21 17:13:59 -02:00
Mikhail Korobov 6fcf9dce50 DOC document from_crawler method for item pipelines; add an example. 2014-09-25 03:13:51 +06:00
Mikhail Korobov e435b3e3a3 DOC simplify extension docs 2014-09-21 00:19:24 +06:00
Rolando Espinoza 28f946b05f DOC Use pipelines module name instead of pipieline following default project files. 2014-02-15 11:01:26 -04:00
Yuri Prezument 060891c01c Remove unused import from code sample 
Item pipeline docs - removed unused import from code sample
 2014-01-03 15:44:17 +02:00
Mikhail Korobov a27d91f0a6 Rename BaseSpider to Spider. See GH-495. 2013-12-30 19:46:41 +06:00
Brian Lange e4c1d8d37d Elaborate on use of order numbers 2013-11-19 17:51:50 -06:00
Brian Lange b878f60b5a Add note to item-pipeline documentation explaining order in the ITEM_PIPELINES setting. 2013-11-19 16:12:54 -06:00
Pablo Hoffman fc388f4636 Make ITEM_PIPELINE setting a dict 
This is for consistency with how spider and downloader middlewares are
defined. ITEM_PIPELINE_BASE was also added and both remain empty.

Backwards compatibility is kept (with a warning) with list-based
ITEM_PIPELINES.
 2013-09-23 17:50:43 -03:00
Chris Tilden aae6aed4fb fixes spelling errors in documentation 2013-01-22 14:52:18 -08:00
Pablo Hoffman babfc6e79b Updated documentation after singleton removal changes. 
Also removed some unused code and made some minor additional
refactoring.
 2012-08-28 18:35:57 -03:00
Pablo Hoffman b4fbc6c5fa Updated Scrapy Tutorial to reference feed exports, instead a custom written pipeline, and extended item pipeline documentation to include a JSON writer. 2010-10-10 20:31:05 -02:00
Pablo Hoffman 9aefa242d5 Applied documentation patch provided by Lucian Ursu (closes #207) 2010-08-21 01:26:35 -03:00
Pablo Hoffman e741a807d2 Added new Feed exports extension with documentation and storage tests. Closes #197. 
Also deprecated File export pipeline (to be removed in Scrapy 0.11).

Still need to add tests for FeedExport main extension code.
 2010-08-17 14:27:48 -03:00
Pablo Hoffman 43d47e5d9b Some improvements to Item Pipeline (closes #195): 
* Made Item Pipeline Manager a subclass of scrapy.middleware.MiddlewareManager
* Added open_spider/close_spider methods with support for returning deferreds from them
* Inverted the process_item() arguments to be more friendly with deferred
  callbacks (backwards compatibility kept through arguments introspection)
* Updated documentation with new methods and process_item() arguments change
 2010-08-12 10:48:37 -03:00
Pablo Hoffman 784722774b moved scrapy.core.signals to scrapy.signals, keeping backwards compatibility 2010-08-10 17:40:53 -03:00
Pablo Hoffman c359a34d7d moved scrapy.core.exceptions to scrapy.exceptions, keeping backwards compatibility 
--HG--
rename : scrapy/core/exceptions.py => scrapy/exceptions.py
 2010-08-10 17:36:48 -03:00
Pablo Hoffman c7d9f6e270 Added JSON item exporter with doc and unittests (closes #192), and also: 
* put all json exporters in scrapy.contrib.exporters and deprecated
  scrapy.contrib.exporters.jsonlines to reduce module nesting
* use JSON exporter with EXPORT_FORMAT=json in file export pipeline
 2010-08-07 15:52:59 -03:00
Pablo Hoffman de32612c99 Automated merge with http://hg.scrapy.org/scrapy-0.8 2010-04-02 02:49:51 -03:00
Pablo Hoffman dfdac356af added missing default values to file xporter doc 2010-04-02 02:49:18 -03:00
Rolando Espinoza La fuente 7235040936 merged upstream 2010-02-19 17:41:45 -04:00
Pablo Hoffman 67858af83c fixed doc typo 2010-01-18 18:16:58 -02:00
Rolando Espinoza La fuente 1402da31c5 docs: fixed typos and updated code examples 2010-01-11 12:28:22 -04:00
Ismael Carnales 93cc3d2715 Correct param formatting in item pipelines doc 2009-11-30 11:04:15 -02:00
Pablo Hoffman 7728a23e99 Changed item pipeline API to pass spider references (instead of domain names) to process_item() method 2009-11-06 13:46:36 -02:00
Pablo Hoffman 97c322707a * Renamed domain_{opened,closed,idle} signals to spider_{opened,closed,idle} 
* Changed them to pass spider instances only (no domains) (refs #105)
 2009-11-03 00:39:02 -02:00
Ismael Carnales fd41f06056 added doc on how to enable an Item Pipeline component 2009-09-16 14:19:16 -03:00
Daniel Grana 062730cbd8 fix csv exporter documentation 2009-09-16 00:17:50 -03:00
Pablo Hoffman ac8f46ce9e added File Export Pipeline reference to Exporters doc 2009-08-31 21:01:35 -03:00