From bd60c3f41fd25e7b5a413cf5c112166b813c2691 Mon Sep 17 00:00:00 2001 From: Shinichi Takayanagi Date: Tue, 11 May 2021 04:58:04 +0900 Subject: [PATCH] More documentation for setting spider atributes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17 * Update feed-exports.rst * Update feed-exports.rst * Reflects the comments * Remove redundant newline * Update docs/topics/feed-exports.rst Co-authored-by: Adrián Chaves * Apply suggestions from code review Co-authored-by: Adrián Chaves Co-authored-by: Adrián Chaves Co-authored-by: Eugenio Lacuesta --- docs/topics/feed-exports.rst | 3 +++ docs/topics/spiders.rst | 8 ++++++++ 2 files changed, 11 insertions(+) diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index e772a461c..26c247cdd 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -135,6 +135,9 @@ Here are some examples to illustrate: - ``s3://mybucket/scraping/feeds/%(name)s/%(time)s.json`` +.. note:: :ref:`Spider arguments ` become spider attributes, hence + they can also be used as storage URI parameters. + .. _topics-feed-storage-backends: diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 2056664c7..a3e9f410f 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -294,6 +294,14 @@ The above example can also be written as follows:: def start_requests(self): yield scrapy.Request(f'http://www.example.com/categories/{self.category}') +If you are :ref:`running Scrapy from a script `, you can +specify spider arguments when calling +:class:`CrawlerProcess.crawl ` or +:class:`CrawlerRunner.crawl `:: + + process = CrawlerProcess() + process.crawl(MySpider, category="electronics") + Keep in mind that spider arguments are only strings. The spider will not do any parsing on its own. If you were to set the ``start_urls`` attribute from the command line,