From e892a484e823470cf37e66904ffbfdb240438842 Mon Sep 17 00:00:00 2001 From: "Deschner, Magdalena" Date: Tue, 16 Jul 2019 13:53:56 +0200 Subject: [PATCH 1/3] add instructions about how to define output file when running scrapy from script instead of cmd --- docs/topics/practices.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 298a078a7..4566eeb6e 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -35,12 +35,17 @@ Here's an example showing how to run a single spider with it. ... process = CrawlerProcess({ - 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)' + 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)', + 'FEED_FORMAT':'json', + 'FEED_URI':'items.json' }) process.crawl(MySpider) process.start() # the script will block here until the crawling is finished +Define settings within dictionary in +CrawlerProcess. FEED_FORMAT and FEED_URI are the equivalent to "-o items.json" when using the scrapy crawl shell command. + Make sure to check :class:`~scrapy.crawler.CrawlerProcess` documentation to get acquainted with its usage details. From 6660020ebb01c6eb240f56347d67106f308dd333 Mon Sep 17 00:00:00 2001 From: "Deschner, Magdalena" Date: Wed, 17 Jul 2019 11:30:02 +0200 Subject: [PATCH 2/3] remove detailed description about individual settings --- docs/topics/practices.rst | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 4566eeb6e..610a39ea9 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -34,8 +34,7 @@ Here's an example showing how to run a single spider with it. # Your spider definition ... - process = CrawlerProcess({ - 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)', + process = CrawlerProcess(settings={ 'FEED_FORMAT':'json', 'FEED_URI':'items.json' }) @@ -43,11 +42,8 @@ Here's an example showing how to run a single spider with it. process.crawl(MySpider) process.start() # the script will block here until the crawling is finished -Define settings within dictionary in -CrawlerProcess. FEED_FORMAT and FEED_URI are the equivalent to "-o items.json" when using the scrapy crawl shell command. - -Make sure to check :class:`~scrapy.crawler.CrawlerProcess` documentation to get -acquainted with its usage details. +Define settings within dictionary in CrawlerProcess. Make sure to check :class:`~scrapy.crawler.CrawlerProcess` +documentation to get acquainted with its usage details. If you are inside a Scrapy project there are some additional helpers you can use to import those components within the project. You can automatically import From c44d49b238f1c6cfc07ffa2fbb65b267e19e381c Mon Sep 17 00:00:00 2001 From: "Deschner, Magdalena" Date: Wed, 17 Jul 2019 13:13:52 +0200 Subject: [PATCH 3/3] minor PEP8 style changes --- docs/topics/practices.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 610a39ea9..a6d4f0d6d 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -35,8 +35,8 @@ Here's an example showing how to run a single spider with it. ... process = CrawlerProcess(settings={ - 'FEED_FORMAT':'json', - 'FEED_URI':'items.json' + 'FEED_FORMAT': 'json', + 'FEED_URI': 'items.json' }) process.crawl(MySpider)