2.5 KiB
Deploying Spiders
This section describes the different options you have for deploying your Scrapy spiders to run them on a regular basis. Running Scrapy spiders in your local machine is very convenient for the (early) development stage, but not so much when you need to execute long-running spiders or move spiders to run in production continously. This is where the solutions for deploying Scrapy spiders come in.
The most popular choices, for deploying Scrapy spiders, are:
:ref:`Scrapy Cloud <deploy-scrapy-cloud>` (cloud-based, easier to setup)
System Message: ERROR/3 (<stdin>, line 16); backlink
Unknown interpreted text role "ref".
:ref:`Scrapyd <deploy-scrapyd>` (open source, harder to setup)
System Message: ERROR/3 (<stdin>, line 17); backlink
Unknown interpreted text role "ref".
Deploying to Scrapy Cloud
Scrapy Cloud is a hosted, cloud-based service by Scrapinghub, the company behind Scrapy.
Advantages:
- easy to setup (no need to setup or manage servers)
- well-designed UI to manage spiders and review scraped items, logs and stats
- cheap pricing (cheaper than renting a server, for small workloads)
Disadvantages:
- it's not open source
To deploy spiders to Scrapy Cloud you can use the shub command line tool. Please refer to the Scrapy Cloud documentation for more information.
The configuration is read from the scrapy.cfg file just like scrapyd-deploy.
Deploying to a Scrapyd Server
Scrapyd is an open source application to run Scrapy spiders. It is maintained by some of the Scrapy developers.
Advantages:
- it's open source, so it can be installed and run anywhere
Disadvantages:
- simple UI (no analytics, graphs or rich log/items browsing)
- requires setting up servers, installing and configuring scrapyd on them. An APT repo with Ubuntu packages is provided by the Scrapyd team
To deploy spiders to Scrapyd, you can use the scrapyd-deploy tool provided by the scrapyd-client package. Please refer to the scrapyd-deploy documentation for more information.