diff --git a/docs/faq.rst b/docs/faq.rst index 71d9e4c4e..b92d173dd 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -154,7 +154,7 @@ For more info see: What is the recommended way to deploy a Scrapy crawler in production? --------------------------------------------------------------------- -See :ref:`topics-scrapyd`. +See :ref:`topics-deploy`. Can I use JSON for large exports? --------------------------------- diff --git a/docs/index.rst b/docs/index.rst index 4097aba31..b442df4ba 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -150,7 +150,7 @@ Solving specific problems topics/leaks topics/images topics/ubuntu - topics/deployment + topics/deploy topics/autothrottle topics/benchmarking topics/jobs @@ -186,8 +186,8 @@ Solving specific problems :doc:`topics/ubuntu` Install latest Scrapy packages easily on Ubuntu -:doc:`topics/deployment` - Deploying your Scrapy project in production. +:doc:`topics/deploy` + Deploying your Scrapy spiders and run them in a remote server. :doc:`topics/autothrottle` Adjust crawl rate dynamically based on load. diff --git a/docs/topics/deploy.rst b/docs/topics/deploy.rst index 8f8c55c2f..f57069921 100644 --- a/docs/topics/deploy.rst +++ b/docs/topics/deploy.rst @@ -1,29 +1,72 @@ -.. _topics-deployment: +.. _topics-deploy: -========== -Deployment -========== +================= +Deploying Spiders +================= -The recommended way to deploy Scrapy projects to a server is through `Scrapyd`_. +This section describes the different options you have for deploying your Scrapy +spiders to run them on a regular basis. Running Scrapy spiders in your local +machine is very convenient for the (early) development stage, but not so much +when you need to execute long-running spiders or move spiders to run in +production continously. This is where the solutions for deploying Scrapy +spiders come in. -.. _Scrapyd: https://github.com/scrapy/scrapyd +The most popular choices, for deploying Scrapy spiders, are: + +* :ref:`Scrapy Cloud ` (open source, easier to setup) +* :ref:`Scrapyd ` (open source, harder to setup) + +.. _deploy-scrapy-cloud: + +Deploying to Scrapy Cloud +========================= + +`Scrapy Cloud`_ is a hosted, cloud-based service by `Scrapinghub`_, the company +behind Scrapy. + +Advantages: + +- easy to setup (no need to setup or manage servers) +- well-designed UI to manage spiders and review scraped items, logs and stats +- cheap pricing (cheaper than renting a server, for small workloads) + +Disadvantages: + +- it's not open source + +To deploy spiders to Scrapy Cloud you can use the `shub`_ command line tool. +Please refer to the `Scrapy Cloud documentation`_ for more information. + +The configuration is read from the ``scrapy.cfg`` file just like +``scrapyd-deploy``. + +.. _deploy-scrapyd: Deploying to a Scrapyd Server ============================= -You can deploy to a Scrapyd server using the `Scrapyd client `_. You can add targets to your ``scrapy.cfg`` file which can be deployed to using the ``scrapyd-deploy`` command. +`Scrapyd`_ is an open source application to run Scrapy spiders. It is +maintained by some of the Scrapy developers. -The basic syntax is as follows: +Advantages: - scrapyd-deploy -p +- it's open source, so it can be installed and run anywhere -For more information please refer to the `Deploying your project`_ section. +Disadvantages: +- simple UI (no analytics, graphs or rich log/items browsing) +- requires setting up servers, installing and configuring scrapyd on them. An + APT repo with Ubuntu packages is provided by the Scrapyd team + +To deploy spiders to Scrapyd, you can use the scrapyd-deploy tool provided by +the `scrapyd-client`_ package. Please refer to the `scrapyd-deploy +documentation`_ for more information. + +.. _Scrapyd: https://github.com/scrapy/scrapyd .. _Deploying your project: https://scrapyd.readthedocs.org/en/latest/deploy.html - -Deploying to Scrapinghub -======================== - -You can deploy to Scrapinghub using Scrapinghub's command line client, `shub`_. The configuration is read from the ``scrapy.cfg`` file just like ``scrapyd-deploy``. - -.. _shub: https://github.com/scrapinghub/shub +.. _Scrapy Cloud: http://scrapinghub.com/scrapy-cloud/ +.. _scrapyd-client: https://github.com/scrapy/scrapyd-client +.. _shub: http://doc.scrapinghub.com/shub.html +.. _scrapyd-deploy documentation: http://scrapyd.readthedocs.org/en/latest/deploy.html +.. _Scrapy Cloud documentation: http://doc.scrapinghub.com/scrapy-cloud.html +.. _Scrapinghub: http://scrapinghub.com/