From 94ead94bf6eb064f62f7b546f0800c5a829d2995 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Thu, 19 Aug 2010 00:04:52 -0300 Subject: [PATCH] Improved documentation of Scrapy command-line tool --HG-- rename : docs/topics/cmdline.rst => docs/topics/commands.rst --- docs/_ext/scrapydocs.py | 5 + docs/index.rst | 9 +- docs/topics/cmdline.rst | 94 ----------- docs/topics/commands.rst | 354 +++++++++++++++++++++++++++++++++++++++ docs/topics/settings.rst | 19 ++- 5 files changed, 375 insertions(+), 106 deletions(-) delete mode 100644 docs/topics/cmdline.rst create mode 100644 docs/topics/commands.rst diff --git a/docs/_ext/scrapydocs.py b/docs/_ext/scrapydocs.py index a7e86f6a5..6f370a5af 100644 --- a/docs/_ext/scrapydocs.py +++ b/docs/_ext/scrapydocs.py @@ -9,3 +9,8 @@ def setup(app): rolename = "signal", indextemplate = "pair: %s; signal", ) + app.add_crossref_type( + directivename = "command", + rolename = "command", + indextemplate = "pair: %s; command", + ) diff --git a/docs/index.rst b/docs/index.rst index 2f6a5929b..e23753bdc 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -50,6 +50,7 @@ Scraping basics .. toctree:: :hidden: + topics/commands topics/items topics/spiders topics/link-extractors @@ -59,6 +60,9 @@ Scraping basics topics/item-pipeline topics/feed-exports +:doc:`topics/commands` + Learn about the command-line tool used to manage your Scrapy project. + :doc:`topics/items` Define the data you want to scrape. @@ -169,15 +173,14 @@ Reference .. toctree:: :hidden: - topics/cmdline topics/request-response topics/settings topics/signals topics/exceptions topics/exporters -:doc:`topics/cmdline` - Understand the command-line tool used to control your Scrapy project. +:doc:`topics/commands` + Learn about the command-line tool and see all :ref:`available commands `. :doc:`topics/request-response` Understand the classes used to represent HTTP requests and responses. diff --git a/docs/topics/cmdline.rst b/docs/topics/cmdline.rst deleted file mode 100644 index affd7df89..000000000 --- a/docs/topics/cmdline.rst +++ /dev/null @@ -1,94 +0,0 @@ -.. _topics-cmdline: - -======================== -Scrapy command line tool -======================== - -Scrapy is controlled through the ``scrapy`` command, which we'll refer to as -the "Scrapy tool" from now on to differentiate it from Scrapy commands. - -The Scrapy tool provides several commands, for different purposes. Each command -supports its own particular syntax. In other words, each command supports a -different set of arguments and options. - -This page doesn't describe each command and its syntax, but instead provides an -introduction to how the ``scrapy`` tool is used. After you learn the basics, -you can get help for each particular command using the ``scrapy`` tool itself. - -Using the ``scrapy`` tool -========================= - -The first thing you would do with the ``scrapy`` tool is to create your Scrapy -project:: - - scrapy startproject myproject - -That will create a Scrapy project under the ``myproject`` directory. - -Next, you go inside the new project directory:: - - cd myproject - -And you're ready to use use the ``scrapy`` command to manage and control your -project from there. For example, to create a new spider:: - - scrapy genspider mydomain mydomain.com - -See all available commands --------------------------- - -To see all available commands type:: - - scrapy -h - -That will print a summary of all available Scrapy commands. - -The first line will print the currently active project, if you're inside a -Scrapy project. - -Example (with an active project):: - - Scrapy X.X.X - project: myproject - - Usage - ===== - - ... - -Example (with no active project):: - - Scrapy X.X.X - no active project - - Usage - ===== - - ... - - -Get help for a particular command ---------------------------------- - -To get help about a particular command, including its description, usage, and -available options type:: - - scrapy -h - -Example:: - - scrapy crawl -h - -Using ``scrapy`` tool outside your project -========================================== - -Not all commands must be run from "inside" a Scrapy project. You can, for -example, use the ``fetch`` command to download a page (using Scrapy built-in -downloader) from outside a project. Other commands that can be used outside a -project are ``startproject`` (obviously) and ``shell``, to launch a -:ref:`Scrapy Shell `. - -Also, keep in mind that some commands may have slightly different behaviours -when running them from inside projects. For example, the fetch command will use -spider arguments (such as ``user_agent`` attribute) if the url being fetched is -handled by some specific project spider that happens to define a custom -``user_agent`` attribute. This is feature, as the ``fetch`` command is meant to -download pages as they would be downloaded from the spider. diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst new file mode 100644 index 000000000..6a5c4191f --- /dev/null +++ b/docs/topics/commands.rst @@ -0,0 +1,354 @@ +.. _topics-commands: + +================= +Command line tool +================= + +Scrapy is controlled through the ``scrapy`` command-line tool, to be referred +here as the "Scrapy tool" to differentiate it from their sub-commands which we +just call "commands", or "Scrapy commands". + +The Scrapy tool provides several commands, for multiple purposes, and each one +accepts a different set of arguments and options. + +Using the ``scrapy`` tool +========================= + +You can start by running the Scrapy tool with no arguments and it will print +some usage help and the available commands:: + + Scrapy X.Y - no active project + + Usage + ===== + + To run a command: + scrapy [options] [args] + + To get help: + scrapy -h + + Available commands + ================== + + [...] + +The first line will print the currently active project, if you're inside a +Scrapy project. In this, it was run from outside a project. If run from inside +a project it would have printed something like this:: + + Scrapy X.Y - project: myproject + + Usage + ===== + + [...] + +Using the ``scrapy`` tool to create projects +============================================ + +The first thing you typically do with the ``scrapy`` tool is create your Scrapy +project:: + + scrapy startproject myproject + +That will create a Scrapy project under the ``myproject`` directory. + +Next, you go inside the new project directory:: + + cd myproject + +And you're ready to use use the ``scrapy`` command to manage and control your +project from there. + +Using the ``scrapy`` tool to control projects +============================================= + +You use the ``scrapy`` tool from inside your projects to control and manage +them. + +For example, to create a new spider:: + + scrapy genspider mydomain mydomain.com + +Some Scrapy commands (like :command:`crawl`) must be run from inside a Scrapy +project. See the :ref:`commands reference ` below for more +information on which commands must be run from inside projects, and which not. + +Also keep in mind that some commands may have slightly different behaviours +when running them from inside projects. For example, the fetch command will use +spider-overridden behaviours (such as custom ``user_agent`` attribute) if the +url being fetched is associated with some specific spider. This is intentional, +as the ``fetch`` command is meant to be used to check how spiders are +downloading pages. + +.. _topics-commands-ref: + +Available tool commands +======================= + +Here's a list of available built-in commands with a description and some usage +examples. Remember you can always get more info about each command by running:: + + scrapy -h + +And you can check all available commands with:: + + scrapy -h + +.. command:: startproject + +startproject +------------ + ++-------------------+----------------------------------------+ +| Syntax: | ``scrapy startproject `` | ++-------------------+----------------------------------------+ +| Requires project: | *no* | ++-------------------+----------------------------------------+ + +Creates a new Scrapy project named ``project_name``, under the ``project_name`` +directory. + +Usage example:: + + $ scrapy startproject myproject + +.. command:: genspider + +genspider +--------- + ++-------------------+--------------------------------------+ +| Syntax: | ``scrapy genspider `` | ++-------------------+--------------------------------------+ +| Requires project: | *yes* | ++-------------------+--------------------------------------+ + +Create a new spider in the current project. + +This is just a convenient shortcut command for creating spiders based on +pre-defined templates, but certainly not the only way to create spiders. You +can just create the spider source code files yourself. + +Usage example:: + + $ scrapy genspider example example.com + Created spider 'example' using template 'crawl' in module: + jobsbot.spiders.example + +.. command:: crawl + +crawl +----- + ++-------------------+-------------------------------+ +| Syntax: | ``scrapy crawl `` | ++-------------------+-------------------------------+ +| Requires project: | *yes* | ++-------------------+-------------------------------+ + +Start crawling a spider. If a URL is passed instead of a spider, it will start +from that URL instead of the spider start urls. + +Usage examples:: + + $ scrapy crawl example.com + [ ... example.com spider starts crawling ... ] + + $ scrapy crawl myspider + [ ... myspider starts crawling ... ] + + $ scrapy crawl http://example.com/some/page.html + [ ... spider that handles example.com starts crawling from that url ... ] + +.. command:: start + +start +----- + ++-------------------+------------------+ +| Syntax: | ``scrapy start`` | ++-------------------+------------------+ +| Requires project: | *yes* | ++-------------------+------------------+ + +Start Scrapy in server mode. + +Usage example:: + + $ scrapy start + [ ... scrapy starts and stays idle waiting for spiders to get scheduled ... ] + +.. command:: list + +list +---- + ++-------------------+-----------------+ +| Syntax: | ``scrapy list`` | ++-------------------+-----------------+ +| Requires project: | *yes* | ++-------------------+-----------------+ + +List all available spiders in the current project. The output is one spider per +line. + +Usage example:: + + $ scrapy list + spider1 + spider2 + +.. command:: fetch + +fetch +----- + ++-------------------+------------------------+ +| Syntax: | ``scrapy fetch `` | ++-------------------+------------------------+ +| Requires project: | *no* | ++-------------------+------------------------+ + +Downloads the given URL using the Scrapy downloader and writes the contents to +standard output. + +The interesting thing about this command is that it fetches the page how the +the spider would download it. For example, if the spider has an ``user_agent`` +attribute which overrides the User Agent, it will use that one. + +So this command can be used to "see" how your spider would fetch certain page. + +If used outside a project, no particular per-spider behaviour would be applied +and it will just use the default Scrapy downloder settings. + +Usage examples:: + + $ scrapy fetch --nolog http://www.example.com/some/page.html + [ ... html content here ... ] + + $ scrapy fetch --nolog --headers http://www.example.com/ + {'Accept-Ranges': ['bytes'], + 'Age': ['1263 '], + 'Connection': ['close '], + 'Content-Length': ['596'], + 'Content-Type': ['text/html; charset=UTF-8'], + 'Date': ['Wed, 18 Aug 2010 23:59:46 GMT'], + 'Etag': ['"573c1-254-48c9c87349680"'], + 'Last-Modified': ['Fri, 30 Jul 2010 15:30:18 GMT'], + 'Server': ['Apache/2.2.3 (CentOS)']} + +.. command:: view + +view +---- + ++-------------------+-----------------------+ +| Syntax: | ``scrapy view `` | ++-------------------+-----------------------+ +| Requires project: | *no* | ++-------------------+-----------------------+ + +Opens the given URL in a browser, as your Scrapy spider would "see" it. +Sometimes spiders see pages differently from regular users, so this can be used +to check what the spider "sees" and confirm it's what you expect. + +Usage example:: + + $ scrapy view http://www.example.com/some/page.html + [ ... browser starts ... ] + +.. command:: shell + +shell +----- + ++-------------------+------------------------+ +| Syntax: | ``scrapy shell [url]`` | ++-------------------+------------------------+ +| Requires project: | *no* | ++-------------------+------------------------+ + +Starts the Scrapy shell for the given URL (if given) or empty if not URL is +given. See :ref:`topics-shell` for more info. + +Usage example:: + + $ scrapy shell http://www.example.com/some/page.html + [ ... scrapy shell starts ... ] + +.. command:: parse + +parse +----- + ++-------------------+----------------------------------+ +| Syntax: | ``scrapy parse [options]`` | ++-------------------+----------------------------------+ +| Requires project: | *yes* | ++-------------------+----------------------------------+ + +Fetches the given URL and parses with the spider that handles it, using the +method passed with the ``--callback`` option, or ``parse`` if not given. + +Supported options: + + * ``--callback`` or ``-c``: spider method to use as callback for parsing the + response + + * ``--noitems``: don't show extracted links + + * ``--nolinks``: don't show scraped items + +Usage example:: + + $ scrapy parse http://www.example.com/ -c parse_item + [ ... scrapy log lines crawling example.com spider ... ] + # Scraped Items - callback: parse ------------------------------------------------------------ + MyItem({'name': u"Example item", + 'category': u'Furniture', + 'length': u'12 cm'} + ) + +.. command:: settings + +settings +-------- + ++-------------------+-------------------------------+ +| Syntax: | ``scrapy settings [options]`` | ++-------------------+-------------------------------+ +| Requires project: | *no* | ++-------------------+-------------------------------+ + +Get the value of a Scrapy setting. + +If used inside a project it'll show the project setting value, otherwise it'll +show the default Scrapy value for that setting. + +Example usage:: + + $ scrapy settings --get BOT_NAME + scrapybot + $ scrapy settings --get DOWNLOAD_DELAY + 0 + +.. command:: runspider + +runspider +--------- + ++-------------------+---------------------------------------+ +| Syntax: | ``scrapy runspider `` | ++-------------------+---------------------------------------+ +| Requires project: | *no* | ++-------------------+---------------------------------------+ + +Run a spider self-contained in a Python file, without having to create a +project. + +Example usage:: + + $ scrapy runspider myspider.py + [ ... spider starts crawling ... ] diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index eb21f2acf..33f8a2f8c 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -24,7 +24,8 @@ Designating the settings When you use Scrapy, you have to tell it which settings you're using. You can do this by using an environment variable, ``SCRAPY_SETTINGS_MODULE``, or the -``--settings`` argument of the :doc:`scrapy command `. +``--settings`` argument of the :doc:`scrapy command-line tool +`. The value of ``SCRAPY_SETTINGS_MODULE`` should be in Python path syntax, e.g. ``myproject.settings``. Note that the settings module should be on the @@ -89,9 +90,10 @@ It's where most of your custom settings will be populated. 4. Default settings per-command ------------------------------- -Each :doc:`/topics/cmdline` command can have its own default settings, which -override the global default settings. Those custom command settings are -specified in the ``default_settings`` attribute of the command class. +Each :doc:`Scrapy tool ` command can have its own default +settings, which override the global default settings. Those custom command +settings are specified in the ``default_settings`` attribute of the command +class. 5. Default global settings -------------------------- @@ -223,8 +225,7 @@ project name). This will be used to construct the User-Agent by default, and also for logging. It's automatically populated with your project name when you create your -project with the :doc:`scrapy ` ``startproject`` -command. +project with the :command:`startproject` command. .. setting:: BOT_VERSION @@ -720,7 +721,7 @@ NEWSPIDER_MODULE Default: ``''`` -Module where to create new spiders using the ``genspider`` command. +Module where to create new spiders using the :command:`genspider` command. Example:: @@ -996,8 +997,8 @@ TEMPLATES_DIR Default: ``templates`` dir inside scrapy module -The directory where to look for template when creating new projects with -:doc:`scrapy startproject ` command. +The directory where to look for templates when creating new projects with +:command:`startproject` command. .. setting:: URLLENGTH_LIMIT