2.7 KiB
Scrapy command line tool
Scrapy is controlled through the scrapy command, which we'll refer to as the "Scrapy tool" from now on to differentiate it from Scrapy commands.
The Scrapy tool provides several commands, for different purposes. Each command supports its own particular syntax. In other words, each command supports a different set of arguments and options.
This page doesn't describe each command and its syntax, but instead provides an introduction to how the scrapy tool is used. After you learn the basics, you can get help for each particular command using the scrapy tool itself.
Using the scrapy tool
The first thing you would do with the scrapy tool is to create your Scrapy project:
scrapy startproject myproject
That will create a Scrapy project under the myproject directory.
Next, you go inside the new project directory:
cd myproject
And you're ready to use use the scrapy command to manage and control your project from there. For example, to create a new spider:
scrapy genspider mydomain mydomain.com
See all available commands
To see all available commands type:
scrapy -h
That will print a summary of all available Scrapy commands.
The first line will print the currently active project, if you're inside a Scrapy project.
Example (with an active project):
Scrapy X.X.X - project: myproject Usage ===== ...
Example (with no active project):
Scrapy X.X.X - no active project Usage ===== ...
Get help for a particular command
To get help about a particular command, including its description, usage, and available options type:
scrapy <command> -h
Example:
scrapy crawl -h
Using scrapy tool outside your project
Not all commands must be run from "inside" a Scrapy project. You can, for example, use the fetch command to download a page (using Scrapy built-in downloader) from outside a project. Other commands that can be used outside a project are startproject (obviously) and shell, to launch a :ref:`Scrapy Shell <topics-shell>`.
System Message: ERROR/3 (<stdin>, line 83); backlink
Unknown interpreted text role "ref".Also, keep in mind that some commands may have slightly different behaviours when running them from inside projects. For example, the fetch command will use spider arguments (such as user_agent attribute) if the url being fetched is handled by some specific project spider that happens to define a custom user_agent attribute. This is feature, as the fetch command is meant to download pages as they would be downloaded from the spider.