diff --git a/docs/experimental/index.rst b/docs/experimental/index.rst index 32efeec7a..ebe529a34 100644 --- a/docs/experimental/index.rst +++ b/docs/experimental/index.rst @@ -21,3 +21,4 @@ it's properly merged) . Use at your own risk. exporters images + scripts diff --git a/docs/experimental/scripts.rst b/docs/experimental/scripts.rst new file mode 100644 index 000000000..c41c70068 --- /dev/null +++ b/docs/experimental/scripts.rst @@ -0,0 +1,156 @@ +.. _topics-scripts: + +================== +Management scripts +================== + +Scrapy is controlled by two commmandline scripts: + +1. :ref:`topics-scripts-scrapy-admin`: used to create Scrapy projects. +2. :ref:`topics-scripts-scrapy-ctl`: located in every project's root dir, used + to manage each project. + +.. _topics-scripts-scrapy-admin: + +scrapy-admin.py +=============== +Usage: ``scrapy-admin.py `` + +This script should be in your system path. + +Available subcommands +--------------------- + +startproject +~~~~~~~~~~~~ +Usage: ``startproject `` + +Starts a new project with name ``project_name`` + + +.. _topics-scripts-scrapy-ctl: + +scrapy-ctl.py +============= +Usage: ``scrapy-admin.py `` + +This script is located in every project's root folder. + + +Available subcommands +--------------------- + +crawl +~~~~~ +Usage: ``crawl [options] ...`` + +Start crawling a domain or URL + + +``--nopipeline`` +"""""""""""""""" +disable scraped item pipeline + +``--restrict`` +"""""""""""""" +restrict crawling only to the given urls + +``-n, --nofollow`` +"""""""""""""""""" +don't follow links (for use with URLs only) + +``-c, --callback`` +"""""""""""""""""" +use the provided callback for starting to crawl the given url + + +fetch +~~~~~ +Usage: ``fetch `` + +Fetch a URL using the Scrapy downloader and print its content to stdout. You +may want to use --nolog to disable logging. + + +``--headers`` +""""""""""""" +print HTTP headers instead of body + + +genspider +~~~~~~~~~ +Usage: ``genspider [options] `` + + +``--template`` +"""""""""""""" +Default: ``crawl`` + +uses a custom template. + +``--force`` +""""""""""" +if the spider already exists, overwrite it with the template. + +``--list`` +~~~~~~~~~~ +list available templates + +``--dump`` +"""""""""""""" +dump ``--template`` to stdout + + +help +~~~~ +Usage: ``help `` + +Provides extended help for the given command. + + +list +~~~~ +List available spiders. + + +parse +~~~~~ +Usage: ``parse [options] `` + +Parse the given URL (using the spider) and print the results. + + +``--nolinks`` +""""""""""""" +don't show extracted links + +``--noitems`` +""""""""""""" +don't show scraped items + +``--nocolour`` +"""""""""""""" +avoid using pygments to colorize the output + +``-r, --rules`` +""""""""""""""" +try to match and parse the url with the defined rules (if any) + +``-c, --callbacks`` +""""""""""""""""""" +use the provided callback(s) for parsing the url (separated with commas) + + +shell +~~~~~ +Usage: ``shell [options] `` + +Interactive console for scraping the given url. For scraping local files you +can use a URL like ``file://path/to/file.html``. See :ref:`topics-shell` for +usage documentation. + + +start +~~~~~ +Start the Scrapy manager but don't run any spider (idle mode) + diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index bd093ee17..b2e60ba29 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -73,7 +73,7 @@ It's where most of your custom settings will be populated. 4. Default settings per-command ------------------------------- -Each scrapy-ctl command can have its own default settings, which override the +Each :ref:`topics-scripts-scrapy-ctl` command can have its own default settings, which override the global default settings. Those custom command settings are located inside the ``scrapy.conf.commands`` module, or you can specify custom settings to override per-comand inside your project, by writing them in the module referenced by the @@ -704,7 +704,8 @@ PROJECT_NAME Default: ``Not Defined`` The name of the current project. It matches the project module name as created -by ``startproject`` command, and is only defined by project settings file. +by :ref:`scrapy-admin.py startproject ` command, +and is only defined by project settings file. .. setting:: REDIRECT_MAX_TIMES @@ -952,7 +953,7 @@ TEMPLATES_DIR Default: ``templates`` dir inside scrapy module The directory where to look for template when creating new projects with -scrapy-admin.py newproject. +:ref:`scrapy-admin.py startproject ` command. .. setting:: URLLENGTH_LIMIT