RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/index.rst

4.1 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Scrapy documentation

This documentation contains everything you need to know about Scrapy.

Getting help

Having trouble? We'd like to help!

First steps

System Message: ERROR/3 (<stdin>, line 30)

Unknown directive type "toctree".

.. toctree::
   :hidden:

   intro/overview
   intro/install
   intro/tutorial
:ref:`intro-overview`

System Message: ERROR/3 (<stdin>, line 38); backlink

Unknown interpreted text role "ref".

Understand what Scrapy is and how it can help you.

:ref:`intro-install`

System Message: ERROR/3 (<stdin>, line 41); backlink

Unknown interpreted text role "ref".

Get Scrapy installed on your computer.

:ref:`intro-tutorial`

System Message: ERROR/3 (<stdin>, line 45); backlink

Unknown interpreted text role "ref".

Write your first Scrapy project.

Scraping basics

System Message: ERROR/3 (<stdin>, line 50)

Unknown directive type "toctree".

.. toctree::
   :hidden:

   topics/items
   topics/spiders
   topics/link-extractors
   topics/selectors
   topics/loaders
   topics/shell
   topics/item-pipeline
:ref:`topics-items`

System Message: ERROR/3 (<stdin>, line 62); backlink

Unknown interpreted text role "ref".

Define the data you want to scrape.

:ref:`topics-spiders`

System Message: ERROR/3 (<stdin>, line 65); backlink

Unknown interpreted text role "ref".

Write the rules to crawl your websites.

:ref:`topics-selectors`

System Message: ERROR/3 (<stdin>, line 68); backlink

Unknown interpreted text role "ref".

Extract the data from web pages.

:ref:`topics-loaders`

System Message: ERROR/3 (<stdin>, line 71); backlink

Unknown interpreted text role "ref".

Populate your items with the extracted data.

:ref:`topics-shell`

System Message: ERROR/3 (<stdin>, line 74); backlink

Unknown interpreted text role "ref".

Test your extracted data in an interactive environment.

:ref:`topics-item-pipeline`

System Message: ERROR/3 (<stdin>, line 78); backlink

Unknown interpreted text role "ref".

Post-process and store your scraped data.

Built-in services

System Message: ERROR/3 (<stdin>, line 83)

Unknown directive type "toctree".

.. toctree::
   :hidden:

   topics/logging
   topics/stats
   topics/email
   topics/telnetconsole
   topics/webconsole
:ref:`topics-logging`

System Message: ERROR/3 (<stdin>, line 93); backlink

Unknown interpreted text role "ref".

Understand the simple logging facility provided by Scrapy.

:ref:`topics-stats`

System Message: ERROR/3 (<stdin>, line 96); backlink

Unknown interpreted text role "ref".

Collect statistics about your scraping crawler.

:ref:`topics-email`

System Message: ERROR/3 (<stdin>, line 99); backlink

Unknown interpreted text role "ref".

Send email notifications when certain events occur.

:ref:`topics-telnetconsole`

System Message: ERROR/3 (<stdin>, line 102); backlink

Unknown interpreted text role "ref".

Inspect a running crawler using a built-in Python console.

:ref:`topics-webconsole`

System Message: ERROR/3 (<stdin>, line 106); backlink

Unknown interpreted text role "ref".

Monitor and control a crawler using a web interface.

Solving specific problems

System Message: ERROR/3 (<stdin>, line 111)

Unknown directive type "toctree".

.. toctree::
   :hidden:

   faq
   topics/firefox
   topics/firebug
   topics/leaks
:ref:`faq`

System Message: ERROR/3 (<stdin>, line 120); backlink

Unknown interpreted text role "ref".

Get answers to most frequently asked questions.

:ref:`topics-firefox`

System Message: ERROR/3 (<stdin>, line 123); backlink

Unknown interpreted text role "ref".

Learn how to scrape with Firefox and some useful add-ons.

:ref:`topics-firebug`

System Message: ERROR/3 (<stdin>, line 126); backlink

Unknown interpreted text role "ref".

Learn how to scrape efficiently using Firebug.

:ref:`topics-leaks`

System Message: ERROR/3 (<stdin>, line 130); backlink

Unknown interpreted text role "ref".

Learn how to find and get rid of memory leaks in your crawler.

Extending Scrapy

System Message: ERROR/3 (<stdin>, line 135)

Unknown directive type "toctree".

.. toctree::
   :hidden:

   topics/architecture
   topics/downloader-middleware
   topics/spider-middleware
   topics/scheduler-middleware
   topics/extensions
:ref:`topics-architecture`

System Message: ERROR/3 (<stdin>, line 145); backlink

Unknown interpreted text role "ref".

Understand the Scrapy architecture.

:ref:`topics-downloader-middleware`

System Message: ERROR/3 (<stdin>, line 148); backlink

Unknown interpreted text role "ref".

Customize how pages get requested and downloaded.

:ref:`topics-spider-middleware`

System Message: ERROR/3 (<stdin>, line 151); backlink

Unknown interpreted text role "ref".

Customize the input and output of your spiders.

:ref:`topics-scheduler-middleware`

System Message: ERROR/3 (<stdin>, line 154); backlink

Unknown interpreted text role "ref".

Customize how pages are scheduled.

:ref:`topics-extensions`

System Message: ERROR/3 (<stdin>, line 159); backlink

Unknown interpreted text role "ref".

Add any custom functionality using :ref:`signals <topics-signals>` and the Scrapy API

System Message: ERROR/3 (<stdin>, line 157); backlink

Unknown interpreted text role "ref".

Reference

System Message: ERROR/3 (<stdin>, line 164)

Unknown directive type "toctree".

.. toctree::
   :hidden:

   topics/request-response
   topics/settings
   topics/signals
   topics/exceptions
:ref:`topics-request-response`

System Message: ERROR/3 (<stdin>, line 173); backlink

Unknown interpreted text role "ref".

Understand the classes used to represent HTTP requests and responses.

:ref:`topics-settings`

System Message: ERROR/3 (<stdin>, line 176); backlink

Unknown interpreted text role "ref".

Learn how to configure Scrapy and see all :ref:`available settings <topics-settings-ref>`.

System Message: ERROR/3 (<stdin>, line 176); backlink

Unknown interpreted text role "ref".
:ref:`topics-signals`

System Message: ERROR/3 (<stdin>, line 179); backlink

Unknown interpreted text role "ref".

See all available signals and how to work with them.

:ref:`topics-exceptions`

System Message: ERROR/3 (<stdin>, line 183); backlink

Unknown interpreted text role "ref".

See all available exceptions and their meaning.

All the rest

System Message: ERROR/3 (<stdin>, line 188)

Unknown directive type "toctree".

.. toctree::
   :hidden:

   api-stability
   experimental/index
:ref:`api-stability`

System Message: ERROR/3 (<stdin>, line 195); backlink

Unknown interpreted text role "ref".

Understand Scrapy versioning and API stability.

:ref:`experimental`

System Message: ERROR/3 (<stdin>, line 197); backlink

Unknown interpreted text role "ref".

Learn about bleeding-edge features.

</html>