RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/index.rst

6.4 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Scrapy |version| documentation

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Getting help

Having trouble? Read the :doc:`FAQ <faq>`, ask a question in GitHub Discussions or report a bug on our issue tracker. See :doc:`intro/getting-help` for more options.

System Message: ERROR/3 (<stdin>, line 18); backlink

Unknown interpreted text role "doc".

System Message: ERROR/3 (<stdin>, line 18); backlink

Unknown interpreted text role "doc".

First steps

System Message: ERROR/3 (<stdin>, line 29)

Unknown directive type "toctree".

.. toctree::
   :caption: First steps
   :hidden:

   intro/overview
   intro/install
   intro/tutorial
   intro/examples
   intro/getting-help
:doc:`intro/overview`

System Message: ERROR/3 (<stdin>, line 40); backlink

Unknown interpreted text role "doc".

Understand what Scrapy is and how it can help you.

:doc:`intro/install`

System Message: ERROR/3 (<stdin>, line 43); backlink

Unknown interpreted text role "doc".

Get Scrapy installed on your computer.

:doc:`intro/tutorial`

System Message: ERROR/3 (<stdin>, line 46); backlink

Unknown interpreted text role "doc".

Write your first Scrapy project.

:doc:`intro/examples`

System Message: ERROR/3 (<stdin>, line 49); backlink

Unknown interpreted text role "doc".

Learn more by playing with a pre-made Scrapy project.

:doc:`intro/getting-help`

System Message: ERROR/3 (<stdin>, line 52); backlink

Unknown interpreted text role "doc".

Find support beyond this documentation.

Basic concepts

System Message: ERROR/3 (<stdin>, line 59)

Unknown directive type "toctree".

.. toctree::
   :caption: Basic concepts
   :hidden:

   topics/commands
   topics/spiders
   topics/selectors
   topics/items
   topics/loaders
   topics/shell
   topics/item-pipeline
   topics/feed-exports
   topics/request-response
   topics/cookies
   topics/link-extractors
   topics/settings
   topics/exceptions
:doc:`topics/commands`

System Message: ERROR/3 (<stdin>, line 78); backlink

Unknown interpreted text role "doc".

Learn about the command-line tool used to manage your Scrapy project.

:doc:`topics/spiders`

System Message: ERROR/3 (<stdin>, line 81); backlink

Unknown interpreted text role "doc".

Write the rules to crawl your websites.

:doc:`topics/selectors`

System Message: ERROR/3 (<stdin>, line 84); backlink

Unknown interpreted text role "doc".

Extract the data from web pages using XPath.

:doc:`topics/items`

System Message: ERROR/3 (<stdin>, line 87); backlink

Unknown interpreted text role "doc".

Define the data you want to scrape.

:doc:`topics/loaders`

System Message: ERROR/3 (<stdin>, line 90); backlink

Unknown interpreted text role "doc".

Populate your items with the extracted data.

:doc:`topics/shell`

System Message: ERROR/3 (<stdin>, line 93); backlink

Unknown interpreted text role "doc".

Test your extraction code in an interactive environment.

:doc:`topics/item-pipeline`

System Message: ERROR/3 (<stdin>, line 96); backlink

Unknown interpreted text role "doc".

Post-process and store your scraped data.

:doc:`topics/feed-exports`

System Message: ERROR/3 (<stdin>, line 99); backlink

Unknown interpreted text role "doc".

Output your scraped data using different formats and storages.

:doc:`topics/request-response`

System Message: ERROR/3 (<stdin>, line 102); backlink

Unknown interpreted text role "doc".

Understand the classes used to represent HTTP requests and responses.

:doc:`topics/cookies`

System Message: ERROR/3 (<stdin>, line 105); backlink

Unknown interpreted text role "doc".

Send and receive cookies.

:doc:`topics/link-extractors`

System Message: ERROR/3 (<stdin>, line 108); backlink

Unknown interpreted text role "doc".

Convenient classes to extract links to follow from pages.

:doc:`topics/settings`

System Message: ERROR/3 (<stdin>, line 111); backlink

Unknown interpreted text role "doc".

Learn how to configure Scrapy and see all :ref:`available settings <topics-settings-ref>`.

System Message: ERROR/3 (<stdin>, line 111); backlink

Unknown interpreted text role "ref".
:doc:`topics/exceptions`

System Message: ERROR/3 (<stdin>, line 115); backlink

Unknown interpreted text role "doc".

See all available exceptions and their meaning.

Built-in services

System Message: ERROR/3 (<stdin>, line 120)

Unknown directive type "toctree".

.. toctree::
   :caption: Built-in services
   :hidden:

   topics/logging
   topics/stats
   topics/telnetconsole
:doc:`topics/logging`

System Message: ERROR/3 (<stdin>, line 129); backlink

Unknown interpreted text role "doc".

Learn how to use Python's built-in logging on Scrapy.

:doc:`topics/stats`

System Message: ERROR/3 (<stdin>, line 132); backlink

Unknown interpreted text role "doc".

Collect statistics about your scraping crawler.

:doc:`topics/telnetconsole`

System Message: ERROR/3 (<stdin>, line 136); backlink

Unknown interpreted text role "doc".

Inspect a running crawler using a built-in Python console.

Solving specific problems

System Message: ERROR/3 (<stdin>, line 141)

Unknown directive type "toctree".

.. toctree::
   :caption: Solving specific problems
   :hidden:

   faq
   topics/debug
   topics/contracts
   topics/practices
   topics/security
   topics/optimize
   topics/developer-tools
   topics/dynamic-content
   topics/leaks
   topics/media-pipeline
   topics/deploy
   topics/autothrottle
   topics/benchmarking
   topics/jobs
   topics/coroutines
   topics/asyncio
:doc:`faq`

System Message: ERROR/3 (<stdin>, line 163); backlink

Unknown interpreted text role "doc".

Get answers to most frequently asked questions.

:doc:`topics/debug`

System Message: ERROR/3 (<stdin>, line 166); backlink

Unknown interpreted text role "doc".

Learn how to debug common problems of your Scrapy spider.

:doc:`topics/contracts`

System Message: ERROR/3 (<stdin>, line 169); backlink

Unknown interpreted text role "doc".

Learn how to use contracts for testing your spiders.

:doc:`topics/practices`

System Message: ERROR/3 (<stdin>, line 172); backlink

Unknown interpreted text role "doc".

Get familiar with some Scrapy common practices.

:doc:`topics/security`

System Message: ERROR/3 (<stdin>, line 176); backlink

Unknown interpreted text role "doc".

Understand the security implications of Scrapy defaults and how to harden them.

:doc:`topics/optimize`

System Message: ERROR/3 (<stdin>, line 179); backlink

Unknown interpreted text role "doc".

Find the bottleneck of your crawls and learn how to address it.

:doc:`topics/developer-tools`

System Message: ERROR/3 (<stdin>, line 182); backlink

Unknown interpreted text role "doc".

Learn how to scrape with your browser's developer tools.

:doc:`topics/dynamic-content`

System Message: ERROR/3 (<stdin>, line 185); backlink

Unknown interpreted text role "doc".

Read webpage data that is loaded dynamically.

:doc:`topics/leaks`

System Message: ERROR/3 (<stdin>, line 188); backlink

Unknown interpreted text role "doc".

Learn how to find and get rid of memory leaks in your crawler.

:doc:`topics/media-pipeline`

System Message: ERROR/3 (<stdin>, line 191); backlink

Unknown interpreted text role "doc".

Download files and/or images associated with your scraped items.

:doc:`topics/deploy`

System Message: ERROR/3 (<stdin>, line 194); backlink

Unknown interpreted text role "doc".

Deploying your Scrapy spiders and run them in a remote server.

:doc:`topics/autothrottle`

System Message: ERROR/3 (<stdin>, line 197); backlink

Unknown interpreted text role "doc".

Adjust crawl rate dynamically based on load.

:doc:`topics/benchmarking`

System Message: ERROR/3 (<stdin>, line 200); backlink

Unknown interpreted text role "doc".

Check how Scrapy performs on your hardware.

:doc:`topics/jobs`

System Message: ERROR/3 (<stdin>, line 203); backlink

Unknown interpreted text role "doc".

Learn how to pause and resume crawls for large spiders.

:doc:`topics/coroutines`

System Message: ERROR/3 (<stdin>, line 206); backlink

Unknown interpreted text role "doc".

Use the :ref:`coroutine syntax <async>`.

System Message: ERROR/3 (<stdin>, line 206); backlink

Unknown interpreted text role "ref".
:doc:`topics/asyncio`

System Message: ERROR/3 (<stdin>, line 209); backlink

Unknown interpreted text role "doc".

Use :mod:`asyncio` and :mod:`asyncio`-powered libraries.

System Message: ERROR/3 (<stdin>, line 209); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 209); backlink

Unknown interpreted text role "mod".

Extending Scrapy

System Message: ERROR/3 (<stdin>, line 216)

Unknown directive type "toctree".

.. toctree::
   :caption: Extending Scrapy
   :hidden:

   topics/architecture
   topics/addons
   topics/downloader-middleware
   topics/spider-middleware
   topics/extensions
   topics/signals
   topics/scheduler
   topics/exporters
   topics/download-handlers
   topics/components
   topics/api
:doc:`topics/architecture`

System Message: ERROR/3 (<stdin>, line 234); backlink

Unknown interpreted text role "doc".

Understand the Scrapy architecture.

:doc:`topics/addons`

System Message: ERROR/3 (<stdin>, line 237); backlink

Unknown interpreted text role "doc".

Enable and configure third-party extensions.

:doc:`topics/downloader-middleware`

System Message: ERROR/3 (<stdin>, line 240); backlink

Unknown interpreted text role "doc".

Customize how pages get requested and downloaded.

:doc:`topics/spider-middleware`

System Message: ERROR/3 (<stdin>, line 243); backlink

Unknown interpreted text role "doc".

Customize the input and output of your spiders.

:doc:`topics/extensions`

System Message: ERROR/3 (<stdin>, line 246); backlink

Unknown interpreted text role "doc".

Extend Scrapy with your custom functionality

:doc:`topics/signals`

System Message: ERROR/3 (<stdin>, line 249); backlink

Unknown interpreted text role "doc".

See all available signals and how to work with them.

:doc:`topics/scheduler`

System Message: ERROR/3 (<stdin>, line 252); backlink

Unknown interpreted text role "doc".

Understand the scheduler component.

:doc:`topics/exporters`

System Message: ERROR/3 (<stdin>, line 255); backlink

Unknown interpreted text role "doc".

Quickly export your scraped items to a file (XML, CSV, etc).

:doc:`topics/download-handlers`

System Message: ERROR/3 (<stdin>, line 258); backlink

Unknown interpreted text role "doc".

Customize how requests are downloaded or add support for new URL schemes.

:doc:`topics/components`

System Message: ERROR/3 (<stdin>, line 262); backlink

Unknown interpreted text role "doc".

Learn the common API and some good practices when building custom Scrapy components.

:doc:`topics/api`

System Message: ERROR/3 (<stdin>, line 266); backlink

Unknown interpreted text role "doc".

Use it on extensions and middlewares to extend Scrapy functionality.

All the rest

System Message: ERROR/3 (<stdin>, line 271)

Unknown directive type "toctree".

.. toctree::
   :caption: All the rest
   :hidden:

   news
   contributing
   versioning
:doc:`news`

System Message: ERROR/3 (<stdin>, line 280); backlink

Unknown interpreted text role "doc".

See what has changed in recent Scrapy versions.

:doc:`contributing`

System Message: ERROR/3 (<stdin>, line 283); backlink

Unknown interpreted text role "doc".

Learn how to contribute to the Scrapy project.

:doc:`versioning`

System Message: ERROR/3 (<stdin>, line 285); backlink

Unknown interpreted text role "doc".

Understand Scrapy versioning and API stability.

Docutils System Messages

System Message: ERROR/3 (<stdin>, line 5); backlink

Undefined substitution referenced: "version".
</html>