.. _topics-spiders:

=======
Spiders
=======

Spiders are classes that define how a site, or a group of sites, is scraped:
which requests to send, and how to parse their responses to extract data and to
send additional requests.

A crawl goes as follows:

1.  Scrapy iterates the :meth:`~scrapy.Spider.start` method of the spider to
    get the initial requests. By default, that method yields a
    :class:`~scrapy.Request` object for each URL in
    :attr:`~scrapy.Spider.start_urls`, with :meth:`~scrapy.Spider.parse` as
    :ref:`callback <callbacks>`.

2.  Scrapy downloads each request and calls its callback with the resulting
    :class:`~scrapy.http.Response`.

3.  Callbacks parse the response, typically using :ref:`topics-selectors`, and
    return or yield :ref:`item objects <topics-items>` with the extracted data
    and :class:`~scrapy.Request` objects to continue the crawl, which go back
    to step 2. See :ref:`callback-output`.

4.  Items go through :ref:`item pipelines <topics-item-pipeline>`, and are
    usually stored through :ref:`topics-feed-exports`.

Scrapy includes different spider classes for different purposes, described
below.

.. _topics-spiders-ref:

scrapy.Spider
=============

.. class:: scrapy.spiders.Spider
.. autoclass:: scrapy.Spider

    .. autoattribute:: name

    .. attribute:: allowed_domains
        :type: list[str]

        The domains that this spider is allowed to crawl, if any. Requests for
        URLs not belonging to the domain names specified in this list (or their
        subdomains) won't be followed if
        :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` is
        enabled.

        .. versionchanged:: VERSION
           Changes to this attribute during a crawl are now taken into account.

        Let's say your target url is ``https://www.example.com/1.html``,
        then add ``'example.com'`` to the list.

        You may modify this attribute while the spider runs, e.g. to allow
        domains that you only learn about from an earlier response. The change
        affects requests scheduled after it.

    .. autoattribute:: start_urls

    .. autoattribute:: custom_settings

    .. autoattribute:: crawler

    .. autoattribute:: settings

    .. autoattribute:: logger

    .. attribute:: state
        :type: dict[str, Any]

        Spider state to persist between batches.
        See :ref:`topics-keeping-persistent-state-between-batches` for details.

    .. automethod:: update_settings

    .. automethod:: from_crawler

    .. automethod:: start

    .. automethod:: parse

    .. method:: closed(reason)

        Called when the spider closes. This method provides a shortcut to
        signals.connect() for the :signal:`spider_closed` signal.

Let's see an example:

.. code-block:: python

    import scrapy


    class MySpider(scrapy.Spider):
        name = "example.com"
        allowed_domains = ["example.com"]
        start_urls = [
            "http://www.example.com/1.html",
            "http://www.example.com/2.html",
            "http://www.example.com/3.html",
        ]

        def parse(self, response):
            self.logger.info("A response from %s just arrived!", response.url)

Return multiple Requests and items from a single callback:

.. code-block:: python

    import scrapy


    class MySpider(scrapy.Spider):
        name = "example.com"
        allowed_domains = ["example.com"]
        start_urls = [
            "http://www.example.com/1.html",
            "http://www.example.com/2.html",
            "http://www.example.com/3.html",
        ]

        def parse(self, response):
            for h3 in response.xpath("//h3").getall():
                yield {"title": h3}

            for href in response.xpath("//a/@href").getall():
                yield scrapy.Request(response.urljoin(href), self.parse)

Instead of :attr:`~.start_urls` you can use :meth:`~scrapy.Spider.start`
directly; to give data more structure you can use :class:`~scrapy.Item`
objects:

.. skip: next
.. code-block:: python

    import scrapy
    from myproject.items import MyItem


    class MySpider(scrapy.Spider):
        name = "example.com"
        allowed_domains = ["example.com"]

        async def start(self):
            yield scrapy.Request("http://www.example.com/1.html", self.parse)
            yield scrapy.Request("http://www.example.com/2.html", self.parse)
            yield scrapy.Request("http://www.example.com/3.html", self.parse)

        def parse(self, response):
            for h3 in response.xpath("//h3").getall():
                yield MyItem(title=h3)

            for href in response.xpath("//a/@href").getall():
                yield scrapy.Request(response.urljoin(href), self.parse)

.. _spiderargs:

Spider arguments
================

Spiders can receive arguments that modify their behaviour. Some common uses for
spider arguments are to define the start URLs or to restrict the crawl to
certain sections of the site, but they can be used to configure any
functionality of the spider.

Spider arguments are passed through the :command:`crawl` command using the
``-a`` option. For example::

    scrapy crawl myspider -a category=electronics

Spiders can access arguments in their `__init__` methods:

.. code-block:: python

    import scrapy


    class MySpider(scrapy.Spider):
        name = "myspider"

        def __init__(self, category=None, *args, **kwargs):
            super().__init__(*args, **kwargs)
            self.start_urls = [f"http://www.example.com/categories/{category}"]
            # ...

The default `__init__` method will take any spider arguments
and copy them to the spider as attributes.
The above example can also be written as follows:

.. code-block:: python

    import scrapy


    class MySpider(scrapy.Spider):
        name = "myspider"

        async def start(self):
            yield scrapy.Request(f"http://www.example.com/categories/{self.category}")

If you are :ref:`running Scrapy from a script <run-from-script>`, you can
specify spider arguments when calling
:meth:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or
:meth:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:

.. skip: next
.. code-block:: python

    process = CrawlerProcess()
    process.crawl(MySpider, category="electronics")

Keep in mind that spider arguments are only strings.
The spider will not do any parsing on its own.
If you were to set the ``start_urls`` attribute from the command line,
you would have to parse it on your own into a list
using something like :func:`ast.literal_eval` or :func:`json.loads`
and then set it as an attribute.
Otherwise, you would cause iteration over a ``start_urls`` string
(a very common python pitfall)
resulting in each character being seen as a separate url.

Spider arguments can also be passed through the Scrapyd ``schedule.json`` API.
See `Scrapyd documentation`_.

.. _spiderargs-scrapy-spider-metadata:

scrapy-spider-metadata parameters
---------------------------------

Another alternative to pass spider arguments is the library `scrapy-spider-metadata`_.

This allows for Scrapy spiders to define, validate, document and pre-process
their arguments as Pydantic models.

The example shows how to define typed parameters where a string argument
is automatically converted to an integer:

.. code-block:: python

    import scrapy
    from pydantic import BaseModel
    from scrapy_spider_metadata import Args


    class MyParams(BaseModel):
        pages: int


    class BookSpider(Args[MyParams], scrapy.Spider):
        name = "bookspider"
        start_urls = ["http://books.toscrape.com/catalogue"]

        async def start(self):
            for start_url in self.start_urls:
                for index in range(1, self.args.pages + 1):
                    yield scrapy.Request(f"{start_url}/page-{index}.html")

        def parse(self, response):
            book_links = response.css("article.product_pod h3 a::attr(href)").getall()
            for book_link in book_links:
                yield response.follow(book_link, self.parse_book)

        def parse_book(self, response):
            yield {
                "title": response.css("h1::text").get(),
                "price": response.css("p.price_color::text").get(),
            }

This spider can be called from the command line::

    scrapy crawl bookspider -a pages=2

.. _start-requests:

Start requests
==============

**Start requests** are :class:`~scrapy.Request` objects yielded from the
:meth:`~scrapy.Spider.start` method of a spider or from the
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` method of a
:ref:`spider middleware <topics-spider-middleware>`.

.. seealso:: :ref:`start-request-order`

.. _start-requests-lazy:

Delaying start request iteration
--------------------------------

Scrapy iterates :meth:`~scrapy.Spider.start` as fast as it yields, so all start
requests reach the scheduler early in the crawl, however many they are. To
minimize the number of requests in the scheduler at any given time, and with it
resource usage (memory, or disk when using :setting:`JOBDIR`), override
:meth:`~scrapy.Spider.start` to pause its iteration whenever there are
scheduled requests:

.. code-block:: python

    async def start(self):
        async for item_or_request in super().start():
            if self.crawler.engine.needs_backout():
                await self.crawler.signals.wait_for(signals.scheduler_empty)
            yield item_or_request

.. _builtin-spiders:

Generic Spiders
===============

Scrapy comes with some useful generic spiders that you can use to subclass
your spiders from. Their aim is to provide convenient functionality for a few
common scraping cases, like following all links on a site based on certain
rules, crawling from `Sitemaps`_, or parsing an XML/CSV feed.

For the examples used in the following spiders, we'll assume you have a project
with a ``TestItem`` declared in a ``myproject.items`` module:

.. code-block:: python

    from dataclasses import dataclass


    @dataclass
    class TestItem:
        id: str | None = None
        name: str | None = None
        description: str | None = None


.. currentmodule:: scrapy.spiders

CrawlSpider
-----------

.. autoclass:: CrawlSpider

    .. autoattribute:: rules

    .. automethod:: parse_start_url

Crawling rules
~~~~~~~~~~~~~~

.. autoclass:: Rule

CrawlSpider example
~~~~~~~~~~~~~~~~~~~

Let's now take a look at an example CrawlSpider with rules:

.. code-block:: python

    from scrapy.spiders import CrawlSpider, Rule
    from scrapy.linkextractors import LinkExtractor


    class MySpider(CrawlSpider):
        name = "example.com"
        allowed_domains = ["example.com"]
        start_urls = ["http://www.example.com"]

        rules = (
            # Extract links matching 'category.php' (but not matching 'subsection.php')
            # and follow links from them (since no callback means follow=True by default).
            Rule(LinkExtractor(allow=(r"category\.php",), deny=(r"subsection\.php",))),
            # Extract links matching 'item.php' and parse them with the spider's method parse_item
            Rule(LinkExtractor(allow=(r"item\.php",)), callback="parse_item"),
        )

        def parse_item(self, response):
            self.logger.info("Hi, this is an item page! %s", response.url)
            item = {}
            item["id"] = response.xpath('//td[@id="item_id"]/text()').re(r"ID: (\d+)")
            item["name"] = response.xpath('//td[@id="item_name"]/text()').get()
            item["description"] = response.xpath(
                '//td[@id="item_description"]/text()'
            ).get()
            item["link_text"] = response.meta["link_text"]
            url = response.xpath('//td[@id="additional_data"]/@href').get()
            return response.follow(
                url, self.parse_additional_page, cb_kwargs=dict(item=item)
            )

        def parse_additional_page(self, response, item):
            item["additional_data"] = response.xpath(
                '//p[@id="additional_data"]/text()'
            ).get()
            return item


This spider would start crawling example.com's home page, collecting category
links, and item links, parsing the latter with the ``parse_item`` method. For
each item response, some data will be extracted from the HTML using XPath, and
a dictionary will be filled with it.

XMLFeedSpider
-------------

.. autoclass:: XMLFeedSpider

    .. autoattribute:: iterator

    .. autoattribute:: itertag

    .. autoattribute:: namespaces

    .. automethod:: adapt_response

    .. automethod:: parse_node

    .. automethod:: process_results


XMLFeedSpider example
~~~~~~~~~~~~~~~~~~~~~

These spiders are pretty easy to use, let's have a look at one example:

.. skip: next
.. code-block:: python

    from scrapy.spiders import XMLFeedSpider
    from myproject.items import TestItem


    class MySpider(XMLFeedSpider):
        name = "example.com"
        allowed_domains = ["example.com"]
        start_urls = ["http://www.example.com/feed.xml"]
        iterator = "iternodes"  # This is actually unnecessary, since it's the default value
        itertag = "item"

        def parse_node(self, response, node):
            self.logger.info(
                "Hi, this is a <%s> node!: %s", self.itertag, "".join(node.getall())
            )

            item = TestItem()
            item.id = node.xpath("@id").get()
            item.name = node.xpath("name").get()
            item.description = node.xpath("description").get()
            return item

Basically what we did up there was to create a spider that downloads a feed from
the given ``start_urls``, and then iterates through each of its ``item`` tags,
prints them out, and stores some random data in an :class:`~scrapy.Item`.

CSVFeedSpider
-------------

.. autoclass:: CSVFeedSpider

    .. autoattribute:: delimiter

    .. autoattribute:: quotechar

    .. autoattribute:: headers

    .. automethod:: adapt_response

    .. automethod:: parse_row

    .. automethod:: process_results

CSVFeedSpider example
~~~~~~~~~~~~~~~~~~~~~

Let's see an example similar to the previous one, but using a
:class:`CSVFeedSpider`:

.. skip: next
.. code-block:: python

    from scrapy.spiders import CSVFeedSpider
    from myproject.items import TestItem


    class MySpider(CSVFeedSpider):
        name = "example.com"
        allowed_domains = ["example.com"]
        start_urls = ["http://www.example.com/feed.csv"]
        delimiter = ";"
        quotechar = "'"
        headers = ["id", "name", "description"]

        def parse_row(self, response, row):
            self.logger.info("Hi, this is a row!: %r", row)

            item = TestItem()
            item.id = row["id"]
            item.name = row["name"]
            item.description = row["description"]
            return item


SitemapSpider
-------------

.. autoclass:: SitemapSpider

    .. autoattribute:: sitemap_urls

    .. autoattribute:: sitemap_rules

    .. autoattribute:: sitemap_follow

    .. autoattribute:: sitemap_alternate_links

    .. automethod:: sitemap_filter


SitemapSpider examples
~~~~~~~~~~~~~~~~~~~~~~

Simplest example: process all urls discovered through sitemaps using the
``parse`` callback:

.. code-block:: python

    from scrapy.spiders import SitemapSpider


    class MySpider(SitemapSpider):
        sitemap_urls = ["http://www.example.com/sitemap.xml"]

        def parse(self, response):
            pass  # ... scrape item here ...

Process some urls with certain callback and other urls with a different
callback:

.. code-block:: python

    from scrapy.spiders import SitemapSpider


    class MySpider(SitemapSpider):
        sitemap_urls = ["http://www.example.com/sitemap.xml"]
        sitemap_rules = [
            ("/product/", "parse_product"),
            ("/category/", "parse_category"),
        ]

        def parse_product(self, response):
            pass  # ... scrape product ...

        def parse_category(self, response):
            pass  # ... scrape category ...

Follow sitemaps defined in the `robots.txt`_ file and only follow sitemaps
whose url contains ``/sitemap_shop``:

.. code-block:: python

    from scrapy.spiders import SitemapSpider


    class MySpider(SitemapSpider):
        sitemap_urls = ["http://www.example.com/robots.txt"]
        sitemap_rules = [
            ("/shop/", "parse_shop"),
        ]
        sitemap_follow = ["/sitemap_shops"]

        def parse_shop(self, response):
            pass  # ... scrape shop here ...

Combine SitemapSpider with other sources of urls:

.. code-block:: python

    from scrapy import Request
    from scrapy.spiders import SitemapSpider


    class MySpider(SitemapSpider):
        sitemap_urls = ["http://www.example.com/robots.txt"]
        sitemap_rules = [
            ("/shop/", "parse_shop"),
        ]

        other_urls = ["http://www.example.com/about"]

        async def start(self):
            async for item_or_request in super().start():
                yield item_or_request
            for url in self.other_urls:
                yield Request(url, self.parse_other)

        def parse_shop(self, response):
            pass  # ... scrape shop here ...

        def parse_other(self, response):
            pass  # ... scrape other here ...

.. _scrapy-spider-metadata: https://scrapy-spider-metadata.readthedocs.io/en/latest/params.html
.. _Sitemaps: https://www.sitemaps.org/index.html
.. _robots.txt: https://www.robotstxt.org/
.. _Scrapyd documentation: https://scrapyd.readthedocs.io/en/latest/