12 KiB
Items
System Message: ERROR/3 (<stdin>, line 7)
Unknown directive type "module".
.. module:: scrapy.item :synopsis: Item and Field classes
The main goal in scraping is to extract structured data from unstructured sources, typically, web pages. :ref:`Spiders <topics-spiders>` may return the extracted data as items, Python objects that define key-value pairs.
System Message: ERROR/3 (<stdin>, line 10); backlink
Unknown interpreted text role "ref".Scrapy supports :ref:`multiple types of items <item-types>`. When you create an item, you may use whichever type of item you want. When you write code that receives an item, your code should :ref:`work for any item type <supporting-item-types>`.
System Message: ERROR/3 (<stdin>, line 14); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 14); backlink
Unknown interpreted text role "ref".Item Types
Scrapy supports the following types of items, via the itemadapter library: :ref:`dictionaries <dict-items>`, :ref:`Item objects <item-objects>`, :ref:`dataclass objects <dataclass-items>`, :ref:`attrs objects <attrs-items>` and :ref:`Pydantic models <pydantic-items>`.
System Message: ERROR/3 (<stdin>, line 24); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 24); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 24); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 24); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 24); backlink
Unknown interpreted text role "ref".Dictionaries
As an item type, :class:`dict` is convenient and familiar.
System Message: ERROR/3 (<stdin>, line 36); backlink
Unknown interpreted text role "class".Item objects
:class:`Item` provides a :class:`dict`-like API plus additional features that make it the most feature-complete item type:
System Message: ERROR/3 (<stdin>, line 43); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 43); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 46)
Unknown directive type "autoclass".
.. autoclass:: scrapy.Item :members: copy, deepcopy, fields :undoc-members:
:class:`Item` objects replicate the standard :class:`dict` API, including its __init__ method.
System Message: ERROR/3 (<stdin>, line 50); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 50); backlink
Unknown interpreted text role "class".:class:`Item` allows the defining of field names, so that:
System Message: ERROR/3 (<stdin>, line 53); backlink
Unknown interpreted text role "class".:class:`KeyError` is raised when using undefined field names (i.e. prevents typos going unnoticed)
System Message: ERROR/3 (<stdin>, line 55); backlink
Unknown interpreted text role "class".
:ref:`Item exporters <topics-exporters>` can export all fields by default even if the first scraped object does not have values for all of them
System Message: ERROR/3 (<stdin>, line 58); backlink
Unknown interpreted text role "ref".
:class:`Item` also allows the defining of field metadata, which can be used to :ref:`customize serialization <topics-exporters-field-serialization>`.
System Message: ERROR/3 (<stdin>, line 62); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 62); backlink
Unknown interpreted text role "ref".:mod:`scrapy.utils.trackref` tracks :class:`Item` objects to help find memory leaks (see :ref:`topics-leaks-trackrefs`).
System Message: ERROR/3 (<stdin>, line 65); backlink
Unknown interpreted text role "mod".System Message: ERROR/3 (<stdin>, line 65); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 65); backlink
Unknown interpreted text role "ref".Example:
System Message: WARNING/2 (<stdin>, line 70)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from scrapy.item import Item, Field
class CustomItem(Item):
one_field = Field()
another_field = Field()
Dataclass objects
:func:`~dataclasses.dataclass` allows the defining of item classes with field names, so that :ref:`item exporters <topics-exporters>` can export all fields by default even if the first scraped object does not have values for all of them.
System Message: ERROR/3 (<stdin>, line 84); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 84); backlink
Unknown interpreted text role "ref".Additionally, dataclass items also allow you to:
define the type and default value of each defined field.
define custom field metadata through :func:`dataclasses.field`, which can be used to :ref:`customize serialization <topics-exporters-field-serialization>`.
System Message: ERROR/3 (<stdin>, line 92); backlink
Unknown interpreted text role "func".
System Message: ERROR/3 (<stdin>, line 92); backlink
Unknown interpreted text role "ref".
Example:
System Message: WARNING/2 (<stdin>, line 97)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from dataclasses import dataclass
@dataclass
class CustomItem:
one_field: str
another_field: int
Note
Field types are not enforced at run time.
attr.s objects
:func:`attr.s` allows the defining of item classes with field names, so that :ref:`item exporters <topics-exporters>` can export all fields by default even if the first scraped object does not have values for all of them.
System Message: ERROR/3 (<stdin>, line 114); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 114); backlink
Unknown interpreted text role "ref".Additionally, attr.s items also allow to:
define the type and default value of each defined field.
define custom field :ref:`metadata <attrs:metadata>`, which can be used to :ref:`customize serialization <topics-exporters-field-serialization>`.
System Message: ERROR/3 (<stdin>, line 122); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 122); backlink
Unknown interpreted text role "ref".
In order to use this type, the :doc:`attrs package <attrs:index>` needs to be installed.
System Message: ERROR/3 (<stdin>, line 125); backlink
Unknown interpreted text role "doc".Example:
System Message: WARNING/2 (<stdin>, line 129)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import attr
@attr.s
class CustomItem:
one_field = attr.ib()
another_field = attr.ib()
Pydantic models
Pydantic models allow the defining of item classes with field names, so that :ref:`item exporters <topics-exporters>` can export all fields by default even if the first scraped object does not have values for all of them.
System Message: ERROR/3 (<stdin>, line 145); backlink
Unknown interpreted text role "ref".Additionally, pydantic items also allow you to:
define the type and default value of each defined field with run-time type validation.
define custom field metadata through pydantic.Field, which can be used to :ref:`customize serialization <topics-exporters-field-serialization>`.
System Message: ERROR/3 (<stdin>, line 155); backlink
Unknown interpreted text role "ref".
benefit from automatic data validation and conversion based on type annotations.
In order to use this type, the pydantic package needs to be installed.
Example:
System Message: WARNING/2 (<stdin>, line 167)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from pydantic import BaseModel, Field
class CustomItem(BaseModel):
one_field: str = Field(default="", description="First field")
another_field: int = Field(default=0, description="Second field")
Note
Unlike other item types, Pydantic models enforce field types at run time and will raise validation errors for invalid data types.
Working with Item objects
Declaring Item subclasses
Item subclasses are declared using a simple class definition syntax and :class:`Field` objects. Here is an example:
System Message: ERROR/3 (<stdin>, line 187); backlink
Unknown interpreted text role "class".System Message: WARNING/2 (<stdin>, line 190)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
class Product(scrapy.Item):
name = scrapy.Field()
price = scrapy.Field()
stock = scrapy.Field()
tags = scrapy.Field()
last_updated = scrapy.Field(serializer=str)
Note
Those familiar with Django will notice that Scrapy Items are declared similar to Django Models, except that Scrapy Items are much simpler as there is no concept of different field types.
Declaring fields
:class:`Field` objects are used to specify metadata for each field. For example, the serializer function for the last_updated field illustrated in the example above.
System Message: ERROR/3 (<stdin>, line 215); backlink
Unknown interpreted text role "class".You can specify any kind of metadata for each field. There is no restriction on the values accepted by :class:`Field` objects. For this same reason, there is no reference list of all available metadata keys. Each key defined in :class:`Field` objects could be used by a different component, and only those components know about it. You can also define and use any other :class:`Field` key in your project too, for your own needs. The main goal of :class:`Field` objects is to provide a way to define all field metadata in one place. Typically, those components whose behaviour depends on each field use certain field keys to configure that behaviour. You must refer to their documentation to see which metadata keys are used by each component.
System Message: ERROR/3 (<stdin>, line 219); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 219); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 219); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 219); backlink
Unknown interpreted text role "class".It's important to note that the :class:`Field` objects used to declare the item do not stay assigned as class attributes. Instead, they can be accessed through the :attr:`~scrapy.Item.fields` attribute.
System Message: ERROR/3 (<stdin>, line 230); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 230); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 234)
Unknown directive type "autoclass".
.. autoclass:: scrapy.Field
The :class:`Field` class is just an alias to the built-in :class:`dict` class and
doesn't provide any extra functionality or attributes. In other words,
:class:`Field` objects are plain-old Python dicts. A separate class is used
to support the :ref:`item declaration syntax <topics-items-declaring>`
based on class attributes.
Note
Field metadata can also be declared for dataclass and attrs items. Please refer to the documentation for dataclasses.field and attr.ib for additional information.
Working with Item objects
Here are some examples of common tasks performed with items, using the Product item :ref:`declared above <topics-items-declaring>`. You will notice the API is very similar to the :class:`dict` API.
System Message: ERROR/3 (<stdin>, line 255); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 255); backlink
Unknown interpreted text role "class".Creating items
System Message: WARNING/2 (<stdin>, line 262)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> product = Product(name="Desktop PC", price=1000)
>>> print(product)
{'name': 'Desktop PC', 'price': 1000}
Getting field values
System Message: WARNING/2 (<stdin>, line 272)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> product["name"]
Desktop PC
>>> product.get("name")
Desktop PC
>>> product["price"]
1000
>>> product["last_updated"]
Traceback (most recent call last):
...
KeyError: 'last_updated'
>>> product.get("last_updated", "not set")
not set
>>> product["lala"] # getting unknown field
Traceback (most recent call last):
...
KeyError: 'lala'
>>> product.get("lala", "unknown field")
'unknown field'
>>> "name" in product # is name field populated?
True
>>> "last_updated" in product # is last_updated populated?
False
>>> "last_updated" in product.fields # is last_updated a declared field?
True
>>> "lala" in product.fields # is lala a declared field?
False
Setting field values
System Message: WARNING/2 (<stdin>, line 314)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> product["last_updated"] = "today"
>>> product["last_updated"]
today
>>> product["lala"] = "test" # setting unknown field
Traceback (most recent call last):
...
KeyError: 'Product does not support field: lala'
Accessing all populated values
To access all populated values, just use the typical :class:`dict` API:
System Message: ERROR/3 (<stdin>, line 329); backlink
Unknown interpreted text role "class".System Message: WARNING/2 (<stdin>, line 331)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> product.keys()
['price', 'name']
>>> product.items()
[('price', 1000), ('name', 'Desktop PC')]
Copying items
To copy an item, you must first decide whether you want a shallow copy or a deep copy.
If your item contains :term:`mutable` values like lists or dictionaries, a shallow copy will keep references to the same mutable values across all different copies.
System Message: ERROR/3 (<stdin>, line 348); backlink
Unknown interpreted text role "term".For example, if you have an item with a list of tags, and you create a shallow copy of that item, both the original item and the copy have the same list of tags. Adding a tag to the list of one of the items will add the tag to the other item as well.
If that is not the desired behavior, use a deep copy instead.
See :mod:`copy` for more information.
System Message: ERROR/3 (<stdin>, line 359); backlink
Unknown interpreted text role "mod".To create a shallow copy of an item, you can either call :meth:`~scrapy.Item.copy` on an existing item (product2 = product.copy()) or instantiate your item class from an existing item (product2 = Product(product)).
System Message: ERROR/3 (<stdin>, line 361); backlink
Unknown interpreted text role "meth".To create a deep copy, call :meth:`~scrapy.Item.deepcopy` instead (product2 = product.deepcopy()).
System Message: ERROR/3 (<stdin>, line 366); backlink
Unknown interpreted text role "meth".Other common tasks
Creating dicts from items:
System Message: WARNING/2 (<stdin>, line 375)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> dict(product) # create a dict from all populated values
{'price': 1000, 'name': 'Desktop PC'}
Creating items from dicts:
System Message: WARNING/2 (<stdin>, line 382)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> Product({"name": "Laptop PC", "price": 1500})
{'name': 'Laptop PC', 'price': 1500}
>>> Product({"name": "Laptop PC", "lala": 1500}) # warning: unknown field in dict
Traceback (most recent call last):
...
KeyError: 'Product does not support field: lala'
Extending Item subclasses
You can extend Items (to add more fields or to change some metadata for some fields) by declaring a subclass of your original Item.
For example:
System Message: WARNING/2 (<stdin>, line 401)
Cannot analyze code. Pygments package not found.
.. code-block:: python
class DiscountedProduct(Product):
discount_percent = scrapy.Field(serializer=str)
discount_expiration_date = scrapy.Field()
You can also extend field metadata by using the previous field metadata and appending more values, or changing existing values, like this:
System Message: WARNING/2 (<stdin>, line 410)
Cannot analyze code. Pygments package not found.
.. code-block:: python
class SpecificProduct(Product):
name = scrapy.Field(Product.fields["name"], serializer=my_serializer)
That adds (or replaces) the serializer metadata key for the name field, keeping all the previously existing metadata values.
Supporting All Item Types
In code that receives an item, such as methods of :ref:`item pipelines <topics-item-pipeline>` or :ref:`spider middlewares <topics-spider-middleware>`, it is a good practice to use the :class:`~itemadapter.ItemAdapter` class to write code that works for any supported item type.
System Message: ERROR/3 (<stdin>, line 426); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 426); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 426); backlink
Unknown interpreted text role "class".