From 05893e17966be9037efae8be9ded494579b28b60 Mon Sep 17 00:00:00 2001 From: Jalil SA <61639983+jxlil@users.noreply.github.com> Date: Wed, 8 Mar 2023 02:59:47 -0600 Subject: [PATCH 1/9] docs: Spider.update_settings --- docs/topics/spiders.rst | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 788bd7678..e5a539fe7 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -145,6 +145,42 @@ scrapy.Spider :param kwargs: keyword arguments passed to the :meth:`__init__` method :type kwargs: dict + .. method:: update_settings(cls, settings) + + The ``update_settings()`` method is used to modify the spider's settings + and can be called during initialization of a spider instance. + + It takes a ``Settings`` object as a parameter and adds or updates the spider's + configuration values. This method is a class method, meaning that it is + called on the Spider class and allows all instances of the Spider to share + the same configuration. + + To create class hierarchies for spiders, it is recommended to use the ``custom_settings`` + attribute instead of ``update_settings()``, as it allows for default settings to be + defined and automatically inherited by subclasses. + + For example, suppose a MySpider needs update FEEDS: + + .. code-block:: python + import scrapy + + + class MySpider(scrapy.Spider): + name = "myspider" + custom_feed = { + "/home/user/documents/items.json": { + "format": "json", + "indent": 4, + } + } + + @classmethod + def update_settings(cls, settings): + settings.setdefault("FEEDS", {}).update(cls.custom_feed) + super().update_settings(settings) + + + .. method:: start_requests() This method must return an iterable with the first Requests to crawl for From 1d862d083104405ba432ddf0d741a304381b5608 Mon Sep 17 00:00:00 2001 From: Jalil SA <61639983+jxlil@users.noreply.github.com> Date: Wed, 8 Mar 2023 03:26:38 -0600 Subject: [PATCH 2/9] fix: remove line breaks --- docs/topics/spiders.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index e5a539fe7..d501466de 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -179,8 +179,6 @@ scrapy.Spider settings.setdefault("FEEDS", {}).update(cls.custom_feed) super().update_settings(settings) - - .. method:: start_requests() This method must return an iterable with the first Requests to crawl for From 96d51c3afa979587412bf3be71351056d2ef885a Mon Sep 17 00:00:00 2001 From: Jalil SA <61639983+jxlil@users.noreply.github.com> Date: Wed, 8 Mar 2023 04:21:21 -0600 Subject: [PATCH 3/9] docs: update --- docs/topics/spiders.rst | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index d501466de..22bbf2ce4 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -145,23 +145,24 @@ scrapy.Spider :param kwargs: keyword arguments passed to the :meth:`__init__` method :type kwargs: dict - .. method:: update_settings(cls, settings) + .. classmethod:: update_settings(settings) The ``update_settings()`` method is used to modify the spider's settings and can be called during initialization of a spider instance. - It takes a ``Settings`` object as a parameter and adds or updates the spider's - configuration values. This method is a class method, meaning that it is - called on the Spider class and allows all instances of the Spider to share - the same configuration. + It takes a :class:`~scrapy.settings.Settings` object as a parameter and + adds or updates the spider's configuration values. This method is a class method, + meaning that it is called on the :class:`~scrapy.Spider` class and allows all instances + of the spider to share the same configuration. - To create class hierarchies for spiders, it is recommended to use the ``custom_settings`` + To create class hierarchies for spiders, it is recommended to use the :attr:`custom_settings` attribute instead of ``update_settings()``, as it allows for default settings to be defined and automatically inherited by subclasses. - For example, suppose a MySpider needs update FEEDS: + For example, suppose a spider needs update :setting:`FEEDS`: .. code-block:: python + import scrapy From 495372648c533fc66196cafd4991dbfd403c7df8 Mon Sep 17 00:00:00 2001 From: Jalil SA <61639983+jxlil@users.noreply.github.com> Date: Thu, 16 Mar 2023 23:14:57 -0600 Subject: [PATCH 4/9] fix: docs update_settings() --- docs/topics/spiders.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 22bbf2ce4..796db2dc8 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -148,16 +148,16 @@ scrapy.Spider .. classmethod:: update_settings(settings) The ``update_settings()`` method is used to modify the spider's settings - and can be called during initialization of a spider instance. + and is called during initialization of a spider instance. It takes a :class:`~scrapy.settings.Settings` object as a parameter and - adds or updates the spider's configuration values. This method is a class method, + can add or updates the spider's configuration values. This method is a class method, meaning that it is called on the :class:`~scrapy.Spider` class and allows all instances of the spider to share the same configuration. - To create class hierarchies for spiders, it is recommended to use the :attr:`custom_settings` - attribute instead of ``update_settings()``, as it allows for default settings to be - defined and automatically inherited by subclasses. + One of the main advantages of ``update_settings()`` is that it allows + you to dynamically add, remove or change settings based on spider arguments + or other external factors. For example, suppose a spider needs update :setting:`FEEDS`: From a1fc37cbff9645116bbf6fa63eaf9df59aac7c0a Mon Sep 17 00:00:00 2001 From: Jalil SA Date: Fri, 17 Mar 2023 12:13:05 -0600 Subject: [PATCH 5/9] Update docs/topics/spiders.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Adrián Chaves --- docs/topics/spiders.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 796db2dc8..dc417614d 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -159,7 +159,7 @@ scrapy.Spider you to dynamically add, remove or change settings based on spider arguments or other external factors. - For example, suppose a spider needs update :setting:`FEEDS`: + For example, suppose a spider needs to modify :setting:`FEEDS`: .. code-block:: python From 24f28c415caadcf20f4a564ebd6bd52bea9b61ad Mon Sep 17 00:00:00 2001 From: Jalil SA <61639983+jxlil@users.noreply.github.com> Date: Fri, 17 Mar 2023 12:16:08 -0600 Subject: [PATCH 6/9] fix: docs update_settings() --- docs/topics/spiders.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index dc417614d..64c2a3ae0 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -156,7 +156,7 @@ scrapy.Spider of the spider to share the same configuration. One of the main advantages of ``update_settings()`` is that it allows - you to dynamically add, remove or change settings based on spider arguments + you to dynamically add, remove or change settings based on other settings or other external factors. For example, suppose a spider needs to modify :setting:`FEEDS`: From 44cdaa442bf25e360c8acf757623116a5eda5bba Mon Sep 17 00:00:00 2001 From: Jalil SA Date: Fri, 17 Mar 2023 13:19:03 -0600 Subject: [PATCH 7/9] Update docs/topics/spiders.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Adrián Chaves --- docs/topics/spiders.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 64c2a3ae0..0f6c2b1ba 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -151,7 +151,7 @@ scrapy.Spider and is called during initialization of a spider instance. It takes a :class:`~scrapy.settings.Settings` object as a parameter and - can add or updates the spider's configuration values. This method is a class method, + can add or update the spider's configuration values. This method is a class method, meaning that it is called on the :class:`~scrapy.Spider` class and allows all instances of the spider to share the same configuration. From 72462a53e2cfcac3ec6068fc94fec657c73e157c Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Wed, 2 Aug 2023 12:32:53 +0400 Subject: [PATCH 8/9] Add more docs for update_settings(). --- docs/topics/settings.rst | 20 ++++++++++++++++++-- docs/topics/spiders.rst | 17 +++++++++++------ 2 files changed, 29 insertions(+), 8 deletions(-) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 219509c1e..d0f2acd89 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -66,8 +66,8 @@ Example:: ---------------------- Spiders (See the :ref:`topics-spiders` chapter for reference) can define their -own settings that will take precedence and override the project ones. They can -do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute: +own settings that will take precedence and override the project ones. One way +to do so is by setting their :attr:`~scrapy.Spider.custom_settings` attribute: .. code-block:: python @@ -81,6 +81,22 @@ do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute: "SOME_SETTING": "some value", } +It's often better to provide a :meth:`~scrapy.Spider.update_settings` instead, +and settings set there should use the "spider" priority explicitly: + +.. code-block:: python + + import scrapy + + + class MySpider(scrapy.Spider): + name = "myspider" + + @classmethod + def update_settings(cls, settings): + settings.set("SOME_SETTING", "some value", priority="spider") + super().update_settings(settings) + 3. Project settings module -------------------------- diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 0f6c2b1ba..97b525bd6 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -151,13 +151,18 @@ scrapy.Spider and is called during initialization of a spider instance. It takes a :class:`~scrapy.settings.Settings` object as a parameter and - can add or update the spider's configuration values. This method is a class method, - meaning that it is called on the :class:`~scrapy.Spider` class and allows all instances - of the spider to share the same configuration. + can add or update the spider's configuration values. This method is a + class method, meaning that it is called on the :class:`~scrapy.Spider` + class and allows all instances of the spider to share the same + configuration. - One of the main advantages of ``update_settings()`` is that it allows - you to dynamically add, remove or change settings based on other settings - or other external factors. + While per-spider settings can be set in + :attr:`~scrapy.Spider.custom_settings`, using ``update_settings()`` + allows you to dynamically add, remove or change settings based on other + settings, spider attributes or other factors and use setting priorities + other than ``'spider'``. Also, it's easy to extend ``update_settings()`` + in a subclass by overriding it, while doing the same with + :attr:`~scrapy.Spider.custom_settings` is hard or impossible. For example, suppose a spider needs to modify :setting:`FEEDS`: From af1be835e4a6c14634acb382568935c1a7e10445 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Wed, 2 Aug 2023 19:46:16 +0400 Subject: [PATCH 9/9] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Adrián Chaves --- docs/topics/settings.rst | 4 ++-- docs/topics/spiders.rst | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index d0f2acd89..0963f835f 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -81,7 +81,7 @@ to do so is by setting their :attr:`~scrapy.Spider.custom_settings` attribute: "SOME_SETTING": "some value", } -It's often better to provide a :meth:`~scrapy.Spider.update_settings` instead, +It's often better to implement :meth:`~scrapy.Spider.update_settings` instead, and settings set there should use the "spider" priority explicitly: .. code-block:: python @@ -94,8 +94,8 @@ and settings set there should use the "spider" priority explicitly: @classmethod def update_settings(cls, settings): - settings.set("SOME_SETTING", "some value", priority="spider") super().update_settings(settings) + settings.set("SOME_SETTING", "some value", priority="spider") 3. Project settings module -------------------------- diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 97b525bd6..5c3bf6e72 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -162,7 +162,7 @@ scrapy.Spider settings, spider attributes or other factors and use setting priorities other than ``'spider'``. Also, it's easy to extend ``update_settings()`` in a subclass by overriding it, while doing the same with - :attr:`~scrapy.Spider.custom_settings` is hard or impossible. + :attr:`~scrapy.Spider.custom_settings` can be hard. For example, suppose a spider needs to modify :setting:`FEEDS`: @@ -182,8 +182,8 @@ scrapy.Spider @classmethod def update_settings(cls, settings): - settings.setdefault("FEEDS", {}).update(cls.custom_feed) super().update_settings(settings) + settings.setdefault("FEEDS", {}).update(cls.custom_feed) .. method:: start_requests()