diff --git a/docs/faq.rst b/docs/faq.rst index 7a0628f88..69edb0993 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -319,6 +319,29 @@ I'm scraping a XML document and my XPath selector doesn't return any items You may need to remove namespaces. See :ref:`removing-namespaces`. +How to split an item into multiple items in an item pipeline? +------------------------------------------------------------- + +:ref:`Item pipelines ` cannot yield multiple items per +input item. :ref:`Create a spider middleware ` +instead, and use its +:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` +method for this puspose. For example:: + + from copy import deepcopy + + from scrapy.item import BaseItem + + + class MultiplyItemsMiddleware: + + def process_spider_output(self, response, result, spider): + for item in result: + if isinstance(item, (BaseItem, dict)): + for _ in range(item['multiply_by']): + yield deepcopy(item) + + .. _user agents: https://en.wikipedia.org/wiki/User_agent .. _LIFO: https://en.wikipedia.org/wiki/Stack_(abstract_data_type) .. _DFO order: https://en.wikipedia.org/wiki/Depth-first_search diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 30b8638bd..0e8210130 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -54,6 +54,8 @@ value. For example, if you want to disable the off-site middleware:: Finally, keep in mind that some middlewares may need to be enabled through a particular setting. See each middleware documentation for more info. +.. _custom-spider-middleware: + Writing your own spider middleware ==================================