RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/logging.rst

12 KiB
Raw Permalink Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Logging

Scrapy uses :mod:`logging` for event logging. We'll provide some simple examples to get you started, but for more advanced use-cases it's strongly suggested to read thoroughly its documentation.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "mod".

Logging works out of the box, and can be configured to some extent with the Scrapy settings listed in :ref:`topics-logging-settings`.

System Message: ERROR/3 (<stdin>, line 11); backlink

Unknown interpreted text role "ref".

Scrapy calls :func:`scrapy.utils.log.configure_logging` to set some reasonable defaults and handle those settings in :ref:`topics-logging-settings` when running commands, so it's recommended to manually call it if you're running Scrapy from scripts as described in :ref:`run-from-script`.

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "func".

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "ref".

Log levels

Python's builtin logging defines 5 different levels to indicate the severity of a given log message. Here are the standard ones, listed in decreasing order:

  1. logging.CRITICAL - for critical errors (highest severity)
  2. logging.ERROR - for regular errors
  3. logging.WARNING - for warning messages
  4. logging.INFO - for informational messages
  5. logging.DEBUG - for debugging messages (lowest severity)

How to log messages

Here's a quick example of how to log a message using the logging.WARNING level:

System Message: WARNING/2 (<stdin>, line 39)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging

    logging.warning("This is a warning")

There are shortcuts for issuing log messages on any of the standard 5 levels, and there's also a general logging.log method which takes a given level as argument. If needed, the last example could be rewritten as:

System Message: WARNING/2 (<stdin>, line 49)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging

    logging.log(logging.WARNING, "This is a warning")

On top of that, you can create different "loggers" to encapsulate messages. (For example, a common practice is to create different loggers for every module). These loggers can be configured independently, and they allow hierarchical constructions.

The previous examples use the root logger behind the scenes, which is a top level logger where all messages are propagated to (unless otherwise specified). Using logging helpers is merely a shortcut for getting the root logger explicitly, so this is also an equivalent of the last snippets:

System Message: WARNING/2 (<stdin>, line 65)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging

    logger = logging.getLogger()
    logger.warning("This is a warning")

You can use a different logger just by getting its name with the logging.getLogger function:

System Message: WARNING/2 (<stdin>, line 75)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging

    logger = logging.getLogger("mycustomlogger")
    logger.warning("This is a warning")

Finally, you can ensure having a custom logger for any module you're working on by using the __name__ variable, which is populated with current module's path:

System Message: WARNING/2 (<stdin>, line 86)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging

    logger = logging.getLogger(__name__)
    logger.warning("This is a warning")

System Message: ERROR/3 (<stdin>, line 93)

Unknown directive type "seealso".

.. seealso::

    Module logging, :doc:`HowTo <howto/logging>`
        Basic Logging Tutorial

    Module logging, :ref:`Loggers <logger>`
        Further documentation on loggers

Logging from Spiders

Scrapy provides a :data:`~scrapy.Spider.logger` within each Spider instance, which can be accessed and used like this:

System Message: ERROR/3 (<stdin>, line 106); backlink

Unknown interpreted text role "data".

System Message: WARNING/2 (<stdin>, line 109)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import scrapy


    class MySpider(scrapy.Spider):
        name = "myspider"
        start_urls = ["https://scrapy.org"]

        def parse(self, response):
            self.logger.info("Parse function called on %s", response.url)

That logger is created using the Spider's name, but you can use any custom Python logger you want. For example:

System Message: WARNING/2 (<stdin>, line 124)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging
    import scrapy

    logger = logging.getLogger("mycustomlogger")


    class MySpider(scrapy.Spider):
        name = "myspider"
        start_urls = ["https://scrapy.org"]

        def parse(self, response):
            logger.info("Parse function called on %s", response.url)

Logging configuration

Loggers on their own don't manage how messages sent through them are displayed. For this task, different "handlers" can be attached to any logger instance and they will redirect those messages to appropriate destinations, such as the standard output, files, emails, etc.

By default, Scrapy sets and configures a handler for the root logger, based on the settings below.

Logging settings

These settings can be used to configure the logging:

The first couple of settings define a destination for log messages. If :setting:`LOG_FILE` is set, messages sent through the root logger will be redirected to a file named :setting:`LOG_FILE` with encoding :setting:`LOG_ENCODING`. If unset and :setting:`LOG_ENABLED` is True, log messages will be displayed on the standard error. If :setting:`LOG_FILE` is set and :setting:`LOG_FILE_APPEND` is False, the file will be overwritten (discarding the output from previous runs, if any). Lastly, if :setting:`LOG_ENABLED` is False, there won't be any visible log output.

System Message: ERROR/3 (<stdin>, line 169); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 169); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 169); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 169); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 169); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 169); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 169); backlink

Unknown interpreted text role "setting".

:setting:`LOG_LEVEL` determines the minimum level of severity to display, those messages with lower severity will be filtered out. It ranges through the possible levels listed in :ref:`topics-logging-levels`.

System Message: ERROR/3 (<stdin>, line 178); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 178); backlink

Unknown interpreted text role "ref".

:setting:`LOG_FORMAT` and :setting:`LOG_DATEFORMAT` specify formatting strings used as layouts for all messages. Those strings can contain any placeholders listed in :ref:`logging's logrecord attributes docs <logrecord-attributes>` and :ref:`datetime's strftime and strptime directives <strftime-strptime-behavior>` respectively.

System Message: ERROR/3 (<stdin>, line 182); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 182); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 182); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 182); backlink

Unknown interpreted text role "ref".

If :setting:`LOG_SHORT_NAMES` is set, then the logs will not display the Scrapy component that prints the log. It is unset by default, hence logs contain the Scrapy component responsible for that log output.

System Message: ERROR/3 (<stdin>, line 188); backlink

Unknown interpreted text role "setting".

Rotating log files

Scrapy's :setting:`LOG_FILE` setting writes logs to a single file. It does not rotate log files automatically, but you can use Python's standard :mod:`logging.handlers` module when running Scrapy from a script.

System Message: ERROR/3 (<stdin>, line 195); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 195); backlink

Unknown interpreted text role "mod".

For example, to rotate the log file every day:

System Message: WARNING/2 (<stdin>, line 203)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging
    from logging.handlers import TimedRotatingFileHandler

    from scrapy.crawler import CrawlerProcess
    from scrapy.utils.project import get_project_settings

    from myproject.spiders.myspider import MySpider

    settings = get_project_settings()
    process = CrawlerProcess(settings, install_root_handler=False)

    handler = TimedRotatingFileHandler(
        "scrapy.log",
        when="midnight",
        backupCount=7,
        encoding=settings.get("LOG_ENCODING"),
    )
    handler.setFormatter(
        logging.Formatter(settings.get("LOG_FORMAT"), settings.get("LOG_DATEFORMAT"))
    )

    root_logger = logging.getLogger()
    root_logger.setLevel(settings.get("LOG_LEVEL"))
    root_logger.addHandler(handler)

    process.crawl(MySpider)
    process.start()

Command-line options

There are command-line arguments, available for all commands, that you can use to override some of the Scrapy settings regarding logging.

  • --logfile FILE

    Overrides :setting:`LOG_FILE`

    System Message: ERROR/3 (<stdin>, line 241); backlink

    Unknown interpreted text role "setting".

  • --loglevel/-L LEVEL

    Overrides :setting:`LOG_LEVEL`

    System Message: ERROR/3 (<stdin>, line 243); backlink

    Unknown interpreted text role "setting".

  • --nolog

    Sets :setting:`LOG_ENABLED` to False

    System Message: ERROR/3 (<stdin>, line 245); backlink

    Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 247)

Unknown directive type "seealso".

.. seealso::

    Module :mod:`logging.handlers`
        Further documentation on available handlers

Custom Log Formats

A custom log format can be set for different actions by extending :class:`~scrapy.logformatter.LogFormatter` class and making :setting:`LOG_FORMATTER` point to your new class.

System Message: ERROR/3 (<stdin>, line 257); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 257); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 261)

Unknown directive type "autoclass".

.. autoclass:: scrapy.logformatter.LogFormatter
   :members:

Advanced customization

Because Scrapy uses stdlib logging module, you can customize logging using all features of stdlib logging.

For example, let's say you're scraping a website which returns many HTTP 404 and 500 responses, and you want to hide all messages like this:

2016-12-16 22:00:06 [scrapy.spidermiddlewares.httperror] INFO: Ignoring
response <500 https://quotes.toscrape.com/page/1-34/>: HTTP status code
is not handled or not allowed

The first thing to note is a logger name - it is in brackets: [scrapy.spidermiddlewares.httperror]. If you get just [scrapy] then :setting:`LOG_SHORT_NAMES` is likely set to True; set it to False and re-run the crawl.

System Message: ERROR/3 (<stdin>, line 280); backlink

Unknown interpreted text role "setting".

Next, we can see that the message has INFO level. To hide it we should set logging level for scrapy.spidermiddlewares.httperror higher than INFO; next level after INFO is WARNING. It could be done e.g. in the spider's __init__ method:

System Message: WARNING/2 (<stdin>, line 290)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging
    import scrapy


    class MySpider(scrapy.Spider):
        # ...
        def __init__(self, *args, **kwargs):
            logger = logging.getLogger("scrapy.spidermiddlewares.httperror")
            logger.setLevel(logging.WARNING)
            super().__init__(*args, **kwargs)

If you run this spider again then INFO messages from scrapy.spidermiddlewares.httperror logger will be gone.

You can also filter log records by :class:`~logging.LogRecord` data. For example, you can filter log records by message content using a substring or a regular expression. Create a :class:`logging.Filter` subclass and equip it with a regular expression pattern to filter out unwanted messages:

System Message: ERROR/3 (<stdin>, line 306); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 306); backlink

Unknown interpreted text role "class".

System Message: WARNING/2 (<stdin>, line 312)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging
    import re


    class ContentFilter(logging.Filter):
        def filter(self, record):
            match = re.search(r"\d{3} [Ee]rror, retrying", record.message)
            if match:
                return False

A project-level filter may be attached to the root handler created by Scrapy, this is a wieldy way to filter all loggers in different parts of the project (middlewares, spider, etc.):

System Message: WARNING/2 (<stdin>, line 329)

Cannot analyze code. Pygments package not found.

.. code-block:: python

 import logging
 import scrapy


 class MySpider(scrapy.Spider):
     # ...
     def __init__(self, *args, **kwargs):
         for handler in logging.root.handlers:
             handler.addFilter(ContentFilter())

Alternatively, you may choose a specific logger and hide it without affecting other loggers:

System Message: WARNING/2 (<stdin>, line 344)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging
    import scrapy


    class MySpider(scrapy.Spider):
        # ...
        def __init__(self, *args, **kwargs):
            logger = logging.getLogger("my_logger")
            logger.addFilter(ContentFilter())

scrapy.utils.log module

System Message: ERROR/3 (<stdin>, line 360)

Unknown directive type "module".

.. module:: scrapy.utils.log
   :synopsis: Logging utils

System Message: ERROR/3 (<stdin>, line 363)

Unknown directive type "autofunction".

.. autofunction:: configure_logging

    ``configure_logging`` is automatically called when using Scrapy commands
    or :class:`~scrapy.crawler.CrawlerProcess`, but needs to be called explicitly
    when running custom scripts using :class:`~scrapy.crawler.CrawlerRunner`.
    In that case, its usage is not required but it's recommended.

    Another option when running custom scripts is to manually configure the logging.
    To do this you can use :func:`logging.basicConfig` to set a basic root handler.

    Note that :class:`~scrapy.crawler.CrawlerProcess` automatically calls ``configure_logging``,
    so it is recommended to only use :func:`logging.basicConfig` together with
    :class:`~scrapy.crawler.CrawlerRunner`.

    This is an example on how to redirect ``INFO`` or higher messages to a file:

    .. code-block:: python

        import logging

        logging.basicConfig(
            filename="log.txt", format="%(levelname)s: %(message)s", level=logging.INFO
        )

    Refer to :ref:`run-from-script` for more details about using Scrapy this
    way.
</html>