3.7 KiB

Raw Blame History

Logging

Scrapy provides a logging facility which can be used through the :mod:`scrapy.log` module. The current underlying implementation uses Twisted logging but this may change in the future.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "mod".

The logging service must be explicitly started through the :func:`scrapy.log.start` function to catch the top level Scrapy's log messages. On top of that, each crawler has its own independent log observer (automatically attached when it's created) that intercepts its spider's log messages.

System Message: ERROR/3 (<stdin>, line 13); backlink

Unknown interpreted text role "func".

Log levels

Scrapy provides 5 logging levels:

:data:`~scrapy.log.CRITICAL` - for critical errors

System Message: ERROR/3 (<stdin>, line 26); backlink

Unknown interpreted text role "data".
:data:`~scrapy.log.ERROR` - for regular errors

System Message: ERROR/3 (<stdin>, line 27); backlink

Unknown interpreted text role "data".
:data:`~scrapy.log.WARNING` - for warning messages

System Message: ERROR/3 (<stdin>, line 28); backlink

Unknown interpreted text role "data".
:data:`~scrapy.log.INFO` - for informational messages

System Message: ERROR/3 (<stdin>, line 29); backlink

Unknown interpreted text role "data".
:data:`~scrapy.log.DEBUG` - for debugging messages

System Message: ERROR/3 (<stdin>, line 30); backlink

Unknown interpreted text role "data".

How to set the log level

You can set the log level using the --loglevel/-L command line option, or using the :setting:`LOG_LEVEL` setting.

System Message: ERROR/3 (<stdin>, line 35); backlink

Unknown interpreted text role "setting".

How to log messages

Here's a quick example of how to log a message using the WARNING level:

from scrapy import log
log.msg("This is a warning", level=log.WARNING)

Logging from Spiders

The recommended way to log from spiders is by using the Spider :meth:`~scrapy.spider.Spider.log` method, which already populates the spider argument of the :func:`scrapy.log.msg` function. The other arguments are passed directly to the :func:`~scrapy.log.msg` function.

System Message: ERROR/3 (<stdin>, line 49); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 49); backlink

Unknown interpreted text role "func".

System Message: ERROR/3 (<stdin>, line 49); backlink

Unknown interpreted text role "func".

scrapy.log module

System Message: ERROR/3 (<stdin>, line 57)

Unknown directive type "module".

.. module:: scrapy.log
   :synopsis: Logging facility

System Message: ERROR/3 (<stdin>, line 60)

Unknown directive type "function".

.. function:: start(logfile=None, loglevel=None, logstdout=None)

    Start the top level Scrapy logger. This must be called before actually
    logging any top level messages (those logged using this module's
    :func:`~scrapy.log.msg` function instead of the :meth:`Spider.log
    <scrapy.spider.Spider.log>` method). Otherwise, messages logged before this
    call will get lost.

    :param logfile: the file path to use for logging output. If omitted, the
        :setting:`LOG_FILE` setting will be used. If both are ``None``, the log
        will be sent to standard error.
    :type logfile: str

    :param loglevel: the minimum logging level to log. Available values are:
        :data:`CRITICAL`, :data:`ERROR`, :data:`WARNING`, :data:`INFO` and
        :data:`DEBUG`.

    :param logstdout: if ``True``, all standard output (and error) of your
        application will be logged instead. For example if you "print 'hello'"
        it will appear in the Scrapy log. If omitted, the :setting:`LOG_STDOUT`
        setting will be used.
    :type logstdout: boolean

System Message: ERROR/3 (<stdin>, line 83)

Unknown directive type "function".

.. function:: msg(message, level=INFO, spider=None)

    Log a message

    :param message: the message to log
    :type message: str

    :param level: the log level for this message. See
        :ref:`topics-logging-levels`.

    :param spider: the spider to use for logging this message. This parameter
        should always be used when logging things related to a particular
        spider.
    :type spider: :class:`~scrapy.spider.Spider` object

System Message: ERROR/3 (<stdin>, line 98)