DOC how to override log level for a specific Scrapy component

This commit is contained in:
Mikhail Korobov 2016-12-16 22:14:54 +05:00
parent 0fc73a9d55
commit da19f0b7b7
1 changed files with 38 additions and 1 deletions

View File

@ -10,7 +10,7 @@ Logging
about the new logging system.
Scrapy uses `Python's builtin logging system
<https://docs.python.org/2/library/logging.html>`_ for event logging. We'll
<https://docs.python.org/3/library/logging.html>`_ for event logging. We'll
provide some simple examples to get you started, but for more advanced
use-cases it's strongly suggested to read thoroughly its documentation.
@ -193,6 +193,43 @@ to override some of the Scrapy settings regarding logging.
Module `logging.handlers <https://docs.python.org/2/library/logging.handlers.html>`_
Further documentation on available handlers
Advanced customization
----------------------
Because Scrapy uses stdlib logging module, you can customize logging using
all features of stdlib logging.
For example, let's say you're scraping a website which returns many
HTTP 404 and 500 responses, and you want to hide all messages like this::
2016-12-16 22:00:06 [scrapy.spidermiddlewares.httperror] INFO: Ignoring
response <500 http://quotes.toscrape.com/page/1-34/>: HTTP status code
is not handled or not allowed
The first thing to note is a logger name - it is in brackets:
``[scrapy.spidermiddlewares.httperror]``. If you get just ``[scrapy]`` then
:setting:`LOG_SHORT_NAMES` is likely set to True; set it to False and re-run
the crawl.
Next, we can see that the message has INFO level. To hide it
we should set logging level for ``scrapy.spidermiddlewares.httperror``
higher than INFO; next level after INFO is WARNING. It could be done
e.g. in the spider's ``__init__`` method::
import logging
import scrapy
class MySpider(scrapy.Spider):
# ...
def __init__(self, *args, **kwargs):
logger = logging.getLogger('scrapy.spidermiddlewares.httperror')
logger.setLevel(logging.WARNING)
super().__init__(*args, **kwargs)
If you run this spider again then INFO messages from
``scrapy.spidermiddlewares.httperror`` logger will be gone.
scrapy.utils.log module
=======================