Module event_processor.scrapy_impl.polite_log_formatter

Classes

class PoliteLogFormatter (*args, **kwargs)

Class for generating log messages for different actions.

All methods must return a dictionary listing the parameters level, msg and args which are going to be used for constructing the log message when calling logging.log.

Dictionary keys for the method outputs:

• level is the log level for that action, you can use those from the python logging library <https://docs.python.org/3/library/logging.html>_ : logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR and logging.CRITICAL.
• msg should be a string that can contain different formatting placeholders. This string, formatted with the provided args, is going to be the long message for that action.
• args should be a tuple or dict with the formatting placeholders for msg. The final log message is computed as msg % args.

Users can define their own LogFormatter class if they want to customize how each action is logged or if they want to omit it entirely. In order to omit logging an action the method must return None.

Here is an example on how to create a custom log formatter to lower the severity level of the log message when an item is dropped from the pipeline::

    class PoliteLogFormatter(logformatter.LogFormatter):
        def dropped(self, item, exception, response, spider):
            return {
                'level': logging.INFO, # lowering the level from logging.WARNING
                'msg': u"Dropped: %(exception)s" + os.linesep + "%(item)s",
                'args': {
                    'exception': exception,
                    'item': item,
                }
            }

Expand source code

class PoliteLogFormatter(logformatter.LogFormatter):
    def dropped(self, item, exception, response, spider):
        return {
            'level': logging.INFO,
            'msg': logformatter.DROPPEDMSG,
            'args': {
                'exception': exception,
                'item': item,
            }
        }

Ancestors

• scrapy.logformatter.LogFormatter

Methods

def dropped(self, item, exception, response, spider)

Logs a message when an item is dropped while it is passing through the item pipeline.

Expand source code

def dropped(self, item, exception, response, spider):
    return {
        'level': logging.INFO,
        'msg': logformatter.DROPPEDMSG,
        'args': {
            'exception': exception,
            'item': item,
        }
    }