Structured Logging Using structlog in Python

Structured logging is a powerful technique in modern software development that enhances traditional logging by adding context and structure to log messages. This blog post will explore what structured logging is, how it differs from traditional logging, and provide a step-by-step explanation of a Python code example using the structlog library. We will also compare structlog with Python's built-in logging module.

What is Structured Logging?

Structured logging is a method of logging where log entries are not just plain text messages but are structured as key-value pairs. This approach provides several benefits over traditional logging:

• Enhanced Readability: Logs are easier to read and understand since they have a consistent structure.
• Better Query ability: Structured logs can be easily queried and analyzed using tools like Elasticsearch, Kibana, and Splunk.
• Improved Context: Additional context can be added to log entries, making it easier to diagnose issues.

Traditional Logging vs. Structured Logging

Traditional Logging:

• Plain text log messages.
• Hard to parse and query.
• Context is often lost in free-form messages.

Structured Logging:

• Log messages are structured as key-value pairs.
• Easy to parse, query, and analyze.
• Provides rich context and metadata with each log entry.

Difference Between structlog and logging Modules in Python

Python’s built-in logging module is widely used for traditional logging. It supports various logging levels and handlers, but it primarily deals with plain text messages. While it can be extended to support structured logging, it requires additional configuration and handling.

structlog is a library designed specifically for structured logging. It integrates seamlessly with the logging module but offers additional features to make structured logging easy and efficient. Key differences include:

• Ease of Use: structlog is designed with structured logging in mind, making it easier to add context and structure to log messages.
• Extensibility: structlog provides a variety of processors to manipulate log entries, such as adding timestamps, formatting exceptions, and rendering logs as JSON.
• Integration: structlog can integrate with the logging module, allowing you to use existing logging infrastructure while benefiting from structured logging features.

Step-by-Step Explanation of the Code

Let’s go through the provided code step by step to understand how structured logging is set up using structlog.

1. Initial Setup

import linecache
import logging
import sys
import structlog
from structlog.types import EventDict, ExcInfo, WrappedLogger

We start by importing the necessary modules, including linecache, logging, sys, and structlog. We also import some types from structlog.

2. Clean Up Existing Logging Configuration

root = logging.getLogger()
if root.handlers:
    for handler in root.handlers:
        root.removeHandler(handler)

This section cleans up any existing logging configuration, which is useful when running in environments like AWS Lambda where logging might be pre-configured.

3. Configure Basic Logging

logging.basicConfig(format="%(message)s", stream=sys.stdout, level=logging.INFO)

We configure basic logging to output messages to stdout with a simple format.

4. Define a Custom Exception Renderer

class StructuredExceptionRenderer:
    def __call__(self, logger: WrappedLogger, name: str, event_dict: EventDict) -> EventDict:
        if not event_dict.pop("exc_info", None):
            return event_dict
    (type, value, tb) = sys.exc_info()
        if type is None:
            return event_dict

        stack = []
        while tb is not None:
            stack.append({
                "methodName": tb.tb_frame.f_code.co_name,
                "fileName": tb.tb_frame.f_code.co_filename,
                "lineNumber": tb.tb_lineno,
                "lineText": linecache.getline(tb.tb_frame.f_code.co_filename, tb.tb_lineno).strip()
            })
            tb = tb.tb_next

        event_dict["exception"] = {
            "message": f"{type.__name__}: {str(value)}",
            "stackTrace": stack
        }

        return event_dict

This class defines a custom exception renderer that formats exception information as a structured dictionary. This is similar to how Java’s net.logstash.logback.argument.StructuredArguments package works. Here you can define custom key pair values that helps you to better understand various errors and exceptions like filename, methodname, etc.

5. Configure structlog

structlog.configure_once(
    processors=[
        structlog.stdlib.filter_by_level,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.processors.TimeStamper(fmt="iso", key="@timestamp"),
        structlog.processors.StackInfoRenderer(),
        StructuredExceptionRenderer(),
        structlog.processors.UnicodeDecoder(),
        structlog.processors.JSONRenderer()
    ],
    context_class=dict,
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

Here, we configure structlog with a series of processors:

• filter_by_level: Filters log messages based on the logging level.
• add_logger_name: Adds the logger’s name to the event dictionary.
• add_log_level: Adds the log level to the event dictionary.
• PositionalArgumentsFormatter: Formats positional arguments.
• TimeStamper: Adds a timestamp to the log entry.
• StackInfoRenderer: Adds stack information to the log entry.
• StructuredExceptionRenderer: Our custom exception renderer.
• UnicodeDecoder: Decodes unicode characters.
• JSONRenderer: Renders the log entry as JSON.

The context_class, logger_factory, and wrapper_class are also set up to ensure proper integration with the logging module.

6. Create a Logger

log = structlog.get_logger(__name__)

Finally, we create a logger instance using structlog

Example Usage in an API

To illustrate how structured logging can be used in various APIs, let’s consider the following example using graphQL resolver:

def some_resolver(root, info):
  try:
    # some functionality here
  except Exception as e:
    log.error(f"Unable to perform certain functionality: {e}", exc_info=True, resolver=info.field_name)

Conclusion

Structured logging is a powerful technique that enhances the readability, queryability, and context of log messages. The structlog library in Python makes it easy to implement structured logging by providing a variety of processors and seamless integration with the traditional logging module. By following the steps outlined in this blog, you can set up structured logging in your Python applications, making it easier to monitor and debug your code.

In this blog, we’ve covered the basics of structured logging, compared structlog with the built-in logging module, and provided a detailed explanation of a structured logging setup using structlog. With this knowledge, you can leverage the benefits of structured logging to improve your application's logging strategy.

📣 Learn Python the Right Way with LearnPython.com!

If you’re eager to boost your Python skills, LearnPython.com is a fantastic resource for beginners and intermediate learners alike. Here’s why it’s worth checking out:

✅ Interactive Learning: Dive into interactive lessons that guide you step-by-step, making it easy to grasp even the most complex concepts.

✅ Structured Curriculum: Whether you’re just starting out or looking to advance your knowledge, their well-structured courses cover everything from Python basics to more advanced topics like data science and machine learning.

✅ Hands-On Practice: You’ll get plenty of practice with real-world examples and exercises, ensuring you can apply what you learn right away.

✅ Flexible Learning: Learn at your own pace, with bite-sized lessons that fit into your busy schedule.

🔗 Explore LearnPython.com

By using my affiliate link, you’ll help support my writing at no extra cost to you. Happy learning, and thank you for your support! 🙌