avatarPete Fison

Summary

The context provides a tutorial on how to use the log2d Python library to create organized log files for every module, class, or instance of a class with minimal effort.

Abstract

The article explains how to use the log2d library in Python to create a more organized logging system, dividing the output into smaller, more digestible log files. The tutorial provides three recipes for organizing log messages based on modules, instances, or classes. The author highlights the benefits of this approach, particularly for web scraping applications, and showcases the simplicity of the log2d library compared to Python's built-in logging module.

Bullet points

  • The article introduces the log2d library for organizing Python logging output.
  • The author explains their use case for the library: an overnight batch job with 30+ Python classes for web scraping.
  • The tutorial provides three recipes for organizing log messages based on modules, instances, or classes.
  • The benefits of this approach include smaller, more digestible log files and the simplicity of the log2d library.
  • The author compares the log2d library to Python's built-in logging module, highlighting its simplicity.
  • The article concludes by encouraging readers to try out the log2d library for their own logging needs.

Get organised with Python logging…

Hi everyone — if you missed my first article on getting started with Python logging the easy way, please head over there now and have a quick read. This article builds on that introduction and shows how you can automatically create a log file for every module, class or instance of a class with minimal effort. Great for getting organised with smaller, more focused logs, rather than wading through one juggernaut and inevitably writing yet more code just to extract, separate, or organise your output after the event. Let’s get you set up for success from the outset!

So… I finally worked out how to save myself a lot of time and stress with an overnight batch job I’ve been running that uses 30+ Python classes (one per website scraper) and I just wanted to share how easy it is to organise your output with log2d :

python -m pip install log2d --upgrade

Instead of ending up with one massive log file with (tens of) thousands of lines of output, all jumbled up because I “cleverly” (hah!) set the scrapers to run in a multi-threaded way, this approach divides the output into a much more digestible form. You can tweak this approach for your particular style and needs but here’s what I was aiming for:

  • One log file (and console output) per MODULE for Progress messages like “Starting Scraper X”.
  • One log file per scraper INSTANCE (Amazon.de, Amazon.co.uk, Amazon.com etc.) for Retries, for example when product pages are successfully scraped, but not on the first attempt.
  • One log file per scraper INSTANCE (Amazon.de, Amazon.co.uk, Amazon.com etc.) for Failures, for example following connection problems, HTTP error messages, or authentication problems.
  • One log file per scraper CLASS for Errors i.e. actual problems with my code that require debugging.

Although this approach of “dividing and conquering” is particularly suited to web scraping applications where you typically have one Class per scraper (the “Scrapy” model), the simple recipes below should help anyone who wants to log output on a per-Module, per-Class or per-Instance basis for any reason whatsoever, without having to delve into the dark depths of Python’s rather convoluted logging module.

So without further ado, here’s the code…

Recipe 1: One logger per Module

Nice colourful logs using VS Code

Recipe 2: One logger per Instance

Recipe 3: One logger per Class

Simple eh? And even shorter to type .log than it is to type print().

That’s all there is to say really — three different recipes depending on how you want to group your log messages, and all the complexity of the standard Python Logging module are discreetly hidden from view thanks to log2d .

Happy logging!

Python
Web Scraping
Scraping
Logging
Object Oriented
Recommended from ReadMedium