Logging in Clojure / JVM – Part 2

In part 1 , we looked at the history of logging in Java. This time, we’ll learn more about SLF4J (Simple Logging Facade for Java).

From part 1, we saw that SLF4J is not a proxy to other logging frameworks, rather it is an API that allows end users to inject their desired logging framework at deployment time. SLF4J comes with adapters for many commonly used logging frameworks.

For my side project, I’m using SLF4J API. As many other projects, my project is dependent on many libraries. Unfortunately, not all libraries use SLF4J; and indeed, some of them use log4j API. You’ll be surprised to see how many newly written libraries use log4j (even though log4j is old and horrid). Even if they see the benefits to changing to SLF4J, it probably won’t happen soon. SLF4J comes with bridging modules for JCL, JUL and log4j to consolidate logging. These bridging modules redirect calls made to log4j, JCL and JUL to SLF4J instead. The image below explains the idea.


[From http://www.slf4j.org/legacy.html]

Mapped Diagnostic Context

Another awesome feature of SLF4J is MDC ( Mapped Diagnostic Context). Even though it sounds very complicated, it is simple to use and yet so powerful. MDC is essentially a hash map maintained by the logging framework that can be inserted into log messages. Applications can update this hash map using SLF4J. Currently only log4j and logback offer MDC functionality. SLF4J will simply delegate to log4j or logback. If you use some other logging framework, then SLF4J will maintain the hash map, but you’ll need to write some custom code to retrieve the information from the map.

What’s the use of MDC?

One of the main goals of logging is to audit and debug complex real-world, distributed systems. These systems handle multiple clients simultaneously. So log messages are going to be interleaved. So, it’s very important to consolidate log messages of a single client or a single API call. The simplest way is to tag all log messages with client-info and trace-id (we’ll discuss more about this in the next part of this series). Without MDC, we’d need to put this information in every logging call. With MDC, all we have to do is setup the context (client-info , trace-id etc) and all our log messages will automatically have this information. This transforms our log messages into an amazing resource to learn about the system and its users.

Using MDC in Clojure:

Unfortunately, as of now, clojure.tools.logging does not support MDC. I think this a big hole in clojure.tools.logging, as Clojure is known for building complex systems. Luckily the Clojure community is vibrant, and there’s an open source project by Malcolm Sparks called clj-logging-config. The main purpose of this library is to programmatically setup logging config files. But there is one function, with-logging-context that allows us to setup MDC. Even though I’m not programmatically setting up logging config in my project right now, I am using this library just for this with-logging-context function. I strongly believe this library (or at least the with-logging-context function) should be part of clojure.tools.logging.

So there! In this part we learnt more about SLF4J and MDC. In the next part we will learn more about structured logging.

Logging in Clojure / JVM – Part 1

Ok this title is little misleading. This blog post is not just about logging in clojure (or the JVM), but also about the whole amazing world of log management and analysis. I am working on this side project during nights and weekends. Since it’s a green-field project, I wanted to get logging right. Ater all, at the end of all my past projects , I always came to the conclusion that I could have done logging better. It isn’t just me, but even when I speak with other software developers they also feel that logging could use more attention. This is my attempt to help myself and in the process help others get logging right.

Logging library in clojure:

It is a very simple ecosystem. We have clojure/tools.logging. It is a set of logging macros that delegates to specific logging implementations. OK, and what does specific logging implementation mean? To understand this better, we need to understand the logging ecosystem in Java.

History of logging in Java:

Log4J was the first well known java logging library. In fact, it is still used in many projects. It’s probably the most popular.

When Sun realized that logging is important, instead of incorporating log4j, they went ahead and created another logging framework. It was called JUL (short form for java.utils.logging). Honestly, I do not see any benefit of using JUL over log4j. They probably went “hey, log4j wasn’t invented here, so lets create something else to do the same thing”. So that pretty much created the first split between java libraries where some were using log4j and some were using JUL. Already, it started getting difficult to make different libraries work together.

As libraries should not impose the use of a particular logging implementation, another project created called Commons Logging came to be. It was advertised as an ultra thin bridge between different logging implementations. So a library using commons logging could change the logging implementation (log4j or JUL ) at runtime. It used class-loader magic and dynamic binding to load specific logging implementations. Of course, it became complex to debug issues. In the end, it created more problems than it solved, and people were not particularly happy about commons logging.

So the creator of Log4j, Ceki Gulcu, created a logging facade called SL4j (short form for “Simple Logging Facade for Java”). SLF4J, unlike commons logging is an API, which allows end users to plug-in their desired logging system at deployment time. It does not have any class-loader magic or dynamic binding like commons loggings. SLF4J binding is hardwired at compile time to use a specific framework. As slf4j is an API, it needed an adapter layer on top of log4j, JUL and commons logging. SLF4J already comes with adaptor layers for commonly used logging frameworks.


[image from http://www.slf4j.org/manual.html]

Ceki Gulcu also felt it was time to improve log4j. So he created logback. You can go through the reasons to switch from log4j (or JUL) to logback. For now logback is considered the best logging implementation we have for the JVM. The best part of logback is it natively implements SLF4J API. So we do not need any slf4j adaptor layer to use logback with slf4j.

Now that we have seen the logging ecosystem in JVM, in the next part of this series, we’ll take a look some more at SLF4J.