This is part of the series of blog posts:
- Application Logging in Java: Creating a Logging Framework
- Application Logging in Java: Adding Configuration
- Application Logging in Java: Converters
- Application Logging in Java: Markers
- Application Logging in Java: Appenders
- Application Logging in Java: Logging Costs
- Application Logging in Java: Encoders
- Application Logging in Java: Tracing 3rd Party Code
- Application Logging in Java: Filters
- Application Logging in Java: Putting it all together
Encoders are a tucked away part of Logback. They are used to convert a
LoggingEvent to a stream of bytes, and so are responsible for the "formatting" of entries inside of appenders.
The attention that encoders receive is minimal enough that the API on the Logback manual is way out of date! It's actually:
The most visible encoder is PatternLayoutEncoder, which is a classic line oriented layout, terminated by newlines. In Application Logging in Java: Converters I quoted:
At the present time, PatternLayoutEncoder is the only really useful encoder. It merely wraps a PatternLayout which does most of the work. Thus, it may seem that encoders do not bring much to the table except needless complexity. However, we hope that with the advent of new and powerful encoders this impression will change.
So now it's time to talk about encoders. This is not the only encoder out there: logstash-logback-encoder provides a JSON lines oriented encoder, but for the most part encoders are not widely used – there's no "XML encoder" or "Avro encoder" that I could find anywhere.
Encoders also don't stack. While there's something like functional composition with converters, it's harder to do effective functional composition with encoders. The reason why is that
PatternLayoutEncoder doesn't compose very well. It expects the parent to be an
Appender, and so you can't simply wrap inside a parent encoder: it will fail with a
not-assignable-from error. So stuff like this:
is difficult to do even if you'd expect it to make intuitive sense.
Encoders are powerful and useful. They give you access to the raw bytes, and let you manipulate them before they get to an appender. But you'll have to put them together inside an appender if you want to do byte transformation.
As an example, say that we want to write out files directly in zstandard or brotli using Logback. The easiest way to do this is to provide a
FileAppender with a swapped out compression encoder, while presenting a public API that looks just like a regular encoder.
Here's the appender as
logback.xml sees it:
Under the hood,
CompressingFileAppender delegates to a regular file appender, but uses commons-compress and a
CompressingEncoder to wrap
From there, the encoder will shove all the input bytes into a compressed stream until there's enough data to make compression worthwhile, and then flush the compressed bytes out through a byte array output stream:
This keeps both
PatternLayoutEncoder happy, while feeding compressed bytes as the stream. Using delegation is generally much easier than trying to extend from
FileAppender has very definite ideas about what kind of output stream it is using, and has all the logic of file rotation and backups encorporated into it, including its own gzip compression scheme for rotated files.
You can also extend this to add dictionary support for ZStandard, and that would remove the need for a buffer to provide effective compression. This does come with the downside of needing to pass the dictionary out of band though.
Encoders can be particularly useful for getting a sense of how much and where logging is coming from. Given that logs are expensive, being able to track down exactly where the budget is going can be very useful, i.e.
The implementation of a budget aware appender is left as an exercise for the reader.