Updated March 16, 2023
Introduction to LogstashEncoder
LogstashEncoder helps get the functionality to log back the layouts, encoders, and the appenders in JSON format inside the logs. It also allows doing so for other formats that Jackson further supports. In this article, we will look at LogstashEncoder and will also study in detail the same by using the subtopics, which include what LogstashEncoder is, how to use LogstashEncoder, LogstashEncoder java examples, and a conclusion about the same.
What is LogstashEncoder?
LogsatshEncoder is the provision that helps log backing the appenders, layouts, and encoders specified in JSON or Jackson supported format. LogstashEncoder supports tIn addition, the log backing of access events and logging events. For access events, logging is carried out by using log back-access, while for logging events, loggers are used logging.
LogstashEncoder was created initially for the output support for JSON format events in Logstash. Still, it soon became and evolved into a mechanism that is structured perfectly for logging with high provisions of configurations and which can be used for general purposes and can log the JSON format and Jackson data formats. We can configure the data present in it and the output structure using LogstashEncoder.
How to use LogstashEncoder?
To use LogstashEncoder, you will have to include it in your project. We can use Logstash-log back-encoder 6.0 or greater versions for java versions 1.8 or greater. For example, for 5.x versions of Logstash log back encoders, the required java versions can be 1.7 or above, while for versions of Logstash log back encoders that are 4.x or lesser than that, we will need java versions with 1.6 version or greater.
You can go for adding the dependency of LogstashEncoder in your maven project inside the pom.xml file by adding the snippet as shown below –
<dependency>
<groupId>net.Logstash.log back</groupId>
<artifactId>Logstash-log back-encoder</artifactId>
<version>7.0.1</version>
</dependency>
<dependency>
<groupId>ch.qos.log back</groupId>
<artifactId>log back-classic</artifactId>
<version>1.2.6</version>
</dependency>Here, you must ensure that your project directly depends on the Logstash access or the Logstash classic. Also, you can specify the tag of <scope> having value as runtime when there is no use of LogstashEncoders or during compile time.
This scenario happens when using and implementing JsonDeneratorDecoder, JsonFactoryDecoder, AppederListener, Markers or StructuredArguments, JsonProvider, Layout, Appender, TurboFilter, Encoder, etc. When executing at runtime, if we get the exceptions of NoClassDefFoundError, ClassNotFoundException, or NosuchMethodError, then you need to cross-check the dependencies that you have specified in the pom.xml of your maven project and make sure all the necessary dependencies are specified with proper versions of it. The path of the runtime is specified in the scope tag.
Some of the dependencies that should be added to the runtime path are as mentioned below –
- Slf4j-api
- Locgback-core of version 1.2.0
- Log back access
- Log back classic
- Jackson-core
- Jackson databind
- Jackson annotations
Now, to use the LogstashEncoder for logging the JSON format, the configurations for log backing either of the following should be made –
- Configuration related to the appended will be made available by the logsatsh log back encoder library.
- Configuration of the appended is provided by any other library or library of log back along with the encoder or layout made available by the library of Logstash log back encoder.
The below table lists out all the encoders, appenders, and layouts that are provided by the library of Logstash log back encoder –
| Function | Protocol | Supported format | AccessEvent | LoggingEvent |
| Appender | Any | Any | AccessEventAsyncDisruptorApper | LoggingEventAsyncDisruptorApper |
| Appender | TCP | Logstash JSON format | LogstashAccessTCPSocketAppender | LogstashTCPSocketAppender |
| Appender | /Syslog/UDP | Logstash JSON format | LogstashAccessUdpSocketAppender | LogstashUdpSocketAppender |
| Layout | Any | General JSON format | AccessEventCompositeJsonLayout | LoggingEventCompositeJsonLayout |
| Layout | Any | Logsatsh JSON format | LogsasthAccessLayout | LogstashLayout |
| Encoder | Any | Logstash JSON structure | LogsatshAccessEncoder | LogstashEncoder |
| Encoder | Any | General JSON format | AccessEventCompositeJsonEncoder | LoggingEventCompositeJsonEncoder |
LogstashEncoder java examples
When we have to send the JSON output to the channel of type UDP or Syslog for all the logging events, we can use LogstashUdpSocketAppender and the presence LoggingEventCompositeJsonLayout or the LogstashLayout while specifying the configurations in our file log back.xml. For this scenario, our configurations can be set as shown in the below example –
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<root level="all">
<appender-ref ref="sampleEducbaStash" />
</root>
<appender class="net.Logstash.log back.appender.LogstashUdpSocketAppender" name="sampleEducbaStash" >
<host>SamplEducbaServerForSystemLogs</host>
<port>555</port>
<layout class="net.Logstash.log back.layout.LogstashLayout"/>
</appender>
</configuration>We recommend configuring the custom fields for a global scope when required in our JSON output. For this, we will add the following specification in our configuration file of Logstash.xml –
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<root level="all">
<appender-ref ref="sampleEducbaStash" />
</root>
<appender class="net.Logstash.log back.appender.LogstashUdpSocketAppender" name="sampleEducbaStash" >
<host>SamplEducbaServerForSystemLogs</host>
<port>555</port>
<layout class="net.Logstash.log back.layout.LogstashLayout">
<customFields>{"appname":"sampleEducbaWebService"}</customFields>
</layout>
</appender>
</configuration>When we require the output in the JSON while using UDP for the access events, then we can make the use of LogsatshAccesUdpSocketAppender along with the use of AccessEventCompositeJsonLayout or as an alternative, you can also use LogstashAccessLayout inside the log back-access.xml as shown below –
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender class="net.Logstash.log back.appender.LogstashAccessUdpSocketAppender" name="educbaLogstash" >
<host>SampleEducbaLogstash</host>
<port>555</port>
<layout class="net.Logstash.log back.layout.LogstashAccessLayout">
<customFields>{"nameOfApplication":"educbaLogstash”} </customFields>
</layout>
</appender>
<appender-ref ref="educbaLogstash" />
</configuration>We will need to set the configurations of input of udp or Syslog along with the codec mentioned as json for getting the inputs of UDP or syslogs in Logstash. The configuration can be set as shown below –
Input {
Syslog{
Codec => "json"
}
}In this way, depending on your input format, like Logstash JSON or general JSON, you will decide the configurations and contents of Logstash config files. Depending on the Logstash log back encoder, you can also decide the functions to be applied, which can be either Layout, appended, or encoder, specified in the pom.xml configuration file of your maven project.
Conclusion
LogstashEncoder allows us to get the log back of various components, including appenders, layouts, and encoders of Logstash in the JSON format, which can be decided depending on the configurations specified in the config file of Logstash and maven, which helps in managing the dependencies in your java-based projects.
Recommended Articles
This is a guide to LogstashEncoder. Here we discuss the various components, including appenders, layouts, and encoders of Logstash in the JSON format. You may also look at the following articles to learn more –
