Updated March 30, 2023
Introduction to Logstash Multiline
Logstash Multiline codec is the plugin available in logstash which was released in September 2021 and the latest version of this plugin available is version 3.1.1 which actually helps us in collapsing the messages that are in multiline format and then result into a single event combining and merging all of the messages. In this article, we will have a deeper study of what logstash multiline is and will try to understand it by using the subtopics which include What is logstash multiline, logstash multiline codec, logstash multiline configuration, and conclusion about the same.
What is logstash multiline?
Logstash multiline is the available functionality in which there are certain scenarios in which events generated are in such a manner that contains the text of multiple lines which are also referred to as multiline events. For handling this type of event in logstash, there needs to be a mechanism using which it will be able to tell which lines inside the event belong to the single event. Proper event ordering needs to be followed as the processing of multiline events is a very critical and complex job. That is why the processing of order arrangement is done at an early stage inside the pipelines.
This is where multiline codec comes into the picture which is a tool for the management of multiline events that processes during the stage of the logstash pipeline.
Logstash Multiline code
Logstash multiline codec is the tool that takes into consideration particular set of rules which makes it possible to merge lines that come from a single input source. It merges all the multiline messages into a single event. The main motive of the logstash multiline codec is to allow the task of combining the multiline messages that come from files and result into a single event. Let us consider an example to understand this which makes it possible to combine messages of the stack trace and java exceptions resulting to a single event.
The configuration for setting the multiline codec plugin will look as shown below –
Input{
Stdin{
Codec => multiline {
Pattern => “regexp”
What => “next” or “previous”
Negate => “false” or “true”
}
}
}
You need to make sure that the part of the multiline event which is a field should satisfy the pattern specified. The what attribute helps in the specification of the relation of multiline events. The attribute negates here can have either true or false value which when not specified is treated to be false. This field means that if the message does not match with the filter for multiline then it will contain a pattern in it and vice versa.
Considering an example to understand this most of the stack traces of java have messages of multiline format and also, they began from the left side of the data containing all the lines properly well-indented. Hence, in such case, we can specify the pattern as “^\s” and what can be given a value of “previous” inside the codec=> multiline for standard input which means that if the line contains the whitespace at the start of it then it will be from the previous line.
Before we go and dive into the configurations and available options, let’s have a look at one example where we will be considering the lines which do not begin with the date and the previous line to be merged. For this, our configurations of the file for the input section will be as shown below –
Input {
File {
Codec => multiline {
Negate => true
Pattern => “^ % {TIMESTAMP_ISO8601}”
What => “previous”
}
Path => “/etc/logs/sampleEducbaApp.log”
}
}
The output of configurations inside the file along with indentation will look as shown below –
This methodology has one more application where it is used quite commonly which is in C programming language when you have to implement line continuations along with backslashes in it then we can set the configurations for multiline logstash using codec as shown below –
Input {
Stdin {
Codec => multiline {
What => “next”
Pattern => “\\$”
}
}
}
Which is the file looks as shown below –
This configuration specifies that if any of the specified lines ends along with the presence of backslash then that particular line should be combined along with the line that will be followed.
logstash multiline configuration
There are certain configuration options that you can specify to define the behavior and working of logstash codec configurations. The below table includes the configuration options for logstash multiline codec –
| Configuration setting | Type of Input | Optional/ Required |
| Charset | String value from the particular set of values mentioned in documents as it defines the standards followed by the character set. For a complete list of supported string values, please refer to this link. | Optional |
| Auto flush interval | An integer that is the number value | Optional |
| Max bytes | Byte value | Optional |
| Pattern | String | Required |
| What | String value which can have either “next” or “previous” value set to it. | Required |
| Pattern dir | It should be an array value | Optional |
| Max lines | Number value | Optional |
| Ecs compatibility | String value | Optional |
| Negate | Boolean value | Optional |
| Multiline tag | String value | Optional |
- Auto_flush_interval – This configuration will allow you to convert a particular event in the case when a new line that is matching is discovered or new data is not appended for the specified second’s value. The default value corresponds to no.
- Pattern – It is the regular expression value that is used for the purpose of matching the parts of lines.
- What – Whenever a match is found for the pattern then recognize if the event is a part of the previous or next event.
- Patterns_dir – If you might be adding some more patterns then you can make use of this configuration as shipping of a bunch of patterns is carried out by default by logstash.
Conclusion
Logstash multiline is the case where some of the events of logstash may generate the messages that are of multiline. In case to handle this, there is an in-built plugin available in logstash named multiline codec logstash plugin which helps in specifying the behavior of multiline event processing and handling of same.
Recommended Articles
This is a guide to Logstash Multiline. Here we discuss the Introduction, What is logstash multiline? Examples with code implementation. You may also have a look at the following articles to learn more –
