Updated March 16, 2023
Introduction to Logstash Debug
Logstash debug is the process where we try to verify and ensure that the existing settings and configurations of the Logstash file and its contents are working properly. If anything goes wrong, we can go for debugging, finding the right reasons or causes of error or warnings. In addition, debugging any program or code snippet gives us a deeper idea and understanding of how the program is running during runtime.
In this article, we will look at Logstash debug and try to understand it by studying the subtopics, including what Logstash debug is, Logstash debug configuration file, and Logstash debug parameters, Logstash debugs Logging, Logstash debugs examples, and conclusion about the same.
What is Logstash Debug?
Logstash plays a key role when it comes to working of ELK stack. You need to know well about the configurations made and the contents of the configuration file, which is of the .conf extension, not the one of Logstash.yml. One more important skill you should possess is debugging the issues and finding the appropriate cause and reason for any of the mistakes in the file or any present misconfigurations, which will add up to the task of how efficiently you will be able to use and run Logstash.
A free tool named grok debugger will enable us to test and debug all the log messages and the contained grok patterns. It performs tasks such as parsing the log data into structured and well-defined data constructions, making it easy to analyze further. It is a plugin that is quite popular among the users of Logstash. We have to use combinations and permutations of a regular expression, which means that if you have good mastery over the use of the regular expression, you can use this plugin a little easier.
We need to consider one important point: even though this plugin tool makes debugging easier, even after passing the grok debugger test, you might still find an error in the configuration or a resultant failure to grok, which gives the error of _grokparsefailure.
Logstash Debug Configuration File
We must take a deeper look at the configuration file before we jump on learning debugging techniques. The configuration file of Logstash has three main sections that help define the running behavior. These sections are input, filter, and output sections. For each section, there is a specification of all the plugins that will be used and the necessary configuration settings corresponding to the plugin.
It would help if you considered some of the points while going for writing the contents of the file or editing it. Make sure that you are using a good text editor, which helps confirm whether all the curly braces are closed and opened properly or not. Further, different configuration settings are supported by each of the plugins, and different rules for defining each. Hence, make sure that you refer to the documentation of the plugin before using and mentioning the same in the configuration file. Finally, only adding those plugins you will be using as overstuffing the useless plugins in the configuration file may degrade performance.
The structure of the configuration file is as shown below –
input {
// settings related to where the get the input data for the event
}
filter {
// plugins and debugging, logging related settings specification
}
output {
// destination where the processed, parsed data is to be stored/placed.
}Logstash Debug Parameters
We can set the value of the log.level to debug in Logstash, which will enable us to log all the information about debugging, which is very useful for developers. It will let us look at the compiled version of configurations inside the message logged while debugging. So ensure that you keep the setting of log.level = debug to enabled. One of the things to be considered important here is that if any of the plugin configuration options pass the password option containing plain text, then even this will appear inside our logs of debugging.
When debugging on the Alibaba cloud elastic search console, we will need to create a new pipeline in one of the clusters and add the necessary configurations for that pipeline. Therefore, we need to pass two parameters to it while debugging: the pipeline id and the config settings. Pipeline id mapping to the path field is automatically done for the parameter of file_extend. At the same time, the configuration settings have the structure mentioned above: input, filter, and output.
Logstash Debug Logging
Once you have crossed the configuration test and the grok patterns are verified with the help of the tool grok debugger, the probability of successfully starting the Logstash pipeline will be greatly enhanced. But note that there are some changes where you might get surprising errors when you feel a little confident about your code in Logstash. In this situation, the logs of Logstash will come to the rescue. You can go here; in the case of Linux, the path for the log file is /var/log/Logstash/Logstash-plain.log, where you will get the culprit and the main reason for the occurrence of errors.
Alternatively, you can even go for debugging the Logstash by using various print statements to give the resultant messages through std out in the events.
Logstash Debug Examples
We need to configure the grok filter plugin in the filter section of the configuration file. But, first, let us consider one sample example of the configuration file’s contents for the grok plugin specification and a suitable expression for matching.
input {
stdin { }
}
filter {
geoip {
source => "ip of the client host"
}
grok {
match => { "data_sent" => "%{EDUCBACONTENT}" }
}
date {
match => [ "on_date" , "dd/MMM/yyyy" ]
}
}
output {
elasticsearch { hosts => ["172.20.120.36:5000"] }
stdout { codec => json}
}Which gives the following output when run for testing and debugging the configuration –
Logstash will accept the input from standard input, and other plugins of filter sections like grok, geoip, and date will help process the data passed that are logs from the input. Further, the output is sent to the host specified concerning its ip and path destination.
Conclusion
Logstash debugging helps to find out the causes of the error or misconfiguration in the configuration file of Logstash. The best available tool for doing so is grokked filter plugins grok debugger. You need a clear view of your configuration file and its content to create a successfully running pipeline in Logstash.
Recommended Articles
This is a guide to Logstash Debug. Here we discuss what Logstash Debug is with its configuration file, parameter, and logging, along with an example. You may also look at the following articles to learn more –
