Updated March 14, 2023
Introduction to Logstash if field exists
Logstash if field exists is the conditional scenario where many events and actions have to be performed only if provided the field exists in the event. In this article, we will be learning about logstash if field exists and will try to understand its integrities and details using the subtopics which include logstash if field exists overviews, how to check logstash if field exists, how to use logstash if field exists, logstash if field exists examples and Conclusion about the same.
Overview of Logstash if field exists
The agent of logstash is used for pipeline processing which is carried out in 3 different stages which include input proceeded with filters and then with the outputs. All three stages perform a particular task which is event generation is carried out at the input stage. Further, the events are modified by filter according to specifications and requirements and at the output stage, the events are shipped somewhere else.
There are various properties also referred to as fields to all the events. Let us for example consider the event of Apache access log which has fields such as HTTP verb which can have either POST or GET value, request path which usually has the value set to / which is root or index.html, status code for the request which can be 404 for not found and 200 for complete, IP address of the client and many others. In logstash, all these properties are referred to as fields.
There are many scenarios when you want some action to take place only if that field exists or there are many options in the configuration settings of logstash which can carry out their functionality only provided if the required fields exist. The reason for having this dependency is because in logstash the inputs are responsible for the generation of the events. In case the required fields don’t exist then there will not be the fields that help in the evaluation of the input blocks.
Also, note that at the input stage as the events are generated which means that during that stage fields are not even created yet. Hence the working of the option settings related to configuration only happens at filter and output stages. And in case if required field does not exist then it will not work there as well.
How to check logstash if field exists?
In order to understand how fields can be checked for their existence. Firstly, let us understand how we can refer to the fields. Usually, when the fields are of top-level we can simply refer to them by enclosing the name of field between square braces like for example [name_of_field]. When the fields are nested fields then starting from their top-level field we drill down and then write the name of our nested field. For example, suppose the top-level field is denoted by tlf and our reference field has named as sample_educba then it can refer as [tlf][sample_educba].
Step 1 – Choose the name of the field that you are going to check the existence of. Suppose, my field name is educba_field then it will be written in logstash while referring as shown in the below image –
Step 2 – Put the if condition along with the field reference inside the filter section or wherever you want to check the existence of the field. In the case of our example, the condition will be somewhat like below –
Step 3 – Run your code and check whether the block of implementation executes when the field does not exist. Note that it will also execute if a field exists and it contains a false value which is a negative Boolean value. Executing the example discussed in the below section with the required implementation gives the following output –
How to use logstash if field exists?
Let us now have a look at how we can go for checking the existence of field and only then proceed with our functionality with the help of an example –
If [educba_field]
// implementation of code which should be executed only if a field exists
Note that in the above example the educba_field is the name of the field and the expression if [educba_field] will return false for three different conditions which also includes the condition where the field does exist but with a false value. All the three scenarios where we will get them if the condition specified above as false are as specified below –
• There is no existence of the field named [educba_field] inside the current event.
• The field named [educba_field] does exist in the event but contains a false value in it.
• The field named [educba_field] does exist in the event but contains the null value in it.
This is the reason why there is no specific way available right now which can allow us to check the existence of the field. Even though the above condition always returns was when field does not exist but returning false is not enough to say that field does not exist. It may even come to the scenario where a field contains null or false values.
logstash if field exists examples
Let us consider one example of logstash file where in the filter section we will confirm if the field does not exist and if it exist then it contains a false value in it and then only go for adding the 42 as the string value to the field that will be created. The contents of the file look as shown below –
input {
http { port => 5559 }
}
filter {
# In case if the field contains a false value that is a boolean then we are carrying out conversion to make it 0 in integer
mutate {
convert => { "educba_field" => "integer" }
}
# The specified condition proves to be true only in case if there is no existence of educba_field or it exists but has false as the boolean value for it
if ![educba_field] or [educba_field] == 0 {
mutate {
replace => { "educba_field" => 42 }
# We will replace the existing value to 42 in string because when field not present
# then it creates the one with 42 value in string format by logstash
# which will be now converted to integer.
convert => { "educba_field" => "integer" }
}
}
}
output {
stdout {}
}
The execution of the above logstash file gives the following output –
This is the output when field exist and is further converted to a string number. When the field does not exist, we get the following output –
Conclusion
Logstash if field exists is to check whether the specified field is present inside the event or not. Though there is not a proper solution designed to check the existence of the field but still by using the if [name of field], we can verify whether the field is not existing or if existing then contains the false Boolean value in it.
Recommended Articles
This is a guide to Logstash if field exists. Here we discuss the Introduction, overview, How to check logstash if field exists?, Examples with code implementation. You may also have a look at the following articles to learn more –
