Updated March 15, 2023
Introduction to logstash.yml
logstash.yml is a settings file used to specify configurations for controlling the execution of logstash. In this article, we will throw certain light on the topic of logstash.yml by looking at the subtopics, including What is logstash.yml, logstash.yml, Configure logstash.yml, and Conclusion about the same.
What is logstash.yml?
Logstash.yml is a configuration settings file that helps maintain control over the execution of logstash. First, we can try to understand the usage and purpose of the logstash.yml configuration settings file by considering a small example. Then, when we have to mention the settings of the pipeline, options related to logging, details of the location of configuration files, and other values of settings, we can use the logstash.yml file.
The settings we specify in this file often also support the usage of command-line flags. Note that if we specify any settings inside the logstash.yml configuration file and change the same setting on the command line by using the flag system, the file value gets overridden by the corresponding command-line flag value.
The configuration file of logstash.yml is written in the format language of YAML, and the location of this file changes as per the platform the user is using. The value of settings mentioned inside the file can be specified in either flat keys or hierarchical format. Let us consider a sample example of how we can specify settings in flat keys format –
Pipeline.batch.delay :65
Pipeline.batch.size: 100
While the same values in hierarchical format can be specified as –
Pipeline:
Batch:
Delay: 65
Size: 100
Interpolation of the environment variables in bash style is also supported by logstash.yml. Along with that, the support for the Keystore secrets inside the values of settings is also supported by logstash, where the specification looks somewhat as shown below –
Pipeline:
Batch:
Delay: $ {BATCH_DELAY:65}
Size: ${BATCH_SIZE}
Path:
Queue: “/c/users/educba/${QUEUE_DIR:queue}”
Node:
Name: “node_ ${LS_NAME_OF_NODE}”
The notation used above of $NAME_OF_VARIABLE: value set to be by default is supported by logstash.
We can even go for the specification of the model inside the configuration settings file of logstash.yml, where the format that is followed should be as shown below –
Modules:
-name: EDUCBA_MODEL1
Var.PLUGIN_TYPE1.SAMPLE_PLUGIN1.SAMPLE_KEY1: SAMPLE_VALUE
Var.PLUGIN_TYPE2.SAMPLE_PLUGIN2.SAMPLE_KEY2: SAMPLE_VALUE
Var.PLUGIN_TYPE3.SAMPLE_PLUGIN3.SAMPLE_KEY3: SAMPLE_VALUE
Var.PLUGIN_TYPE3.SAMPLE_PLUGIN4.SAMPLE_KEY2: SAMPLE_VALUE
Var.PLUGIN_TYPE4.SAMPLE_PLUGIN5.SAMPLE_KEY4: SAMPLE_VALUE
-name: EDUCBA_MODEL2
Var.PLUGIN_TYPE1.SAMPLE_PLUGIN1.SAMPLE_KEY1: SAMPLE_VALUE
Var.PLUGIN_TYPE2.SAMPLE_PLUGIN1.SAMPLE_KEY2: SAMPLE_VALUE
There will be ignorance of the values specified inside the logstash.yml file for defining the modules if the usage of –modules is a command line flag for modules.
logstash.yml.
There are various settings present inside the logstash.yml file that we can set. Some of them are as mentioned in the below table –
| Settings name | The value set by default | Description |
| Path.data | LOGSTASH_HOME/data | This directory is used by the plugins and the logstash itself for all the persistent requirements. |
| Node.name | The hostname of the machine | It is used for assigning the descriptive name to the node. |
| Pipeline.java_execution | Set to true | Used to specify whether to use or not the java execution engine. |
| Pipeline.id | Main | It is the ID that is an identifier set to the pipeline. |
| Pipeline.batch.size | 125 | It specifies that the maximum amount of events will be collected by an individual worker thread before executing the output and filter. Larger the size of the batch is the efficiency but note that it also comes along with the overhead for the memory requirement. |
| Pipeline.workers | It is set to the value cores count of CPU cores present for the host. | This is the count of workers working in parallel and going through the filters and the output stage executions. |
| Pipeline.batch.delay | The default value is 50 | The number of milliseconds to wait while pipeline even batches creation for every event before the dispatch of the batch to the workers. |
| Pipeline.plugins_classloaders | False | Whether to load the plugins of java to independently running class loaders for the segregation of the dependency or not. |
| Pipeline.unsafe_shutdown | False. Refuses to exit if any event is inflight. | Whether to force the logstash to close and exit while the shutdown is performed even though some of the inflight events are present inside the memory of the system or not. |
| Path.config | The default value is set as per the platform being used. Refer to this link for more details. | For the main pipeline, the path to navigate for the configuration of logstash is set in this setting. |
| Config.debug | False | Whether to display the debug notifications for the completely compiled configurations or not. |
| Modules | Nothing (None) value | Modules are used and described in the structure format of YAML in a nested way when configured in the logstash.yml file. |
Many other settings can still be configured and specified in the logstash.yml file. For a complete list, refer to this link.
Configure logstash.yml
There are two files for the configuration of logstash, which include the settings file and the pipeline configuration files used for the specification of execution and startup-related options that control logstash execution and help define the processing pipeline of logstash respectively.
Logstash.yml is one of the settings files defined in the installation of logstash and can be configured simply by specifying the values of various settings required in the file or by using command line flags.
To configure logstash, a config file needs to be created, which will contain the details about all the plugins that will be required and the details of settings regarding each of the specified plugins. We can simplify the config file by specifying the input and output inside. We can define the standard input-output of the customized ones from the elasticsearch and host value specification. Further, you can run it by executing the command of
bin/logstash -f logstash-simple.conf
where -f is for the configuration file that results in the following output –
Conclusion
Logstash.yml is used to specify flags of settings for the configuration of logstash, which defines the execution of logstash. We can specify and set many settings in this file discussed priorly.
Recommended Articles
This is a guide to logstash.yml. Here we discuss What logstash.yml is, logstash.yml, Configure logstash.yml, and Conclusion. You may also look at the following articles to learn more –
