Updated March 16, 2023
In+troduction to Logstash Pipeline Configuration
Logstash pipeline configuration is the setting about the details of each pipeline we will have in logstash in the file named logstash.yml. In this article, we will focus on logstash pipeline configuration and study it thoroughly, considering its subpoints, including overviews, logstash pipeline configuration, logstash pipeline configuration file, examples, and a Conclusion about the same.
logstash pipeline configuration overviews
There are various settings inside the logstash.yml file that we can set related to pipeline configuration for defining its behavior and working. Some of them are as mentioned in the below table –
| Pipeline settings name | The value set by default | Description |
| Pipeline.java_execution | Set to true | Used to specify whether to use or not the java execution engine. |
| Pipeline.id | Main | It is the ID that is an identifier set to the pipeline. |
| Pipeline.batch.size | 125 | It specifies that before going for the execution of output and filter, the maximum amount of events as that will be collected by an individual worker thread. The larger the batch size, the more the efficiency, but note that it also comes along with the overhead for the memory requirement. |
| Pipeline.workers | It is set to the value cores count of CPU cores present for the host. | This is the count of workers working in parallel and going through the filters and the output stage executions. |
| Pipeline.batch.delay | The default value is 50 | The number of milliseconds to wait while pipeline even batches creation for every event before the dispatch of the batch to the workers. |
| Pipeline.plugins_classloaders | False | Whether to load the plugins of java to independently running class loaders for the segregation of the dependency or not. |
| Pipeline.unsafe_shutdown | False. Refuses to exit if any event is in flight. | Whether to force the logstash to close and exit while the shutdown is performed even though some of the events of inflight are present inside the memory of the system or not. |
| Path.config | The default value is set as per the platform being used. Refer to this link for more details. | For the main pipeline, the path to navigate for the configuration of logstash is set in this setting. |
There are still many other settings that can be configured and specified in the logstash.yml file other than the ones related to the pipeline. For a complete list, refer to this link.
logstash pipeline configuration
We can have a single pipeline or multiple in our logstash, so we need to configure them accordingly. The process for setting the configurations for the logstash is as mentioned below –
- Open the configuration file of logstash named logstash.yml that is by default located in path etc/logstash. If not, you can find it where you have installed logstash. Start editing it.
- You will have to define the id and the path for all the configuration directories where you might make a logstash run.config property for your pipelines. For example, in the case of the single pipeline for sample purposes, we can specify the following details –
Pipeline.id : sample-educba-pipeline
Path.config: “/Users/Program Files/logstah/sample-educba-pipeline/*.conf”
- You will now need to check how you have installed logstash and restart or start logstash. We have used systemctl for installation and hence can use the below command to start logstash –
Sudo systemctl restart logstsh
Execution of the above command gives the following output –
logstash pipeline configuration file
Logstash.yml is a configuration settings file that helps maintain control over the execution of logstash. First, we can try to understand the usage and purpose of the logstash.yml configuration settings file by considering a small example. Then, when we have to mention the settings of the pipeline, options related to logging, details of the location of configuration files, and other values of settings, we can use the logstash.yml file.
The configuration file of logstash.yml is written in the format language of YAML, and the location of this file changes as per the platform the user is using. The value of settings mentioned inside the file can be specified in either flat keys or hierarchical format. Let us consider a sample example of how we can specify settings in flat keys format –
Pipeline.batch.delay :65
Pipeline.batch.size: 100
While the same values in hierarchical format can be specified as –
Pipeline:
Batch:
Delay: 65
Size: 100
Interpolation of the environment variables in bash style is also supported by logstash.yml. Along with that, the support for the Keystore secrets inside the values of settings is also supported by logstash, where the specification looks somewhat as shown below –
Pipeline:
Batch:
Delay: $ {BATCH_DELAY:65}
Size: ${BATCH_SIZE}
Path:
Queue: “/c/users/educba/${QUEUE_DIR:queue}”
Node:
Name: “node_ ${LS_NAME_OF_NODE}”
The notation used above of $NAME_OF_VARIABLE: value set to be by default is supported by logstash.
We can even go for the specification of the model inside the configuration settings file of logstash.yml, where the format that is followed should be as shown below –
Modules:
-name: EDUCBA_MODEL1
Var.PLUGIN_TYPE1.SAMPLE_PLUGIN1.SAMPLE_KEY1: SAMPLE_VALUE
Var.PLUGIN_TYPE2.SAMPLE_PLUGIN2.SAMPLE_KEY2: SAMPLE_VALUE
Var.PLUGIN_TYPE3.SAMPLE_PLUGIN3.SAMPLE_KEY3: SAMPLE_VALUE
Var.PLUGIN_TYPE3.SAMPLE_PLUGIN4.SAMPLE_KEY2: SAMPLE_VALUE
Var.PLUGIN_TYPE4.SAMPLE_PLUGIN5.SAMPLE_KEY4: SAMPLE_VALUE
-name: EDUCBA_MODEL2
Var.PLUGIN_TYPE1.SAMPLE_PLUGIN1.SAMPLE_KEY1: SAMPLE_VALUE
Var.PLUGIN_TYPE2.SAMPLE_PLUGIN1.SAMPLE_KEY2: SAMPLE_VALUE
There will be ignorance of the values specified inside the logstash.yml file for defining the modules if the usage of –modules is the command line flag for modules.
logstash pipeline configuration examples
There are two files for the configuration of logstash, which include the settings file and the pipeline configuration files used for the specification of execution and startup-related options that control logstash execution and help define the processing pipeline of logstash respectively.
Logstash.yml is one of the settings files defined in the installation of logstash and can be configured simply by specifying the values of various settings that are required in the file or by using command line flags.
To configure logstash, a config file needs to be created, which will contain the details about all the plugins that will be required and the details of settings regarding each of the specified plugins. We can create the config file simply by specifying the input and output inside which we can define the standard input output of the customized ones from the elasticsearch and host value specification. Further, you can run it by executing the command of
bin/logstash -f logstash-simple.conf
where -f is for the configuration file that results in the following output –
Conclusion
Logstash pipeline configuration can be set either for a single pipeline or have multiple pipelines in a file named logstash.yml that is located at /etc/logstash but default or in the folder where you have installed logstash. As mentioned in the table, we can set many configuration settings besides id and path.
Recommended Articles
This is a guide to Logstash Pipeline Configuration. Here we discuss the various settings present inside the logstash.yml file that we can set related to pipeline configuration. You may also look at the following articles to learn more –
