Many Conditional Checks in Logstash Conf

[sjabiulla (JB)]

Hi,

Will it be a performance impact If I have many conditional statements in Logstash config ?

My idea was to create a single Logstash setup whose input is FileBeat, WinLogBeat and HeartBeat. Based on the beat type and source log I am planning to push to different indexes in ElasticSearch. My have now finished a small POC and I have a bunch of if-else conditions in my conf file. So Wanted to check, will it impact the performance of Logstash ?

[pastechecker (Charlie)]

Why do you want to keep all in one config file instead of running separate ones per beat family product?
Input is not multi-threaded, only filter processing gets distributed over multiple threads, so that will be the limitation here. If you separate this you will get performance boost.

From my experience I did not see any noticeable performance impact with simple if conditions.
(My EPS was 3000-4000).

[sjabiulla (JB)]

Thanks for the reply.

@pastechecker

Why do you want to keep all in one config file instead of running separate ones per beat family product?

From my understanding, If we have multiple config files ( May be one per each beat family ), it just increases readability. But logstash concatenates and runs as a single config . So when logstash runs all these config files, it's going to have same conditional operators irrespective of single huge config file or multiple config files !!

@pastechecker

Input is not multi-threaded, only filter processing gets distributed over multiple threads, so that will be the limitation here. If you separate this you will get performance boost.

Could you please elaborate it a bit more. I didn't get this part.

[pastechecker (Charlie)]

When the pipeline starts Logstash assigns workers to the filter and the output processing stage.
Number of workers by default assumes the number of the host’s CPU cores. I think that you process input one by one and then spread across multiple threads when you pass the data to the filter section.

Have a look also on the: https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html

To determine the performance impact on your machine you can use the stats new API. https://www.elastic.co/guide/en/logstash/current/node-stats-api.html

[Christian_Dahlqvist]

@sjabiulla

From my understanding, If we have multiple config files ( May be one per each beat family ), it just increases readability. But logstash concatenates and runs as a single config . So when logstash runs all these config files, it's going to have same conditional operators irrespective of single huge config file or multiple config files !!

Not if you use the multiple pipelines feature and pipeline to pipeline communication.

@sjabiulla

Could you please elaborate it a bit more. I didn't get this part.

Having separate pipelines per input will ensure they run on separate threads and do not block eachother.

[sjabiulla (JB)]

@Christian_Dahlqvist

Not if you use the multiple pipelines feature

Thanks for the info @Christian_Dahlqvist. I'm now using multiple pipelines feature. But If I have same input.beats.port in two config files, I was getting "Address Already Bind" error while starting logstash. When I used different ports, it worked. Is this expected ?

[Christian_Dahlqvist]

Yes, you can only have one process listen to a single port, so all inputs need to use different ports. You could also create a separate pipeline that contains the input plugin and the sends data to different pipelines based on fields in the data. Each pipeline does not need to contain all types of plugins.