Logstash Tunings for Large Config (1000 Conditional Statements)

Elastic Stack Logstash
1

Hi All,

Are there any config tunings other than heap size that can be modified to allow logstash to load a large config file ( on the order of 1000 conditional statements )?

Having anything more than about 200 conditional statements causes CPU spikes and the pipeline never starts up. The last thing seen in the logs is...
[INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

I've tried upping -Xms and -Xmx to 2g, 4g, and 8g without any success. I'm attempting to load this on a 8cpu, 16GB RAM CentOS7 VM.

Example Config:

2

Why don't you create a combined field based on the two you are analysing and then use this with the translate filter to look up the correct value?

EDIT: Got a bit further down and saw that the fields change a bit. It should however be possible to shrink this a lot by using some conditionals together with a few translate lookups.

3

Hi Christian, Thanks for the reply. I realize that some of the specific conditional statements here can be collapsed and optimized but I'm really interested in seeing if there are tunings that exist that could allow the config to be loaded as it is.

I'm trying to determine if logstash could be used to modify arbitrary events by evaluating them against large sets of conditional statements where each conditional statement could be made up of complex expressions.

4

I don't know if there is anything to optimise for that type of configuration, but it looks very inefficient. For that kind of complexity you might be better off creating a custom plugin.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.