Large Logstash Config File

(Priyaranjan Mudliar) #1

What is the file size limit of the logstash config file ? Actually I have many grok patterns to fill in it and it might big and cumbersome to handle. Please guide me how to overcome this. Will it slow down the pipeline performance of logstash? I am using ELK for centralized logging of logs.

Please Help

@magnusbaeck Can you help me on this one?

(Christian Dahlqvist) #3

In recent versions you can split out processing of different data types into multiple pipelines. Once you have identified the data type you can send the data to the appropriate pipeline. This generally makes it easier to manage larger configs.

Lots of unnecessary grok processing can be very inefficient and slow down the pipeline, so I would also recommend reading this blog post about grok optimisation. If your data is delimited, it may be faster and more efficient to use the dissect filter where applicable.