Best practice to handle logstash configuratioon

Elastic Stack Logstash
1

Hello All,

My logstash configuration have almost 1500+ lines of code where i am taking input from beats and applying filters for extracting fields of various log types and output to kafka. Is there any way to handle huge configuration files or any best practices for logstash parsers?

Thanks in advance.

2

I'd split that file into multiple files, but apart from that it's hard to give advice without seeing what it looks like.

I'd also strongly suggest using my tool Logstash Filter Verifier to test your filter configurations.

3

Does this means splitting 1 large conf to many confs and run multiple logstash instances?

This is awsome i will try this tool.

Thanks in advance.

4

Does this means splitting 1 large conf to many confs and run multiple logstash instances?

That's one possibility, but I was thinking of splitting into multiple files and running a single instance.

5

Heres the filter part of my conf. Please suggest on this.

Thanks in advance.

6

I obviously haven't studied all 2000 lines in any detail, but a few things stand out:

  • You're overusing DATA and GREEDYDATA in your grok expression. That's very inefficient and can lead to incorrect matches.
  • Having conditionals that inspect the exact source filename (if [source] == "/archives/log/10.90.250.151.log" etc) is probably not a great idea.
7

can u please suggest any best practices for maintaining logstash parser confs where we have custom log patterns (other than usual standard log patterns provided by vendors).

8

I don't think I have any particular suggestions. In what way is your current situation unmanageable?

9

with number of conf lines increased, we were facing latency for logs getting parsed. How to efficiently handle in this scenario? By writing multiple confs will improve performance? If so how to do this?

10

Have you looked into using the Logstash monitoring API to see which filters are adding the most time to the processing?

But as I said, reduce the amount of DATA and GREEDYDATA patterns. Don't spend time on other optimizations until your grok expressions have been cleaned up.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.