Externalize Grok Patterns for logstash


(Omair Khalid) #1

Hi,
So we have build a demo POC application using ELK stack for an enterprise system. After its success, we are now in a phase to develop the actual product itself. So we are trying to follow as much as good design approaches as possible.

For Logstash part, we have many logstash.conf file with a lot of grok patterns in it like this.

 #grok for scenario 1
  grok 
  {
    match => [ "message", "%{LOGLEVEL:logLevel} : %{NOTSPACE:data} e.t.c "]
    add_field => 
    { 
      "status" => "tag 1"
    }
  }
 #grok for scenario 2
  grok 
  {
    match => [ "message", "%{LOGLEVEL:logLevel} : %{NOTSPACE:data} e.t.c "]
    add_field => 
    { 
      "status" => "tag 2"
    }
  }
 #grok for scenario 3
  grok 
  {
    match => [ "message", "%{LOGLEVEL:logLevel} : %{NOTSPACE:data} e.t.c "]
    add_field => 
    { 
      "status" => "tag 3"
    }
  }

As we move on and on, the patterns would increase and the other bits related to those as well. Now what we are thinking of different approaches.

Approach 1: Make multiple logstash file and run multiple logstash instances. - Fair enough - But what if those individual files become bigger and bigger.
Approach 2: Break one logstash file into multiples and include all those into one main Logstash file as we do in programming (header files) - Achievable ???
Approach 3: Externalize the grok patterns and some other bits. What i mean to say is that all the patterns that it have to match should come from a property file.

How can we achieve approach 3. What ideally we need is some one to share his experience on how should we design logstash part for production ready.

PS: We have studied the models (filebeat + logstash e.t.c) but we are more concerned about how to make the logstash script itself more configurable and neat.
Regards.


(Omair Khalid) #2

Hello,
So after posting the question, i came across something located inside installation directory of logstash.
\logstash-5.5.2\vendor\bundle\jruby\1.9\gems\logstash-patterns-core-4.1.1\patterns.
It seems like we can define our own custom patterns there. This way i think we can externalize the grok patterns from the logstash.conf file but is there another way.
and if we did follow the other approach, then do we have to re-compile logstash e.t.c
Regards.


(Mark Walkom) #3

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_custom_patterns should address some of these questions, but if there's something you don't see answered let us know!


(Magnus B├Ąck) #4

Approach 1: Make multiple logstash file and run multiple logstash instances. - Fair enough - But what if those individual files become bigger and bigger.

Why do you think you would have to run multiple instances? You can, but a desire to have multiple configuration files isn't a good reason for it.

Approach 2: Break one logstash file into multiples and include all those into one main Logstash file as we do in programming (header files) - Achievable ???

Put all files in a directory and point Logstash to that directory. The files will be read in alphabetical order. This is entirely equivalent to having a single large file that you've created by concatenating the original files.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.