Hi,
So we have build a demo POC application using ELK stack for an enterprise system. After its success, we are now in a phase to develop the actual product itself. So we are trying to follow as much as good design approaches as possible.
For Logstash part, we have many logstash.conf file with a lot of grok patterns in it like this.
#grok for scenario 1
grok
{
match => [ "message", "%{LOGLEVEL:logLevel} : %{NOTSPACE:data} e.t.c "]
add_field =>
{
"status" => "tag 1"
}
}
#grok for scenario 2
grok
{
match => [ "message", "%{LOGLEVEL:logLevel} : %{NOTSPACE:data} e.t.c "]
add_field =>
{
"status" => "tag 2"
}
}
#grok for scenario 3
grok
{
match => [ "message", "%{LOGLEVEL:logLevel} : %{NOTSPACE:data} e.t.c "]
add_field =>
{
"status" => "tag 3"
}
}
As we move on and on, the patterns would increase and the other bits related to those as well. Now what we are thinking of different approaches.
Approach 1: Make multiple logstash file and run multiple logstash instances. - Fair enough - But what if those individual files become bigger and bigger.
Approach 2: Break one logstash file into multiples and include all those into one main Logstash file as we do in programming (header files) - Achievable ???
Approach 3: Externalize the grok patterns and some other bits. What i mean to say is that all the patterns that it have to match should come from a property file.
How can we achieve approach 3. What ideally we need is some one to share his experience on how should we design logstash part for production ready.
PS: We have studied the models (filebeat + logstash e.t.c) but we are more concerned about how to make the logstash script itself more configurable and neat.
Regards.