# Externalize Grok Patterns for logstash

**URL:** <https://discuss.elastic.co/t/externalize-grok-patterns-for-logstash/119389>\
**Category:** Logstash\
**Created:** [February 11, 2018, 7:38pm UTC](https://discuss.elastic.co/t/externalize-grok-patterns-for-logstash/119389 "2018-02-11T19:38:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Omair\_Khalid](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/omair_khalid/32/23533_2.png) [@Omair\_Khalid](https://discuss.elastic.co/u/Omair_Khalid)\
**Post date:** [February 11, 2018, 7:38pm UTC](https://discuss.elastic.co/t/externalize-grok-patterns-for-logstash/119389/1 "2018-02-11T19:38:37Z")

</div>

Hi,  
So we have build a demo POC application using ELK stack for an enterprise system. After its success, we are now in a phase to develop the actual product itself. So we are trying to follow as much as good design approaches as possible.

For Logstash part, we have many logstash.conf file with a lot of grok patterns in it like this.

```
 #grok for scenario 1
  grok 
  {
    match => ["message", "%{LOGLEVEL:logLevel} : %{NOTSPACE:data} e.t.c "]
    add_field => 
    { 
      "status" => "tag 1"
    }
  }
 #grok for scenario 2
  grok 
  {
    match => ["message", "%{LOGLEVEL:logLevel} : %{NOTSPACE:data} e.t.c "]
    add_field => 
    { 
      "status" => "tag 2"
    }
  }
 #grok for scenario 3
  grok 
  {
    match => ["message", "%{LOGLEVEL:logLevel} : %{NOTSPACE:data} e.t.c "]
    add_field => 
    { 
      "status" => "tag 3"
    }
  }

```

As we move on and on, the patterns would increase and the other bits related to those as well. Now what we are thinking of different approaches.

Approach 1: Make multiple logstash file and run multiple logstash instances. - Fair enough - But what if those individual files become bigger and bigger.  
Approach 2: Break one logstash file into multiples and include all those into one main Logstash file as we do in programming (header files) - Achievable ???  
Approach 3: Externalize the grok patterns and some other bits. What i mean to say is that all the patterns that it have to match should come from a property file.

How can we achieve approach 3. What ideally we need is some one to share his experience on how should we design logstash part for production ready.

PS: We have studied the models (filebeat + logstash e.t.c) but we are more concerned about how to make the logstash script itself more configurable and neat.  
Regards.

---

<div class="post-metadata">

**Author:** ![Omair\_Khalid](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/omair_khalid/32/23533_2.png) [@Omair\_Khalid](https://discuss.elastic.co/u/Omair_Khalid)\
**Post date:** [February 11, 2018, 9:02pm UTC](https://discuss.elastic.co/t/externalize-grok-patterns-for-logstash/119389/2 "2018-02-11T21:02:20Z")

</div>

Hello,  
So after posting the question, i came across something located inside installation directory of logstash.  
\logstash-5.5.2\vendor\bundle\jruby\1.9\gems\logstash-patterns-core-4.1.1\patterns.  
It seems like we can define our own custom patterns there. This way i think we can externalize the grok patterns from the logstash.conf file but is there another way.  
and if we did follow the other approach, then do we have to re-compile logstash e.t.c  
Regards.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 12, 2018, 12:17am UTC](https://discuss.elastic.co/t/externalize-grok-patterns-for-logstash/119389/3 "2018-02-12T00:17:14Z")

</div>

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#\_custom\_patterns](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_custom_patterns) should address some of these questions, but if there's something you don't see answered let us know!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 12, 2018, 8:41pm UTC](https://discuss.elastic.co/t/externalize-grok-patterns-for-logstash/119389/4 "2018-02-12T20:41:48Z")

</div>

> Approach 1: Make multiple logstash file and run multiple logstash instances. - Fair enough - But what if those individual files become bigger and bigger.

Why do you think you would have to run multiple instances? You can, but a desire to have multiple configuration files isn't a good reason for it.

> Approach 2: Break one logstash file into multiples and include all those into one main Logstash file as we do in programming (header files) - Achievable ???

Put all files in a directory and point Logstash to that directory. The files will be read in alphabetical order. This is entirely equivalent to having a single large file that you've created by concatenating the original files.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 12, 2018, 8:42pm UTC](https://discuss.elastic.co/t/externalize-grok-patterns-for-logstash/119389/5 "2018-03-12T20:42:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.