# How to make a custom grok filter

**URL:** <https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765>\
**Category:** Logstash\
**Created:** [February 21, 2018, 5:56am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765 "2018-02-21T05:56:42Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [February 21, 2018, 5:56am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/1 "2018-02-21T05:56:43Z")

</div>

Hello  
I have to create a custom grok filter for my logs so please let me know the procedure to do that as in the pattern which I want where it should be saved the extension of the file in which i write my custom pattern also how to use pattern\_dir .

Regards  
Gaurav

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 21, 2018, 6:05am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/2 "2018-02-21T06:05:05Z")

</div>

Have you read the grok filter documentation's fairly long section about custom patterns? If yes I'd expect you to be able to ask a slightly more specific question.

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [February 21, 2018, 6:28am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/3 "2018-02-21T06:28:06Z")

</div>

Just for an example for my question  
let this be my custom pattern  
CUST\_DATE %{MONTH} %{MONTHDAY} %{TIME}  
which I saved in the bin folder of logstash with the name patterns.txt

Now this is my config file  
input{  
stdin{  
}  
}  
filter{  
grok{  
patterns\_dir =\> ["./patterns"]  
match =\> {"message" =\> "%{CUST\_DATE:date}"}  
}  
}  
output  
{  
stdout{  
}  
}  
this again I have saved in the bin folder of logstash as filename.conf

but when I try to execute this I am getting this error

[2018-02-21T11:05:12,119][WARN][logstash.outputs.elasticsearch] You are using a deprecated config s  
etting "document\_type" set in elasticsearch. Deprecated settings will continue to work, but are sche  
duled for removal from logstash in the future. Document types are being deprecated in Elasticsearch  
6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this  
, please visit the #logstash channel on freenode irc. {:name=\>"document\_type", :plugin=\>\<LogStash:😮  
utputs::ElasticSearch hosts=\>[[http://localhost:9200](http://localhost:9200)], bulk\_path=\>"/\_xpack/monitoring/\_bulk?system\_id  
=logstash&system\_api\_version=2&interval=1s", manage\_template=\>false, document\_type=\>"%{[@metadata][d  
ocument\_type]}", sniffing=\>false, id=\>"ad524e5a1a68d2ca7086e1144ec98005bcfc1ad3103a990fb9bbf21aa44aa  
140", enable\_metric=\>true, codec=\>\<LogStash::Codecs::Plain id=\>"plain\_f8baa5fb-f149-4087-a45d-a88d57  
246b18", enable\_metric=\>true, charset=\>"UTF-8"\>, workers=\>1, template\_name=\>"logstash", template\_ove  
rwrite=\>false, doc\_as\_upsert=\>false, script\_type=\>"inline", script\_lang=\>"painless", script\_var\_name  
=\>"event", scripted\_upsert=\>false, retry\_initial\_interval=\>2, retry\_max\_interval=\>64, retry\_on\_confl  
ict=\>1, action=\>"index", ssl\_certificate\_verification=\>true, sniffing\_delay=\>5, timeout=\>60, pool\_ma  
x=\>1000, pool\_max\_per\_route=\>100, resurrect\_delay=\>5, validate\_after\_inactivity=\>10000, http\_compres  
sion=\>false\>}

[2018-02-21T11:05:12,475][WARN][logstash.licensechecker.licensereader] Detected a 6.x and above clu  
ster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}  
[2018-02-21T11:05:12,503][INFO][logstash.pipeline] Pipeline started {"pipeline.id"=\>".monit  
oring-logstash"}  
[2018-02-21T11:05:12,901][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>  
"main", :plugin=\>"#\<LogStash::FilterDelegator:0x7c9c5dc0 @metric\_events\_out=org.jruby.proxy.org.logs  
tash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric\_events\_in=org.jruby  
.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @logger=#\<Log  
Stash::Logging::Logger:0x12b2a0c1 @logger=#Java::OrgApacheLoggingLog4jCore::Logger:0x54166931\>, @m  
etric\_events\_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name  
: duration\_in\_millis value:0, @id="b2db37c04c07ac307db2757ab270f769693c0efaba09a9a7751a09970f65ca9b  
", @klass=LogStash::Filters::Grok, @metric\_events=#\<LogStash::Instrument::NamespacedMetric:0xbd850f  
f @metric=#\<LogStash::Instrument::Metric:0xceb4ded @collector=#\<LogStash::Instrument::Collector:0x6b  
d3e2a0 @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x5dbfe853 @store=#\<Concurrent:  
🗺0x00000000000fb8 entries=3 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0x142904b3, @fa  
st\_lookup=#\<Concurrent:🗺0x00000000000fbc entries=59 default\_proc=nil\>\>\>\>, @namespace\_name=[:stat  
s, :pipelines, :main, :plugins, :filters, :b2db37c04c07ac307db2757ab270f769693c0efaba09a9a7751a09970  
f65ca9b, :events]\>, @filter=\<LogStash::Filters::Grok patterns\_dir=\>["./patterns"], match=\>{"messa  
ge"=\>"%{CUST\_DATE:date}"}, id=\>"b2db37c04c07ac307db2757ab270f769693c0efaba09a9a7751a09970f65ca9b  
", enable\_metric=\>true, periodic\_flush=\>false, patterns\_files\_glob=\>"\*", break\_on\_match=\>true, na  
med\_captures\_only=\>true, keep\_empty\_captures=\>false, tag\_on\_failure=\>["\_grokparsefailure"], timeou  
t\_millis=\>30000, tag\_on\_timeout=\>"\_groktimeout"\>\>", :error=\>"pattern %{CUST\_DATE:date} not defined  
", :thread=\>"#\<Thread:0x36c99bf3@C:/Users/gagarwal3/Downloads/logstash/logstash-core/lib/logstash/pi  
peline.rb:245 run\>"}  
[2018-02-21T11:05:12,904][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline  
\_id=\>"main", :exception=\>#\<Grok::PatternError: pattern %{CUST\_DATE:date} not defined\>, :backtrace=\>[  
"C:/Users/gagarwal3/Downloads/logstash/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.4/lib/grok-pure.  
rb:123:in `block in compile'", "org/jruby/RubyKernel.java:1292:in`loop'", "C:/Users/gagarwal3/Downl  
oads/logstash/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.4/lib/grok-pure.rb:93:in `compile'", "C:/ Users/gagarwal3/Downloads/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.1/lib/log stash/filters/grok.rb:286:in`block in register'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Us ers/gagarwal3/Downloads/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.1/lib/logst ash/filters/grok.rb:280:in`block in register'", "org/jruby/RubyHash.java:1343:in `each'", "C:/Users /gagarwal3/Downloads/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.1/lib/logstash /filters/grok.rb:275:in`register'", "C:/Users/gagarwal3/Downloads/logstash/logstash-core/lib/logsta  
sh/pipeline.rb:343:in `register_plugin'", "C:/Users/gagarwal3/Downloads/logstash/logstash-core/lib/l ogstash/pipeline.rb:354:in`block in register\_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Users/gagarwal3/Downloads/logstash/logstash-core/lib/logstash/pipeline.rb:354:in`register\_plugi  
ns'", "C:/Users/gagarwal3/Downloads/logstash/logstash-core/lib/logstash/pipeline.rb:744:in `maybe_se tup_out_plugins'", "C:/Users/gagarwal3/Downloads/logstash/logstash-core/lib/logstash/pipeline.rb:364 :in`start\_workers'", "C:/Users/gagarwal3/Downloads/logstash/logstash-core/lib/logstash/pipeline.rb:  
288:in `run'", "C:/Users/gagarwal3/Downloads/logstash/logstash-core/lib/logstash/pipeline.rb:248:in`block in start'"], :thread=\>"#\<Thread:0x36c99bf3@C:/Users/gagarwal3/Downloads/logstash/logstash-cor  
e/lib/logstash/pipeline.rb:245 run\>"}  
[2018-02-21T11:05:12,910][ERROR][logstash.agent] Failed to execute action {:id=\>:main, :a  
ction\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: LogStash:😛  
ipelineAction::Create/pipeline\_id:main, action\_result: false", :backtrace=\>nil}  
[2018-02-21T11:05:12,917][INFO][logstash.inputs.metrics] Monitoring License OK  
[2018-02-21T11:05:13,534][INFO][logstash.pipeline] Pipeline terminated {"pipeline.id"=\>".mo

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 21, 2018, 7:01am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/4 "2018-02-21T07:01:18Z")

</div>

> ```
> patterns_dir => ["./patterns"]
> 
> ```

But this isn't consistent with you placing patterns.txt in the Logstash bin direcotry.

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [February 21, 2018, 7:05am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/5 "2018-02-21T07:05:14Z")

</div>

that is where i m stuck can u please help me out how to get this fixed what should i do to make it consistent.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 21, 2018, 7:13am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/6 "2018-02-21T07:13:23Z")

</div>

The `patterns_dir` option should point to whatever directory where you've chosen to store patterns.txt.

As a side note, I recommend storing patterns.txt along with your Logstash configuration files (but not in the conf.d directory!) so there's no risk of losing it when you upgrade Logstash.

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [February 21, 2018, 7:24am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/7 "2018-02-21T07:24:33Z")

</div>

input{  
stdin{  
}  
}  
filter{  
grok{  
patterns\_dir =\> [".C:\Users\gagarwal3\Downloads\logstash\bin\patterns.txt"]  
match =\> {"message" =\> "%{CUST\_DATE:date}"}  
}  
}  
output  
{  
stdout{  
}  
}

Is this correct?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 21, 2018, 7:44am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/8 "2018-02-21T07:44:33Z")

</div>

No.

- Your `patterns_dir` value starts with a period.
- The `patterns_dir` option should point to whatever **directory** where you've chosen to store patterns.txt.

And again, the bin directory is a bad choice. Quoting the documentation:

> Note that Grok will read all files in the directory matching the patterns\_files\_glob and assume it’s a pattern file (including any tilde backup files).

So, create a directory where you only store pattern files and point `patterns_dir` to that directory.

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [February 21, 2018, 8:39am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/9 "2018-02-21T08:39:31Z")

</div>

I have created a new directory named "pat" where in I have saved the patterns file so can u please let me know if this config file is correct if not not can you mention the syntax for **patterns\_dir**  
input{  
stdin{  
}  
}  
filter{  
grok{  
patterns\_dir =\> ["C:\Users\gagarwal3\Downloads\logstash\pat\patterns.conf"]  
match =\> {"message" =\> "%{CUST\_DATE:date}"}  
}  
}  
output  
{  
stdout{  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 21, 2018, 8:57am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/10 "2018-02-21T08:57:56Z")

</div>

Is C:\Users\gagarwal3\Downloads\logstash\pat\patterns.conf a directory?

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [February 21, 2018, 9:12am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/11 "2018-02-21T09:12:09Z")

</div>

this is the exact path for my file where I have the custom pattern  
so basically under pat folder i have my file patterns.conf

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 21, 2018, 9:17am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/12 "2018-02-21T09:17:35Z")

</div>

Yes, and for the third time the `patterns_dir` option should point to whatever **directory** where you've chosen to store patterns.conf. Over and out.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 21, 2018, 9:17am UTC](https://discuss.elastic.co/t/how-to-make-a-custom-grok-filter/120765/13 "2018-03-21T09:17:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.