# Grok pattern for custom application log

**URL:** <https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864>\
**Category:** Logstash\
**Created:** [May 30, 2018, 11:36am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864 "2018-05-30T11:36:57Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![pkshara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkshara/32/48222_2.png) [@pkshara](https://discuss.elastic.co/u/pkshara)\
**Post date:** [May 30, 2018, 11:36am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/1 "2018-05-30T11:36:58Z")

</div>

Hi Team,

I have error log as mentioned below , can you suggest me the grok pattern for this,

20180209 00:00:08,696 ERROR WebContainer : 34989 dao.OrderServiceDAOImpl Acct\_Nr=724377 Imp\_Acct\_Nr= Exception in the method checkCampaignForPUPorg.springframework.dao.EmptyResultDataAccessException: Incorrect result size: expected 1, actual 0

I tried as  
%{TIMESTAMP\_ISO8601:timestamp}\*

but the output comes as timestamp:null.

can you please suggest solution for this

TIA

---

<div class="post-metadata">

**Author:** ![arkady\_renko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arkady_renko/32/30160_2.png) [@arkady\_renko](https://discuss.elastic.co/u/arkady_renko)\
**Post date:** [May 30, 2018, 12:12pm UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/2 "2018-05-30T12:12:38Z")

</div>

Date field in the log doesn't match with TIMESTAMP\_ISO8601 pattern.

> <https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns>

---

<div class="post-metadata">

**Author:** ![pkshara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkshara/32/48222_2.png) [@pkshara](https://discuss.elastic.co/u/pkshara)\
**Post date:** [June 1, 2018, 11:00am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/3 "2018-06-01T11:00:08Z")

</div>

Hi Sezgin,

Thanks for the info.

this is my error log:  
20180209 00:00:08,696 ERROR WebContainer : 34989 dao.OrderServiceDAOImpl Acct\_Nr=724377 Imp\_Acct\_Nr= Exception in the method checkCampaignForPUPorg.springframework.dao.EmptyResultDataAccessException: Incorrect result size: expected 1, actual 0

I have used grok pattern as  
%{YEAR}%{MONTHNUM}%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{WORD:file} : %{NUMBER:linenumber}%{SPACE} %{WORD}.%{WORD}_%{SPACE}%{DATA:account}=%{NUMBER:acct\_nr}_%{SPACE}%{DATA:errordet}=%{GREEDYDATA:log}

I am getting the output each and every filter which i mentioned above as separate,but i want to get %{YEAR}%{MONTHNUM}%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} this complete details in single variable, may be assigned to time or timestamp like that.

can you suggest me how to add that custom pattern in logstash config or ./pattern file. I have no idea about ./pattern file concept.Requesting you to provide some solution.

TIA

---

<div class="post-metadata">

**Author:** ![arkady\_renko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arkady_renko/32/30160_2.png) [@arkady\_renko](https://discuss.elastic.co/u/arkady_renko)\
**Post date:** [June 1, 2018, 12:22pm UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/4 "2018-06-01T12:22:58Z")

</div>

Hi,

you can check link below creating custom patterns.

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#\_custom\_patterns](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_custom_patterns)

add line to file and save file any folder. set patterns\_dir value to folder's path.

`MYDATEPATTERN %{YEAR}%{MONTHNUM}%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}`

sample grok filter with custom pattern.

```
filter {
grok {
patterns_dir => ["C:\development\elk\logstash\custom_pattern"]
match => { "message" => "^%{MYDATEPATTERN:my_date}" }
}
}
```

---

<div class="post-metadata">

**Author:** ![pkshara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkshara/32/48222_2.png) [@pkshara](https://discuss.elastic.co/u/pkshara)\
**Post date:** [June 4, 2018, 6:18am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/5 "2018-06-04T06:18:15Z")

</div>

Hi,

I tried as you mentioned,but i am facing grokpattern error, its not taking up custom patterns ,

Code:  
input {  
file {  
path =\> "D:\logstash-6.2.4\bin\webe.log"  
start\_position =\> "beginning"  
}  
}

filter {  
grok {  
patterns\_dir =\> ["D:\logstash-6.2.4\bin\custom\_pattern"]  
match =\> { "message" =\> "%{DATE:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{LINENUMBER:file}%{SPACE} %{WORD}.%{WORD}_%{SPACE}%{DATA:account}=%{NUMBER:acct\_nr}_%{SPACE}%{DATA:errordet}=%{GREEDYDATA:log}" }  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "logs\_parameter\_index"

}  
stdout { codec =\> rubydebug }  
}

Error:  
[2018-06-04T11:36:17,780][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"#\<LogStash::FilterDelegator:0x39d6bae9 @metric\_events\_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric\_events\_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric\_events\_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration\_in\_millis value:0, @id="9f6b1fb0721974fe26ef58f6626709fdb2832c6b5f081a6fcac6ec7e8b0e329b", @klass=LogStash::Filters::Grok, @metric\_events=#\<LogStash::Instrument::NamespacedMetric:0x2c183e59 @metric=#\<LogStash::Instrument::Metric:0x23a3ad60 @collector=#\<LogStash::Instrument::Collector:0x148ca8e7 @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x1881596d @store=#\<Concurrent:🗺0x00000000000fb4 entries=3 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0x3a4a6fa6, @fast\_lookup=#\<Concurrent:🗺0x00000000000fb8 entries=63 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :filters, :"9f6b1fb0721974fe26ef58f6626709fdb2832c6b5f081a6fcac6ec7e8b0e329b", :events]\>, @filter=\<LogStash::Filters::Grok patterns\_dir=\>["D:\\avoN\\logstash-6.2.4\\bin\\custom\_pattern"], match=\>{"message"=\>"^%{AVONDATE:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{LINENUMBER:file}%{SPACE} %{WORD}.%{WORD}_%{SPACE}%{DATA:account}=%{NUMBER:acct\_nr}_%{SPACE}%{DATA:errordet}=%{GREEDYDATA:log}"}, id=\>"9f6b1fb0721974fe26ef58f6626709fdb2832c6b5f081a6fcac6ec7e8b0e329b", enable\_metric=\>true, periodic\_flush=\>false, patterns\_files\_glob=\>"\*", break\_on\_match=\>true, named\_captures\_only=\>true, keep\_empty\_captures=\>false, tag\_on\_failure=\>["\_grokparsefailure"], timeout\_millis=\>30000, tag\_on\_timeout=\>"\_groktimeout"\>\>", :error=\>"pattern %{DATE:timestamp} not defined", :thread=\>"#\<Thread:0x6f6570b5 run\>"}  
[2018-06-04T11:36:17,795][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<Grok::PatternError: pattern %{AVONDATE:timestamp} not defined\>, :backtrace=\>["D:/avoN/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.4/lib/grok-pure.rb:123:in `block in compile'", "org/jruby/RubyKernel.java:1292:in`loop'", "D:/avoN/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.4/lib/grok-pure.rb:93:in `compile'", "D:/avoN/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:281:in`block in register'", "org/jruby/RubyArray.java:1734:in `each'", "D:/avoN/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:275:in`block in register'", "org/jruby/RubyHash.java:1343:in `each'", "D:/avoN/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:270:in`register'", "D:/avoN/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:342:in `register_plugin'", "D:/avoN/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:353:in`block in register\_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "D:/avoN/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:353:in`register\_plugins'", "D:/avoN/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:731:in `maybe_setup_out_plugins'", "D:/avoN/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:363:in`start\_workers'", "D:/avoN/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:290:in `run'", "D:/avoN/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:250:in`block in start'"], :thread=\>"#\<Thread:0x6f6570b5 run\>"}  
[2018-06-04T11:36:17,842][ERROR][logstash.agent] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: LogStash::PipelineAction::Create/pipeline\_id:main, action\_result: false", :backtrace=\>nil}

---

<div class="post-metadata">

**Author:** ![arkady\_renko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arkady_renko/32/30160_2.png) [@arkady\_renko](https://discuss.elastic.co/u/arkady_renko)\
**Post date:** [June 4, 2018, 8:24am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/6 "2018-06-04T08:24:20Z")

</div>

Hi,

I added your grok patterns to my pattern file and they are compiled without problem.

patterns\_dir sets directory containing files for custom pattern, not absolute path of file.

```
[2018-06-04T11:08:46,070][DEBUG][logstash.filters.grok] Adding pattern {"SQUID3"=>"%{NUMBER:timestamp}\\s+%{NUMBER:duration}\\s%{IP:client_address}\\s%{WORD:cache_result}/%{POSINT:status_code}\\s%{NUMBER:bytes}\\s%{WORD:request_method}\\s%{NOTSPACE:url}\\s(%{NOTSPACE:user}|-)\\s%{WORD:hierarchy_code}/%{IPORHOST:server}\\s%{NOTSPACE:content_type}"}
[2018-06-04T11:08:46,075][DEBUG][logstash.filters.grok] Adding pattern {"MYDATEPATTERN"=>"%{YEAR}%{MONTHNUM}%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}"}
[2018-06-04T11:08:46,075][DEBUG][logstash.filters.grok] Adding pattern {"AVONDATE"=>"%{YEAR}%{MONTHNUM}%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}"}
[2018-06-04T11:08:46,076][DEBUG][logstash.filters.grok] Adding pattern {"LINENUMBER"=>"%{WORD} : %{NUMBER}"}
[2018-06-04T11:08:46,076][DEBUG][logstash.filters.grok] Adding pattern {"FILENAME"=>"%{WORD}.%{WORD}"}
[2018-06-04T11:08:46,076][DEBUG][logstash.filters.grok] Adding pattern {"ACCOUNT"=>"%{DATA}=%{NUMBER}"}
```

---

<div class="post-metadata">

**Author:** ![pkshara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkshara/32/48222_2.png) [@pkshara](https://discuss.elastic.co/u/pkshara)\
**Post date:** [June 4, 2018, 9:21am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/7 "2018-06-04T09:21:25Z")

</div>

Hi,

I changed the pattern file to directory path, i am not facing any issue now, but i dont see any logs running in the logstash,

[2018-06-04T14:39:30,095][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]}  
[2018-06-04T14:39:31,340][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\<Thread:0x641ecfe0 run\>"}  
[2018-06-04T14:39:31,517][INFO][logstash.agent] Pipelines running {:count=\>1, :pipelines=\>["main"]}

after this line its not going further, so according to you is it working fine, because whenevr i run logstash file in the screen i see my logs going to elasticsearch according to the parameters defined, can you suggest for the same.

---

<div class="post-metadata">

**Author:** ![arkady\_renko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arkady_renko/32/30160_2.png) [@arkady\_renko](https://discuss.elastic.co/u/arkady_renko)\
**Post date:** [June 4, 2018, 11:08am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/8 "2018-06-04T11:08:29Z")

</div>

Everyhing looks OK. If your logs stored in ES index, your pipeline is working fine.  
To see detailed logstash logs change log level to debug in logstash.yml file.

---

<div class="post-metadata">

**Author:** ![pkshara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkshara/32/48222_2.png) [@pkshara](https://discuss.elastic.co/u/pkshara)\
**Post date:** [June 26, 2018, 11:03am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/9 "2018-06-26T11:03:20Z")

</div>

It doesnt create index in elasticsearch as well. What can be the reason for logstash to get stuck after  
[2018-06-04T14:39:31,517][INFO][logstash.agent] Pipelines running {:count=\>1, :pipelines=\>["main"]}

---

<div class="post-metadata">

**Author:** ![pkshara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkshara/32/48222_2.png) [@pkshara](https://discuss.elastic.co/u/pkshara)\
**Post date:** [June 26, 2018, 11:25am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/10 "2018-06-26T11:25:09Z")

</div>

Solved.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 24, 2018, 11:25am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864/11 "2018-07-24T11:25:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.