# Issue with filter configuration in logstash **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [June 12, 2020, 2:08pm UTC](https://discuss.elastic.co/t/issue-with-filter-configuration-in-logstash/236911 "2020-06-12T14:08:01Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![kittu9999](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kittu9999/32/72350_2.png) [@kittu9999](https://discuss.elastic.co/u/kittu9999)\ **Post date:** [June 12, 2020, 2:08pm UTC](https://discuss.elastic.co/t/issue-with-filter-configuration-in-logstash/236911/1 "2020-06-12T14:08:02Z")
Hello, we are having an issue with filter configuration in the logstash. we have upgraded the logstash form 5 to logstash 6.8.1 version. the filter configuration was working fine in logstash 5.4 version, but not in the logstash 6.8.1. if we remove filter configuration the logstash is working fine, but logstash is failing after adding the filter. Can you please help me here? ## logstash.conf: input { syslog { port =\> 5514 type =\> "syslog" } beats { port =\> 5044 } } filter { if [source] =~ "app1" { grok { tag\_on\_failure =\> ["\_grokdateparsefailure"] match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:logdate}.\*"} } date { match =\> ["logdate", "YYYY-MM-dd HH:mm:ss,SSS"] target =\> "logdateTime" timezone =\> "America/New\_York" remove\_field =\> ["logdate"] } ``` if [message] =~ "JSON:" { grok { patterns_dir => ["/appdata/config/logstash/patterns"] tag_on_failure => ["_jsonparsefailure"] break_on_match => true keep_empty_captures => false match => { "message" => ".*JSON:%{JSON:requestJSON}.*"} } json{ source => "requestJSON" target => "doc" remove_field => ["requestJSON"] } } } if [source] =~ "app2" { if [message] =~ "\[.*\]" { grok { patterns_dir => ["/appdata/config/logstash/patterns"] break_on_match => true tag_on_failure => ["_grokfixparsefailure"] match => { "message" => ".*\[%{GREEDYDATA:FIXMESSAGE}\]"} } kv { value_split => "=" field_split => "\001" source => "APPMESSAGE" } } grok { patterns_dir => ["/appdata/config/logstash/patterns"] break_on_match => true tag_on_failure => ["_grokdateparsefailure"] match => { "message" => "%{APPDATETIMESTAMP:logdate}.*"} } date { match => ["logdate", "YYYY/MM/dd HH:mm:ss:SSS"] target => "logdateTime" timezone => "America/New_York" remove_field => ["logdate"] remove_field => ["APPMESSAGE"] } ``` } } output { if [type] == "syslog" { elasticsearch { hosts =\> ["[http://localhost:9200](http://localhost:9200)"] user =\> "xxxxxx" password =\> "xxxxxxxx" index =\> "syslog-%{+YYYY.MM.dd}" ssl =\> false ssl\_certificate\_verification =\> false ilm\_enabled =\> false } } else { elasticsearch { hosts =\> ["[http://localhost:9200](http://localhost:9200)"] manage\_template =\> false user =\> "xxxxxxxxx" password =\> "xxxxxxxxxx" index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" ssl =\> false ssl\_certificate\_verification =\> false ilm\_enabled =\> false } } } ## Logstash Logs: 2020-06-11T10:50:20.586-04:00 [2020-06-11T14:50:20,586][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"#LogStash::FilterDelegator:0x10981ad5", :error=\>"pattern %{JSON:requestJSON} not defined", :thread=\>"#\"} 2020-06-11T10:50:20.740-04:00 [2020-06-11T14:50:20,739][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\, :backtrace=\>["/etc/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'", "org/jruby/RubyKernel.java:1425:in `loop'", "/etc/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in `compile'", "/etc/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:281:in `block in register'", "org/jruby/RubyArray.java:1792:in `each'", "/etc/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:275:in `block in register'", "org/jruby/RubyHash.java:1419:in `each'", "/etc/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:270:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in `register'", "/etc/logstash/logstash-core/lib/logstash/pipeline.rb:259:in `register\_plugin'", "/etc/logstash/logstash-core/lib/logstash/pipeline.rb:270:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "/etc/logstash/logstash-core/lib/logstash/pipeline.rb:270:in `register_plugins'", "/etc/logstash/logstash-core/lib/logstash/pipeline.rb:612:in `maybe\_setup\_out\_plugins'", "/etc/logstash/logstash-core/lib/logstash/pipeline.rb:280:in `start_workers'", "/etc/logstash/logstash-core/lib/logstash/pipeline.rb:217:in `run'", "/etc/logstash/logstash-core/lib/logstash/pipeline.rb:176:in `block in start'"], :thread=\>"#\"} 2020-06-11T10:50:20.756-04:00 [2020-06-11T14:50:20,756][ERROR][logstash.agent] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: PipelineAction::Create, action\_result: false", :backtrace=\>nil} ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 12, 2020, 2:23pm UTC](https://discuss.elastic.co/t/issue-with-filter-configuration-in-logstash/236911/2 "2020-06-12T14:23:14Z")
> [@kittu9999](#): > > :error=\>"pattern %{JSON:requestJSON} not defined", It is telling you that there is not "JSON"pattern defined. What is in "/appdata/config/logstash/patterns"? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 12, 2020, 2:52pm UTC](https://discuss.elastic.co/t/issue-with-filter-configuration-in-logstash/236911/4 "2020-06-12T14:52:19Z")
That looks OK. If you set log.level trace do the detail messages indicate it is reading /appdata/config/logstash/patterns/jsonPattern? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 12, 2020, 4:38pm UTC](https://discuss.elastic.co/t/issue-with-filter-configuration-in-logstash/236911/6 "2020-06-12T16:38:32Z")
You could use a site like [https://gist.github.com/](https://gist.github.com/) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 12, 2020, 5:18pm UTC](https://discuss.elastic.co/t/issue-with-filter-configuration-in-logstash/236911/8 "2020-06-12T17:18:40Z")
It logs ``` Grok patterns path {:paths=>["/appdata/config/logstash/patterns"]} ``` but never logs a ``` Grok loading patterns from file {:path=>"... ``` for any file under that directory. This really suggests that /appdata/config/logstash/patterns/jsonPattern does not exist. Is it possible you are running in a container and the container has not mounted the /appdata/config/logstash/patterns directory? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 13, 2020, 5:40pm UTC](https://discuss.elastic.co/t/issue-with-filter-configuration-in-logstash/236911/12 "2020-07-13T17:40:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.