Custom GROK Patterns not work

Elastic Stack Logstash
1

Hi all,

we tried to implement some custom grok patterns into a seperate file and load it into logstash conf by using patterns_dir. Each time running logstash with the following conf output a pattern not defined error message. We also tried it by defining the patterns with pattern_definitions param, which also not work. Whats going wrong? We are using Logstash 5.3 from tar.gz

Here our config an pattern files:

input {
  file {
    path => "/home/test/access.log"
  }
}
filter {
  grok {
    patterns_dir => ["./patterns"]
    patterns_files_glob => "*"
    match => {"message" => "%{UNIX:timestamp} %{BASE10NUM:elapsed} %{IPV4:remotehost} %{STATUS:status} %{BASE10NUM:bytes} %{HTTPMETHOD:method} %{URI:u$
    }
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

./patterns/extra

UNIX \b(?:\d{10}\.{1}\d{3})\b
STATUS \b(?:.{3}_.+/\d{3})\b
HTTPMETHOD \b(?:\w{3,6})\b
PEERSTATUS \b(?:[A-Za-z_]+)\b
PEERHOST \b(?:%{IP})\b
MIMETYPE \b(?:[a-z/]+)\b
REST \b(?:.*\b)

Here is our call and output:

 bin/logstash -f config/logstash-squid.conf
 Sending Logstash's logs to /home/test/Downloads/logstash-5.3.0/logs which is now configured via log4j2.properties
[2017-04-18T15:57:35,624][ERROR][logstash.pipeline        ] Error registering plugin {:plugin <LogStash::FilterDelegator:0x43d64823 @id=\"f2074095a5cbdf94f08313d09dfedd6deadef18a-2\", @klass=LogStash::Filters::Grok, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x1c0c9f07 @metric=#<LogStash::Instrument::Metric:0x15481a1c @collector=#<LogStash::Instrument::Collector:0x1827f39 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x7064a87a @store=#<Concurrent::Map:0x8e8e5d6 @default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x5da5962d>, @fast_lookup=#<Concurrent::Map:0x4f4c5117 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :\"f2074095a5cbdf94f08313d09dfedd6deadef18a-2\", :events]>, @logger=#<LogStash::Logging::Logger:0x55b78728 @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0x1aeb7afa>>, @filter=<LogStash::Filters::Grok patterns_dir=>[\"./patterns\"], patterns_files_glob=>\"*\", match=>{\"message\"=>\"%{UNIX:timestamp} %{BASE10NUM:elapsed} %{IPV4:remotehost} %{STATUS:status} %{BASE10NUM:bytes} %{HTTPMETHOD:method} %{URI:url} %{PEERSTATUS:peerstatus}\\\\/%{PEERHOST:peerhost} %{MIMETYPE:mime}\"}, id=>\"f2074095a5cbdf94f08313d09dfedd6deadef18a-2\", enable_metric=>true, periodic_flush=>false, break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>[\"_grokparsefailure\"], timeout_millis=>30000, tag_on_timeout=>\"_groktimeout\">>", :error=>"pattern %{UNIX:timestamp} not defined"}
[2017-04-18T15:57:35,640][ERROR][logstash.agent           ] Pipeline aborted due to error {:exception=>#<Grok::PatternError: pattern %{UNIX:timestamp} not defined>, :backtrace=>["/home/test/Downloads/logstash-5.3.0/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:123:in `compile'", "org/jruby/RubyKernel.java:1479:in `loop'", "/home/test/Downloads/logstash-5.3.0/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:93:in `compile'", "/home/test/Downloads/logstash-5.3.0/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:286:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/home/test/Downloads/logstash-5.3.0/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:280:in `register'", "org/jruby/RubyHash.java:1342:in `each'", "/home/test/Downloads/logstash-5.3.0/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:275:in `register'", "/home/test/Downloads/logstash-5.3.0/logstash-core/lib/logstash/pipeline.rb:257:in `register_plugin'", "/home/test/Downloads/logstash-5.3.0/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/home/test/Downloads/logstash-5.3.0/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugins'", "/home/test/Downloads/logstash-5.3.0/logstash-core/lib/logstash/pipeline.rb:278:in `start_workers'", "/home/test/Downloads/logstash-5.3.0/logstash-core/lib/logstash/pipeline.rb:207:in `run'", "/home/test/Downloads/logstash-5.3.0/logstash-core/lib/logstash/agent.rb:389:in `start_pipeline'"]}
[2017-04-18T15:57:35,679][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-04-18T15:57:38,660][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}

Best regards, David

2

Do u set permission for "patterns" folder?

chown -R logstash. /path/to/patterns

1 Like
3

All permission setup correctly.

4

I'm looking ur log

I think your "pattern extra" can't load. i should be use path static like

patterns_dir => ["/etc/logstash/patterns"]

1 Like
5

Thats it. So it was a mistake from myself. Thanks. Now it works. The Path to my paterns directory was wrong

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.