Pattern not defined even though the first matches


(Maren Sonnenschein) #1

Hi everybody,

I'm pretty new to the ELK stack but I think I understand the basics.

So far, I've been able to define a pattern to match my log file and an additional grok pattern. I also checked if it works as expected by using the Grok Constructor.

My grok patterns are defined as follows

CLIENT \d{1,3}
TEXT (?:(?!\s{2,}).)*

whereas my configuration looks like this:

input {
  file {
    path => [ "/media/SAP/logs" ]
    type => [ "security_audit_log" ]
    start_position => [ "beginning" ]
    codec => plain {
      charset => "CP1252"	
    }
  }
}

filter {
  if [type] == "security_audit_log" {
    grok {
      patterns_dir => ["/media/ELK/logstash-5.4.1/config/patterns"]
      match => { 
        "message" => "\|%{DATE_EU:date}?%{SPACE}?\|%{TIME:time}?%{SPACE}?\|%{CLIENT:client}?%{SPACE}?\|%{USERNAME:user}?%{SPACE}?\|%{TEXT:terminal}?%{SPACE}?\|%{TEXT:tcode}?%{SPACE}?\|%{TEXT:program}?%{SPACE}?\|%{TEXT:text}?%{SPACE}?\|"
      }
    }
    mutate {
      add_field => {
        "timestamp" => "%{date} %{time}" 
      }
      remove_field => [ "date", "time" ]
    }
    date {
      match => [ "timestamp" , "dd.MM.yyyy HH:mm:ss" ]
      timezone => [ "Europe/Berlin" ]
      target => [ "@timestamp" ]
    }
  }
}

output {
  if [type] == "security_audit_log" {
    elasticsearch {
      hosts => [ "localhost:9200" ]
      index => [ "logstash-security_audit_log" ]
    }
    stdout {
      codec => rubydebug
    }
  }	
}

Now, when runnin ./bin/logstash as I've downloaded the zip, I receive the following error:

elk@ELK:/media/ELK/logstash-5.4.1/bin$ ./logstash
Sending Logstash's logs to /media/ELK/logstash-5.4.1/logs which is now configured via log4j2.properties
[2017-07-28T11:09:57,742][WARN ][logstash.filters.date    ] Date filter now use BCP47 format for locale, replacing underscore with dash
[2017-07-28T11:09:58,398][INFO ][...]
[2017-07-28T11:09:58,400][INFO ][...]
[2017-07-28T11:09:58,576][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x6d065f34 URL:http://localhost:9200/>}
[2017-07-28T11:09:58,578][INFO ][...]
[2017-07-28T11:09:58,657][INFO ][...]
[2017-07-28T11:09:58,670][INFO ][...]
[2017-07-28T11:09:58,768][ERROR][logstash.pipeline        ] Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x5aa1742e @id=\"36038c963454ac945ea4e7fd04113944e40ae1c9-3\", @klass=LogStash::Filters::Grok, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x5b4d7979 @metric=#<LogStash::Instrument::Metric:0x78a26672 @collector=#<LogStash::Instrument::Collector:0x48f5595f @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x1924546b @store=#<Concurrent::Map:0x29127087 @default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x42e81221>, @fast_lookup=#<Concurrent::Map:0x20358482 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :\"36038c963454ac945ea4e7fd04113944e40ae1c9-3\", :events]>, @logger=#<LogStash::Logging::Logger:0x2125c64d @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0x6831045c>>, @filter=<LogStash::Filters::Grok patterns_dir=>[\"/media/ELK/logstash-5.4.1/config/patterns\"], match=>{\"message\"=>\"\\\\|%{DATE_EU:date}?%{SPACE}?\\\\|%{TIME:time}?%{SPACE}?\\\\|%{CLIENT:client}?%{SPACE}?\\\\|%{USERNAME:user}?%{SPACE}?\\\\|%{TEXT:terminal}?%{SPACE}?\\\\|%{TEXT:tcode}?%{SPACE}?\\\\|%{TEXT:program}?%{SPACE}?\\\\|%{TEXT:text}?%{SPACE}?\\\\|\"}, id=>\"36038c963454ac945ea4e7fd04113944e40ae1c9-3\", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>\"*\", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>[\"_grokparsefailure\"], timeout_millis=>30000, tag_on_timeout=>\"_groktimeout\">>", :error=>"pattern %{TEXT:terminal} not defined"}
[2017-07-28T11:09:58,795][ERROR][logstash.agent           ] Pipeline aborted due to error {:exception=>#<Grok::PatternError: pattern %{TEXT:terminal} not defined>, :backtrace=>["/media/ELK/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:123:in `compile'", "org/jruby/RubyKernel.java:1479:in `loop'", "/media/ELK/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:93:in `compile'", "/media/ELK/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:286:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/media/ELK/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:280:in `register'", "org/jruby/RubyHash.java:1342:in `each'", "/media/ELK/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:275:in `register'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugin'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:279:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:279:in `register_plugins'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:289:in `start_workers'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:214:in `run'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/agent.rb:398:in `start_pipeline'"]}
[2017-07-28T11:09:58,859][INFO ][...]
[2017-07-28T11:10:01,830][WARN ][...]

whereas the most important part seems to be the pattern %{TEXT:terminal} not defined part. I'm wondering: why is that? As the CLIENT patterns seems to be applied, why does logstash not find the second one?

I've already been looking for a solution here but the only hints that I've found is checking the ownership of the pattern file/directory (which applies) and using the complete path to the patterns directory (what I did as well).

Any suggestions why this error is displayed? Or even a solution?

Best regards and thanks in advance,

Hisushi

Edit
As suggested in another post, I used logstash with --debug which shows me the following:

[2017-07-31T10:18:57,996][DEBUG][logstash.filters.grok    ] Adding pattern {"CLIENT"=>"(\\d{1,3})\r\rTEXT ((?:(?!\\s{2,}).)*)"}

Why is that?


(Maren Sonnenschein) #2

Anybody?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.