Hi everybody,
I'm pretty new to the ELK stack but I think I understand the basics.
So far, I've been able to define a pattern to match my log file and an additional grok pattern. I also checked if it works as expected by using the Grok Constructor.
My grok patterns are defined as follows
CLIENT \d{1,3}
TEXT (?:(?!\s{2,}).)*
whereas my configuration looks like this:
input {
file {
path => [ "/media/SAP/logs" ]
type => [ "security_audit_log" ]
start_position => [ "beginning" ]
codec => plain {
charset => "CP1252"
}
}
}
filter {
if [type] == "security_audit_log" {
grok {
patterns_dir => ["/media/ELK/logstash-5.4.1/config/patterns"]
match => {
"message" => "\|%{DATE_EU:date}?%{SPACE}?\|%{TIME:time}?%{SPACE}?\|%{CLIENT:client}?%{SPACE}?\|%{USERNAME:user}?%{SPACE}?\|%{TEXT:terminal}?%{SPACE}?\|%{TEXT:tcode}?%{SPACE}?\|%{TEXT:program}?%{SPACE}?\|%{TEXT:text}?%{SPACE}?\|"
}
}
mutate {
add_field => {
"timestamp" => "%{date} %{time}"
}
remove_field => [ "date", "time" ]
}
date {
match => [ "timestamp" , "dd.MM.yyyy HH:mm:ss" ]
timezone => [ "Europe/Berlin" ]
target => [ "@timestamp" ]
}
}
}
output {
if [type] == "security_audit_log" {
elasticsearch {
hosts => [ "localhost:9200" ]
index => [ "logstash-security_audit_log" ]
}
stdout {
codec => rubydebug
}
}
}
Now, when runnin ./bin/logstash as I've downloaded the zip, I receive the following error:
elk@ELK:/media/ELK/logstash-5.4.1/bin$ ./logstash
Sending Logstash's logs to /media/ELK/logstash-5.4.1/logs which is now configured via log4j2.properties
[2017-07-28T11:09:57,742][WARN ][logstash.filters.date ] Date filter now use BCP47 format for locale, replacing underscore with dash
[2017-07-28T11:09:58,398][INFO ][...]
[2017-07-28T11:09:58,400][INFO ][...]
[2017-07-28T11:09:58,576][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x6d065f34 URL:http://localhost:9200/>}
[2017-07-28T11:09:58,578][INFO ][...]
[2017-07-28T11:09:58,657][INFO ][...]
[2017-07-28T11:09:58,670][INFO ][...]
[2017-07-28T11:09:58,768][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x5aa1742e @id=\"36038c963454ac945ea4e7fd04113944e40ae1c9-3\", @klass=LogStash::Filters::Grok, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x5b4d7979 @metric=#<LogStash::Instrument::Metric:0x78a26672 @collector=#<LogStash::Instrument::Collector:0x48f5595f @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x1924546b @store=#<Concurrent::Map:0x29127087 @default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x42e81221>, @fast_lookup=#<Concurrent::Map:0x20358482 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :\"36038c963454ac945ea4e7fd04113944e40ae1c9-3\", :events]>, @logger=#<LogStash::Logging::Logger:0x2125c64d @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0x6831045c>>, @filter=<LogStash::Filters::Grok patterns_dir=>[\"/media/ELK/logstash-5.4.1/config/patterns\"], match=>{\"message\"=>\"\\\\|%{DATE_EU:date}?%{SPACE}?\\\\|%{TIME:time}?%{SPACE}?\\\\|%{CLIENT:client}?%{SPACE}?\\\\|%{USERNAME:user}?%{SPACE}?\\\\|%{TEXT:terminal}?%{SPACE}?\\\\|%{TEXT:tcode}?%{SPACE}?\\\\|%{TEXT:program}?%{SPACE}?\\\\|%{TEXT:text}?%{SPACE}?\\\\|\"}, id=>\"36038c963454ac945ea4e7fd04113944e40ae1c9-3\", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>\"*\", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>[\"_grokparsefailure\"], timeout_millis=>30000, tag_on_timeout=>\"_groktimeout\">>", :error=>"pattern %{TEXT:terminal} not defined"}
[2017-07-28T11:09:58,795][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<Grok::PatternError: pattern %{TEXT:terminal} not defined>, :backtrace=>["/media/ELK/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:123:in `compile'", "org/jruby/RubyKernel.java:1479:in `loop'", "/media/ELK/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:93:in `compile'", "/media/ELK/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:286:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/media/ELK/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:280:in `register'", "org/jruby/RubyHash.java:1342:in `each'", "/media/ELK/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:275:in `register'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugin'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:279:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:279:in `register_plugins'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:289:in `start_workers'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:214:in `run'", "/media/ELK/logstash-5.4.1/logstash-core/lib/logstash/agent.rb:398:in `start_pipeline'"]}
[2017-07-28T11:09:58,859][INFO ][...]
[2017-07-28T11:10:01,830][WARN ][...]
whereas the most important part seems to be the pattern %{TEXT:terminal} not defined part. I'm wondering: why is that? As the CLIENT patterns seems to be applied, why does logstash not find the second one?
I've already been looking for a solution here but the only hints that I've found is checking the ownership of the pattern file/directory (which applies) and using the complete path to the patterns directory (what I did as well).
Any suggestions why this error is displayed? Or even a solution?
Best regards and thanks in advance,
Hisushi
Edit
As suggested in another post, I used logstash with --debug which shows me the following:
[2017-07-31T10:18:57,996][DEBUG][logstash.filters.grok ] Adding pattern {"CLIENT"=>"(\\d{1,3})\r\rTEXT ((?:(?!\\s{2,}).)*)"}
Why is that?