I work with ELK 6
after that I have installed aggregate plugin -logstash refuses to work:
C:\DoLense\logstash\bin>gem install --local -f logstash-filter-aggregate-2.6.4.gem
Successfully installed logstash-filter-aggregate-2.6.4
Parsing documentation for logstash-filter-aggregate-2.6.4
Done installing documentation for logstash-filter-aggregate after 1 seconds
1 gem installed
C:\DoLense\logstash\bin>logstash -f C:\DoLense\logstash\config\logstash.conf
Sending Logstash's logs to C:/DoLense/logstash/logs which is now configured via log4j2.properties
[2017-11-02T15:40:48,461][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/DoLense/logstash/modules/fb_apache/configuration"}
[2017-11-02T15:40:48,479][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/DoLense/logstash/modules/netflow/configuration"}
[2017-11-02T15:40:48,541][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2017-11-02T15:40:49,479][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-11-02T15:40:51,850][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"aggregate", :path=>"logstash/filters/aggregate", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["C:/DoLense/logstash/logstash-core/lib/logstash/plugins/registry.rb:226:in namespace_lookup'", "C:/DoLense/logstash/logstash-core/lib/logstash/plugins/registry.rb:162:inlegacy_lookup'", "C:/DoLense/logstash/logstash-core/lib/logstash/plugins/registry.rb:138:in lookup'", "C:/DoLense/logstash/logstash-core/lib/logstash/plugins/registry.rb:180:inlookup_pipeline_plugin'", "C:/DoLense/logstash/logstash-core/lib/logstash/plugin.rb:140:in lookup'", "C:/DoLense/logstash/logstash-core/lib/logstash/pipeline.rb:143:inplugin'", "(eval):64:in <eval>'", "org/ jruby/RubyKernel.java:1000:ineval'", "C:/DoLense/logstash/logstash-core/lib/logstash/pipeline.rb:82:in initialize'", "C:/DoLense/logstash/logstash-core/lib/logstash/pipeline.rb:215:ininitialize'", "C:/DoLense/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:35:in execute'", "C:/DoLense/logstash/logstash-core/lib/logstash/agent.rb:335:inblock in converge_state'", "C:/DoLense/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "C:/DoLense/logstash/logstash-core/lib/logstash/agent.rb:332:inblock in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "C:/DoLense/logstash/logstash-core/lib/logstash/agent.rb:319:inconverge_state'", "C:/DoLense/logstash/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "C:/DoLens e/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "C:/DoLense/logstash/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "C:/DoLense/logstash/logstash-core/lib/logstash/agent.rb:90:inexecute'", "C:/DoLense/logstash/logstash-core/lib/logstash/runner.rb:359:in block in execute'", "C:/DoLense/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}
[2017-11-02T15:40:51,901][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any filter plugin named 'aggregate'. Are you sure this is correct? Trying to load the aggregate filter plugin resulted in this error: Problems loading the requested plugin named aggregate of type filter. Error: NameError NameError"}
logstash.conf:
input {
beats{
type => beats
port => 5044
}
}
filter{
if [type] == "beats" {
grok {
break_on_match => true
keep_empty_captures => false
match => {
message => [
"%{NUMBER:threadid};%{TIMESTAMP_ISO8601:eventutctime};%{DATA:username};%{JAVACLASS:apiname};%{WORD:apireturnstatus};%{WORD:componentname};%{BASE10NUM:customisation-level};%{BASE10NUM:cputimeinmillisec};%{WORD:runmode};%{WORD:errorcodeiffailure}",
"%{NUMBER:threadid};%{TIMESTAMP_ISO8601:eventutctime};%{DATA:username};%{JAVACLASS:apiname};%{WORD:apireturnstatus};%{WORD:componentname};%{BASE10NUM:customisation-level};%{BASE10NUM:cputimeinmillisec};%{WORD:runmode};(%{GREEDYDATA:errorcodeiffailure})\s*%{GREEDYDATA:error_description}"
]
}
patterns_dir => "C:\DoLense\Patterns\patterns"
}
date {
match => ["eventutctime" , "yyyy-MM-dd HH:mm:ss.SSS"]
}
aggregate {
task_id => "%{apiname}"
code => "map['threadid'] ; map['username']; map['customisation-level']"
}
}
}
output {
elasticsearch {
hosts=>"localhost:9200"
index => "logstash%{+YYYY.MM.dd}%{type}"
}
stdout {codec=> rubydebug }
}