DIfferent Kinds of events from filebeat to Logstash, Assorting and Parsing

Elastic Stack Logstash
1

Hello,
I am trying to use filebeat to send two different types of events each from a different log to Logstash which does a different grok parse on each and send with a different id, but the same index to elasticsearch. I will be showing my filebeat input config and pipeline config in logstash, but when I try to use it- it just gives a very unhelpful error in log. I am not sure if my syntax is wrong or if something else is the problem.

FIlebeat input config:

filebeat.inputs:

  • type: log
    paths:

    • /home/surya/final.log
      enabled: true
      ignore_older: 30m
      scan_frequency: 5m
      harvester_limit: 2
      close_inactive: 10m
      #include/exclude_lines:
      fields: {log_type: SYSSTATS}

  • type: log
    paths:

    • /home/surya/trunk.log
      enabled: true
      ignore_older: 30m
      scan_frequency: 5m
      harvester_limit: 2
      close_inactive: 10m
      include_lines: ['^tcp']
      fields: {log_type: NETSTATS}

LOGSTASH PIPELINE CONFIG:
input{
beats{
port => 5044
host => "10.0.2.15"
}
}

filter {
if [fields][log_type] == "SYSSTATS" {
grok {
match => {
"message" => "%{SPACE:1}%{NUMBER:pid}%{SPACE:1}%{WORD:user}%{SPACE:1}%{INT:priority}%{SPACE:1}%{INT:nice_value}%{SPACE:1}%{NOTSPACE:virtual_memory}%{SPACE:1}%{NOTSPACE:physical_memory}%{SPACE:1}%{NOTSPACE:shared_memory}%{SPACE:1}%{WORD:status}%{SPACE:1}%{BASE16FLOAT:cpu_usage}%{SPACE:1}%{BASE16FLOAT:ram_usage}%{SPACE:1}%{NOTSPACE:activity_time}%{SPACE:1}%{WORD:command}%{SPACE:1}"
}
remove_field => [ "1", "host.name", "version", "_id", "_index", "_score", "_type", "beat.hostname", "beat.name", "beat.version" ]
}
mutate { add_field => { "[@metadata][test]" => "SYSUSAGE" } }
}
if [fields][log_type] == "NETSTATS" {
grok {
match => {
"message" => "%{WORD:protoc}%{SPACE:2}%{NUMBER:recv_q}%{SPACE:2}%{NUMBER:send_q}%{SPACE:2}%{NOTSPACE:local_add}%{SPACE:2}%{NOTSPACE:forn_add}%{SPACE:2}%{WORD:status}"
}
remove_field => ["2"]
}
mutate { add_field => { "[@metadata][test]" => "NETUSAGE" } }
}
}
output{
if [@metadata][test] == "NETUSAGE" {
stdout{}
elasticsearch {
host => "10.0.2.15"
manage_template => false
index => "%{[@metadata][version]-%{+YYYY.MM.dd}"
id => "Network_Usage"
}
}
if [@metadata][test] == "SYSUSAGE" {
stdout{}
elasticsearch {
host => "10.0.2.15"
manage_template => false
index => "%{[@metadata][version]-%{+YYYY.MM.dd}"
id => "System_Usage"
}
}
}

ERROR WAS:
[2018-07-17T16:16:34,649][ERROR][logstash.outputs.elasticsearch] Unknown setting 'host' for elasticsearch
[2018-07-17T16:16:34,713][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:89:in config_init'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:62:ininitialize'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:202:in initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:68:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/plugin_factory.rb:93:in plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:110:inplugin'", "(eval):116:in <eval>'", "org/jruby/RubyKernel.java:994:ineval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:82:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:inblock in converge_state'"]}

I am new here so not really been able to pick up simple mistakes, could be very helpful if u could look at it.

Thanks

2

How could the error message "Unknown setting 'host' for elasticsearch" be improved? What do you find unclear about it?

host => "10.0.2.15" should be hosts => "10.0.2.15"

3

Hey thanks again Badger. host => was working fine for me earlier, i did not expect to change it. Thanks, it worked.
Surya

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.