Logstash doesn't receive Metricbeat data

[sumitbiswas]

Hi,

I am trying to make Metricbeat (running in remote server) to send data to Logstash and from there to Elasticsearch.

I already have Filebeat (running in same remote server) to send data to same Logstash and then to Elasticsearch. This is working well.

Since I am sending data through Logstash, I ran below command

curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/metricbeat' -d@metricbeat.template.json

metricbeat.yml looks like below. I have entered same ip address and port for Logstash as mentioned in filebeat.yml file.

metricbeat.modules:
#------------------------------- System Module -------------------------------

• module: system
  metricsets:

  CPU stats

  • cpu
    ...
    output.logstash:
    hosts: ["ip:5044"]

I have commented elasticsearch module in above file.

My Logstash confile looks like below

input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:ProcessName} %{INT:Latency}" }
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:SubProcessName} %{INT:Count}" }
}

mutate {
convert => { "Latency" => "integer" }
}

mutate {
convert => { "Count" => "integer" }
}

date {
match => [ "timestamp" , "MMM dd HH:mm:ss.SSSSSS" ]
}

if "_grokparsefailure" in [tags] {
drop { }
}

mutate {
remove_field => [ "ProcessName", "SubProcessName" ]
}

}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

When I start all processes, I see index like filebeat-2017.02.07, but nothing appears for Metricbeat.

I must be doing something wrong but don't know what. Can you please help me to resolve it?

[sumitbiswas]

I think the problem is with loading template. I.e.

curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/metricbeat' -d@metricbeat.template.json

I copied metricbeat.template.json from Remote Server where I placed Metricbeat to Logstash server. Then I ran above command in Logstash server from the directory I copied to. The acknowledgement was true as its result output.

Is this the problem? Not able to figure it out.

[andrewkroh]

I wonder if it is something to do with the LS filters. I'm assuming those are only for Filebeat data, so could you try putting a conditional around all that logic so that it only applies to the Filebeat data.

filters {
  if [@metadata][beat] == "filebeat" {
    // Filebeat specific logic.
  }
}

[sumitbiswas]

Thanks Andrew. You made my day. That was right. So I guess this is what was happening. The event from Metricbeat was either dropped or removed. So now after the if condition, Metricbeat event goes separately and doesn't get filtered with Filebeat conditions.