Metricbeat

[Manuel_Laesser]

Hello
I'm pretty new to ELK and installed the whole stack one one machine.

Filebeat and Metricbeat --> Logstash --> Elasticsearch <-- Kibana

Filebeat is working great but today I installed metricbeat. For me it seems, because I'm sending metricbeats trough logstash to elasticsearch that all metrics will be stored in the logstash-* indicies. I have a logstash-* and a metricbeats-* index pattern but if I go to discover in Kibana all Metricbeats Fields appear only if I choose logstash-. If I choose metricbeats- there are no Available or Selected Fields. I think I missed something in /etc/logstash/conf.d/ and send metricbeats data to metricbeats-* instead logstash-*.

My Input/Filter/Output file looks like

input {
beats {
port => "5043"
}
}

filter {
if [type] == "apache-access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
} else {
grok {
match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:msg}" }
}
}

output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["localhost:9200"]
user => yyyy
password => xxxx
}
}

Do I have to create an additional filter to send type metricsets to index metricbeat like described in https://www.elastic.co/guide/en/beats/metricbeat/current/logstash-output.html?

Thank you very much for your help.

Manuel

[andrewkroh]

You need to add the options to the elasticsearch output as described on the page you linked. These additional outputs specify which index to write the data to. By default data goes to logstash-YYYY.MM.dd but you need it to go to metricbeat-YYYY.MM.dd.

output {
  elasticsearch {
    host => "localhost"
    port => "9200"
    protocol => "http"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

[Manuel_Laesser]

@andrewkroh

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

Hello Andrew
Thank you for your answer. It seems that then everything will be sent to metricbeat-*. I tried with

output {
  stdout { codec => rubydebug }
  if [document_type] {
    "%{[@metadata][type]}" {
      elasticsearch {
        hosts => ["localhost:9200"]
        user => xxxx
        password => yyyy
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      }
    } else {
     elasticsearch {
       hosts => ["localhost:9200"]
       user => xxxx
       password => yyyyy
     }
  }
}

but this seems to be wrong.

Regars Manuel

[andrewkroh]

That doesn't look like valid configuration. What condition are you trying to test for?

[Manuel_Laesser]

Hello Andrew
I'm trying to send logs from filebeat to indicies logstash-* and data coming from metricbeat to indicies metricbeat-* that's it. IMHO, I think that with the output below everything will be sent to metricbeat-*

output {
  elasticsearch {
    host => "localhost"
    port => "9200"
    protocol => "http"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }

}

After I configured output like this there weren't any messages listed in kibana using indicies logstash-*

Thank you very much for your help.

Manuel

[andrewkroh]

I think this will get you closer to what you want.

output {
  if [@metadata][beat] and [@metadata][beat] != "filebeat" {
    # Send all Beat data except Filebeat through this output.
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  } else {
       elasticsearch {
         hosts => ["http://localhost:9200"]
       }
  }
}