Metricbeat to Logstash to ElasticSearch - Cannot See Any Hosts Defined, But Data Is Definitely Coming In

Elastic Stack Kibana
2

Hi @jthart Welcome to the community apologies that you're having some struggles getting this set up Perhaps we can help.

Assuming you want to run an architecture like this

Metricbeat (1 to Many) -> Logstash -> Elasticsearch

Basically using Logstash as a collect and pass through

Here is my recommendation try to resist the urge to make this more complex. Do not try to manually load index templates dashboards anything else follow the quick start / basic setup.

  1. Clean everything up we're starting over.

  2. Follow the exact steps 1 through
    6 here... At this point metricbeat should be properly configured and sending telemetry directly to Elasticsearch and you should be able to view it in Kibana in metrics app.

  1. Stop metricbeat.

  2. Use this Logstash config. This configuration acts as a pass through and will take any and all the metadata associated with metric beat including if there's any pipelines etcetera to be run and pass them through logstash.

################################################
# beats->logstash->es default config.
################################################
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      user => "elastic"
      password => "secret"
    }
  }
}

  1. Start Logstash. It should be listening on the beats input port 5044.

  2. Edit the metricbeat.yml
    Comment out the Kibana setup and the output.elasticsearch configurations.

Configure metricbeat to point at Logstash

output.logstash:
      # The Logstash hosts
      hosts: ["localhost:5044"]
  1. Start metricbeat.

Now you should be sending metricbeat data through logstash to elasticsearch with all the mappings, index templates ILM dashboards etc

Note you only need to run the setup step once whether you're collecting metrics from one host or a thousand hosts setup only needs to be run once