Configure metricbeat to use logstash--tutorial error?

[John_McDonnell]

I've set up a 7.11 Kibana-Elastisearch-Logstash-Beats stack more-or-less successfully, but I am unable to configure metricbeat to use logstash. (I am able to see metricbeat data sending it directly to elastisearch.)

Following the tutorial, at Add user information in Logstash | Elasticsearch Reference [7.11] | Elastic, step 3 on this page says that the link Getting started with the Elastic Stack | Getting Started [7.11] | Elastic showed how to configure metricbeat to use logstash, but it doesn't. The referenced tutorial only explains how to configure metricbeat to use elasticsearch directly. I have been unable to find a tutorial explaining how to get metricbeat to feed logstash, and have been unable to figure out how to do it. I'm guessing it isn't complicated, but that I'm just missing something that will be obvious in retrospect.

I also wanted to report what appears to be an error in the tutorial.

To answer the obvious question, "What's your goal?" the answer is simple--I just want to understand how to do this. Once I get it working, I'll evaluate whether logstash is helping in this use case.

--John Mc

[Badger]

I do not have a metricbeat install handy, but I would expect it just requires a couple of edits to the metricbeat.yml.

Uncomment some lines so that you have

output.logstash:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # The Logstash hosts
  hosts: ["localhost:5044"]

and configure logstash with a beats input.

[John_McDonnell]

That's what I assumed initially and tried it; no errors, but also no data going to Kibana. I'm assuming there are specific metricbeat Logstash ingest/parsing/output files required, but can't find any tutorials on that. I have a Logstash pipeline set up based on the link you provided, and it's working for filebeats.

[stephenb (Stephen Brown)]

Here is the minimum for metricbeat -> logstash -> elasticsearch with all defaults on localhost.

Download all elasticsearch and kibana change nothing, start elasticsearch, then kibana.

Download metricbeat change nothing run

metricbeat setup -e

Then edit metricbeat.yml and comment out output.elasticsearch section and simple uncomment the output.logstash so it looks like

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

Download logstash I would start with the tar.gz first as it is a good way to learn / debug change nothing.

Create this beats-logstash.conf, This is the minimum it will also automatically pass on the right metadata form beats that have pipelines etc.

################################################
# beats->logstash->es default config.
################################################
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      user => "elastic"
      password => "secret"
    }
  }
}

start logstash using the -f

./bin/logstash -f /path/to/conf/beats-logstash.conf

Start metricbeat

metricbeat -e

Observe data in Kbana and Elasticsearch.