Metricbeat can't output to logstash

Elastic Stack Beats
1

Hi team,

I had my metricbeat configured to elasticsearch, everything was working fine. I now need to include logstash, however, after changing the metricbeat output to logstash with my host, "http://<ip_address>:5044"(enabling the module and commenting out the elastic output), i get the following errors:

  1. after using, ./metricbeat -e setup:
{"log.level":"error","@timestamp":"2024-03-15T13:26:25.118Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1315},"message":"Exiting: index management requested but the Elasticsearch output is not configured/enabled","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: index management requested but the Elasticsearch output is not configured/enabled
  1. after using ./metricbeat -e:
{"log.level":"error","@timestamp":"2024-03-15T13:26:30.022Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1315},"message":"Exiting: error connecting to Kibana: fail to get the Kibana version: fail to parse kibana version (): passed version is not semver: ","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: error connecting to Kibana: fail to get the Kibana version: fail to parse kibana version (): passed version is not semver:

I have also tried looking into the logstash and metricbeat logs but they don't shed any new light on the problem. If i revert back to the elastic output it works fine. I am very confused why it is not giving any signs that it is connecting to logstash.

my pipeline.conf input is such that:

 input {
  beats  {
    port => 5044
    host => "0.0.0.0"
    codec => "json"
  }
}

Any help would be greatly appreciated

2

Hello,

Please share your metricbeat.yml so it is possible to know what the configuration is.

3

Hi @leandrojmp, thank you for your response, certainly here it is:

metribeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.dashboards.enabled: true
setup.kibana: 
  host: "http://<ip_address>:5601"

#output.elasticsearch:
  #hosts: ["https://<ip_address>:9200"]
  # ssl.enabled: true
  # ssl.certificate_authorities: "path/to/cert.crt"
  #ssl.verification_mode: none
  # username: "elastic"
  # password: "changeme"

output.logstash:
  enabled: true
  hosts: ["http://<ip_address>:5044"]

I have using it with activemq and system modules

4

This is wrong, logstash beats input does not use http, remove the http and configure just the ip address and port.

This error means that you tried to setup metricbeat with metricbeat -e setup, but the elasticsearch output was disabled, to run metricbeat -e setup you need to configure your output to elasticsearch or it will not work.

If you didn't run setup before, you need to change your output to elasticsearch, run the setup, and change it back to logstash.

This error means that you metricbeat cannot get the kibana version for some reason, you have setup.dashboards.enabled and setup.kibana.hosts configured, so metricbeat will always try to connect to kibana.

You need to check if your metricbeat server can connect to kibana or remove those settings from your configuration file.

1 Like
5

In regards to,

If you didn't run setup before, you need to change your output to elasticsearch, run the setup, and change it back to logstash.

I have ran the setup before with elasticsearch as the output. I got neither of the metricbeat errors above.

Am I right in thinking that i need to do the following steps:

  1. revert back to the elasticsearch output run ./metricbeat -e setup
  2. remove http:// from the logstash output then run ./metricbeat -e setup again?

Just out of curiosity, is this do with changing the indices or something more uncomplex under the hood?

I now need to change my output to Kafka, would I have have to follow the same steps as listed in this post ( if they are correct) or different steps?

Thank you very much for your help,
I look forward to your response.

6

You just need to run setup once, when you configure a new module or when you upgrade the version, there is no need to keep running it.

To run metricbeat -e setup the output needs to be elasticsearch, it won't work if the output is logtash or kafka.

Have you added a new module or upgraded it? If no, then there is no need to run sertup again.

As mentioned, the setup parameter only works if the output is elasticsearch, but if you want to send data to logstash you need to remove the http from the output configuration because the logstash output does not use http.

Not sure if I got, but you run setup to configure the ingest pipelines in Elasticsearch, index templates etc, there is no need to run it multiple times.

7

Thank you for your help everything is sorted now :slight_smile:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.