Beat-xpack module doesn’t work on localhost 5066 port

[PodarcisMuralis (Murat K.)]

Hello,

I am trying to use Beat Module for monitoring Metricbeat on Kibana-Stack Monitoring. I am using this in metricbeat.yml

  - module: beat
    xpack.enabled: true
    period: 10s
    hosts: [ "http://localhost:5066" ]
    username: "xxx"
    password: "xxx"

http.enabled: true
http.host: localhost
http.port: 5066

But I get this error if I check the connection with ./metricbeat test modules beat

beat...
  state...
    error... ERROR error making http request: Get "http://localhost:5066/state": dial tcp [::1]:5066: connect: connection refused
  stats...
    error... ERROR error making http request: Get "http://localhost:5066/stats": dial tcp [::1]:5066: connect: connection refused

Edit 1: Other modules like Kibana, Elasticsearch and Logstash are working well.
And interestingly if I use ./metricbeat test modules beat I got the error but if I start metricbeat with ./metricbeat -e, "Disable self monitoring" appears nicely on Stack Monitoring.

Edit 2: If I remove http parts from metricbeat.yml, ./metricbeat test modules beat gives the same error but I can not see "Disable self monitoring" on Stack Monitoring.

Do you have an idea?
Thanks

[stephenb (Stephen Brown)]

What version of Beats / Elastic?

Easiest way to monitor beats is with internal monitoring ... normally what I use..

Use internal collection to send monitoring data | Metricbeat Reference [8.11]...

Is there a reason you are using the more complex method?

I suspect if you look in the logs or netstat metricbeat is unable to open the 5066 port

@PodarcisMuralis

Edit 1: Other modules like Kibana, Elasticsearch and Logstash are working well.
And interestingly if I use ./metricbeat test modules beat I got the error but if I start metricbeat with ./metricbeat -e, "Disable self monitoring" appears nicely on Stack Monitoring.

Did you read about monitoring .... Monitoring in a production environment | Elasticsearch Guide [8.11] | Elastic

Specifically

GET _cluster/settings

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": false
  }
}

@PodarcisMuralis

Edit 2: If I remove http parts from metricbeat.yml, ./metricbeat test modules beat gives the same error but I can not see "Disable self monitoring" on Stack Monitoring.

That just tests and does not send data that makes sense because no data is sent... .

Monitoring can be a bit confusing...

I have found

First and foremost, you also have to monitor the elasticsearch cluster if you want to show the beats in stack monitoring... Just Beat Monitoring will not show...

a) For beat the internal Collection is easiest.

b) Setup Elasticsearch-xpack modules

Then you should get both the beat and elasticsearch on stack monitoring

in my metricbeat.yml this is all I added

# Set to true to enable the monitoring reporter.
monitoring.enabled: true

and then for me I just enabled the elasticsearch-xpack module

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["http://localhost:9200"]
  #username: "user"
  #password: "secret"

Then ran metricbeat and here it is

I highly recommend reading the docs

Monitoring overview | Elasticsearch Guide [8.11] | Elastic

[PodarcisMuralis (Murat K.)]

Hi Stephen,

we actually wanted to disable Self-Monitoring. Because as I read on some sources Self Monitoring will be gone in future. We unified the system and monitoring everything with Metricbeat.

If you enter "Enter Setup Mode" there are 3 sentences:

• Monitored with Metricbeat
• Disableself Monitoring
• Monitor with Metricbeat

I used this

  - module: beat
    xpack.enabled: true
    period: 10s
    hosts: [ "http://localhost:5066" ]
    username: xxx"
    password: "xxx"

http.enabled: true
http.host: localhost
http.port: 5066

It worked but then it said automatically. In order to disable self monitoring change
monitoring:
enabled: false (it was true).

Then it worked for me. I have everywhere the same sentence.

• Monitored with Metricbeat

[stephenb (Stephen Brown)]

H @PodarcisMuralis

Yep, it can be confusing.

You're confusing self-monitoring which is slowly deprecating versus just the simple metricbeat internal monitoring two different things.

Self-monitoring is only in reference to elasticsearch has nothing to do with beats at all, and yes that is "deprected/ing"

Metricbeat internal monitoring is not going away as far as I understand and it's perfectly valid and I use it all the time.

You're welcome to use whatever method you like. I was just sharing what I consider to be the simple and supported path.

I did nothing more than when I showed you above.

The problem there can be some issues if you already enabled self-monitoring and then are going back to try to do monitoring with metricbeat

[PodarcisMuralis (Murat K.)]

Hi Stephen.

Thank you very much for your kind assistance. I really appreciate it.
I have already set up this method. Monitoring Beats with Metricbeat.

I just changed to your method and saw that it says "Monitor with Metricbeat". I think there is no difference between them and you are right, I chose the hard way.

Thanks again and have a nice weekend.

[stephenb (Stephen Brown)]

We learn most when we take the hard way .. but I could skip it a few times myself

Our docs / guidance could be a bit clearer.

[PodarcisMuralis (Murat K.)]

The magic answer why I got this error is, to start metricbeat itself.
sudo systemctl start metricbeat.
Then you can test your metricbeat modules.
./metricbeat test modules beat