Unable to get local Kibana to read any data from remote Metricbeat

Elastic Stack Beats
1

I have Kibana installed on my local machine and have Elasticsearch and Metricbeat installed on a remote AWS EC2 instance. For simplicity, let's say the public IP address of the AWS box is http://1.2.3.4. In my local kibana,yml, I have elastic.hosts set to http://1.2.3.4 and I am able to run queries against the remote Elasticsearch instance with no problems.

The problem comes in when trying to get Metricbeat data. In Kibana, I go to Management -> Stack Monitoring, and it guides me through a wizard to set up monitoring via Metricbeat - it first asks me enter the remote server address of the monitoring cluster, so I enter http://1.2.3.4:9200. I then follow the instructions in the wizard which instructs me to install Metricbeat and Elasticsearch on the same machine (done). Then it tells me to run a command on the remote machine to enable x-pack (done). Then it tells me to "Configure Metricbeat to send data to the monitoring cluster..." by modifying the output.elasticsearch.hosts param in metricbeat.yml to be the remote server IP address. Now, I'm not sure why I wouldn't just use the default value of localhost:9200 since Metricbeat and ES are on the same machine, but fine, I'll follow instructions, and I change that value to the actual public IP address of the remote machine. Then it tells me to start Metricbeat. (done).

However, I get nothing in Kibana - the wizard says "No monitoring data detected, but we'll continue checking."

What am I missing here? I also noticed there is no ES index called metricbeat-xxx but I don't know if that's an issue or not.

Any ideas?

5

What do your Metricbeat logs show?

6

There are only 2 lines in the log as such:

|2022-10-19T22:13:46.296Z|INFO|instance/beat.go:645|Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
|2022-10-19T22:13:46.296Z|INFO|instance/beat.go:653|Beat ID: ad6ac307-a7d1-4a98-89eb-32bf1e92aeb3|
7

Ok well here's an obvious issue. Since I didn't receive any error message when running 'sudo service start metricbeat', I assumed all was good. But when I run 'sudo systemctl status metricbeat', I get this:

metricbeat.service - Metricbeat is a lightweight shipper for metrics.

Loaded: loaded (/lib/systemd/system/metricbeat.service; disabled; vendor preset: enabled)

Active: **failed** (Result: exit-code) since Thu 2022-10-20 14:50:23 UTC; 42s ago

Docs: https://www.elastic.co/products/beats/metricbeat

Process: 1654 ExecStart=/usr/share/metricbeat/bin/metricbeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OP>

Main PID: 1654 (code=exited, status=1/FAILURE)

Oct 20 14:50:23 ip-172-31-22-220 systemd[1]: metricbeat.service: Scheduled restart job, restart counter is at 5.

Oct 20 14:50:23 ip-172-31-22-220 systemd[1]: Stopped Metricbeat is a lightweight shipper for metrics..

Oct 20 14:50:23 ip-172-31-22-220 systemd[1]: **metricbeat.service: Start request repeated too quickly.**

Oct 20 14:50:23 ip-172-31-22-220 systemd[1]: **metricbeat.service: Failed with result 'exit-code'.**

Oct 20 14:50:23 ip-172-31-22-220 systemd[1]: **Failed to start Metricbeat is a lightweight shipper for metrics..**

I don't quite see what the actual issue is though from this message