Heartbeat No matching indices found for index pattern heartbeat*

[BenKenobi (Ben)]

Hi, I have installed and configured heartbeat according your documentation.
I have created a cfg-file tcpdemo.yml and it looks like this:

Elastic and Kibana run on the same cluster and heartbeat is installed on another server (not in the cluster!) What I try is to run a simple test and send the data to elastic/kibana.
Furthermore I have created a new Kibana user with corresponing rights.

When I run

heartbeat setup

I dont get any errors.
So I go to Kibana and create an index pattern heartbeat-* which matches the existing index heartbeat-7.12.7-2023.09.04-000001

As you can also see: The heartbeat index has 0 docs! (any idea why?)

But when I go to Observability --> Uptime I get following error message:

What can I do here to solve the issue?
Thanks a lot in advance

[abdulz (Abdul Zahid)]

Hi @BenKenobi ,

Thanks for your query.

You seeing the heartbeat-7.17.12xxxx... index confirms that your heartbeat setup command ran successfully. Docs count 0 shows heartbeat hasn't indexed any ping/result document yet.

Let's go through a few steps first to get heartbeat working:

1. A successful execution of the setup command confirms that your heartbeat can connect to ES. You can however additionally confirm it by running heartbeat test output and see if everything's OK.
2. Heartbeat needs to be running in order to index results documents to the index. heartbeat -c <path-to-config> runs the heartbeat which schedules the monitor runs per their schedule. The config your provided has errors. So

   > heartbeat -c tcpdemo.yml -e // should error out with something like:
   
    Exiting: ... unknown unit " seconds" in duration "5 seconds" ...
   

   because "@every 5 seconds" should be "@every 5s". -e flag logs the errors in console instead of writing it to the log file. Also note that timeouts should be timeout. You can see a reference to all the available configuration here.
3. To test the setup, create a new file with just the following test monitor:

   - type: tcp
     id: test-tcp-monitor
     enabled: true
     name: TCP Test
     schedule: '@every 5m'
     hosts: [ "tcpbin.com" ]
     ports: [4242]
     timeout: 10s
   

   and run via heartbeat -c <path-to-config.yml> -e. In case of a failure, the command will exit with some message. In case of a successful run, you should see the first ping logged like (and so on after every scheduled time):

   {"monitoring":{"metrics":{"heartbeat":{"browser":{...},"http":{...},"icmp":{...},"scheduler":{...},"tcp":{"endpoint_starts":1,"endpoint_stops":1,"monitor_starts":1,"monitor_stops":1}}}}}
   

   Note that it's recommended to always use the id field and have schedule greater than the timeout.
4. Confirm that the Docs count has incremented.
5. Visit Uptime app and see if you see the pings listed.
6. Start adding your desired monitors and ensure that heartbeat -c <path-to-config.yml> -e doesn't exit with an error.

I am not sure why you are getting "Error fetching index ...". Delete the index pattern that you created manually and go through above steps (or if possible setup a fresh ES instance) and let us know how if you still get the error.

[BenKenobi (Ben)]

Hello @abdulz , Thanks for your great feedback with instructions.

I found the root cause: When starting the service and checking the status, I got the message that
the cfg file could not be parsed. So I disabled my own yml config file and put the information into the heartbeat.yml monitors inline part.

After restarting the service the error messages disappeared and the index was created.
Furthermore the index has now docs count greater nil:

What I would like to know is, if it is mandatory to install heartbeat on 1 server i.e. the Elasticsearch server or any other or is it also possible to install heartbeat on every server that shall be monitored?

Thanks

[abdulz (Abdul Zahid)]

@BenKenobi think of Heartbeat as an observer for measuring the uptime of your application. The goal is to measure uptime from the perspective of the consumers, whether they are end users or other machines.

You don't necessarily need to install Heartbeat on every individual server that you want to monitor. Instead, you should install Heartbeat on observer locations. These observer locations are where Heartbeat will periodically check the status of your servers and services. Multiple Heartbeat instances can write to a single ES for centralized monitoring, alerting and analysis.

If monitoring your application from a single observer location accurately represents its uptime for your use case, then one observer location is sufficient. However, in scenarios where you want to ensure uptime from multiple geographical regions or network segments, you can install Heartbeat on multiple observers to provide a more comprehensive view of your application's availability. To make this as simplest as possible, we introduced the new Synthetics app where defining and running a monitor from various location around the world is just a few clicks away. Synthetics app also allows you to run monitors from private servers while providing the same seamless GUI or project based workflow of defining and managing monitors.