Uptime showing no uptime moniters but I do see heartbeat info in discovery tab

I recently installed elasticsearch / kibana / hearbeat 8.9.1 (each on a separate server)

created automated terraform and ansible to do that.

I configured heartbeat and was able to see the information for one day in the "uptime" tab.

A day passed by and I no longer see anything In the uptime tab but I can see the data inside discover tab being updated regularly.

what am I missing?

heartbeat config file

################### Heartbeat Configuration Example #########################

# This file is an example configuration file highlighting only some common options.
# The heartbeat.reference.yml file in the same directory contains all the supported options
# with detailed comments. You can use it for reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/heartbeat/index.html

############################# Heartbeat ######################################

# Define a directory to load monitor definitions from. Definitions take the form
# of individual yaml files.
heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: true
  # How often to check for changes
  reload.period: 5s

# Configure monitors inline
heartbeat.monitors:
  - type: http
    # Set enabled to true (or delete the following line) to enable this example monitor
    enabled: false
    # ID used to uniquely identify this monitor in elasticsearch even if the config changes
    id: my-monitor
    # Human readable display name for this service in Uptime UI and elsewhere
    name: My Monitor
    # List or urls to query
    urls: ["http://localhost:9200"]
    # Configure task schedule
    schedule: '@every 10s'
    # Total test connection and data exchange timeout
    #timeout: 16s
    # Name of corresponding APM service, if Elastic APM is in use for the monitored service.
    #service.name: my-apm-service-name

# Experimental: Set this to true to run heartbeat monitors exactly once at startup
#heartbeat.run_once: true

# ======================= Elasticsearch template setting =======================

output.elasticsearch.index: "%NODENAME%-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.name: "%NODENAME%-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.pattern: "%NODENAME%-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "%KIBANAHOST%"
  username: "elastic"
  password: "%ELASTICPASSWORD%"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Heartbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.inventory.yml` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of inventory.yml to connect to.
  hosts: ["%ELASTICHOSTONE%","%ELASTICHOSTTWO%"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "%ELASTICPASSWORD%"
  ssl:
    enable: true
    ca_trusted_fingerprint: "%CAFINGERPRINT%"
  # ------------------------------ Logstash Output -------------------------------
  #output.logstash:
  # The Logstash inventory.yml
  #inventory.yml: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
# ================================= Processors =================================

processors:
  - add_observer_metadata:
    # Optional, but recommended geo settings for the location Heartbeat is running in
    #geo:
    # Token describing this location
    #name: us-east-1a
    # Lat, Lon "
    #location: "37.926868, -78.024902"


    # ================================== Logging ===================================

    # Sets log level. The default log level is info.
    # Available log levels are: error, warning, info, debug
    #logging.level: debug

    # At debug level, you can selectively enable logging only for some components.
    # To enable all selectors use ["*"]. Examples of other selectors are "beat",
    # "publisher", "service".
    #logging.selectors: ["*"]

    # ============================= X-Pack Monitoring ==============================
    # Heartbeat can export internal metrics to a central Elasticsearch monitoring
    # cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
    # reporting is disabled by default.

    # Set to true to enable the monitoring reporter.
    #monitoring.enabled: false

    # Sets the UUID of the Elasticsearch cluster under which monitoring data for this
    # Heartbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
    # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
    #monitoring.cluster_uuid:

    # Uncomment to send the metrics to Elasticsearch. Most settings from the
    # Elasticsearch output are accepted here as well.
    # Note that the settings should point to your Elasticsearch *monitoring* cluster.
    # Any setting that is not set is automatically inherited from the Elasticsearch
    # output configuration, so if you have the Elasticsearch output configured such
    # that it is pointing to your Elasticsearch monitoring cluster, you can simply
    # uncomment the following line.
    #monitoring.elasticsearch:

    # ============================== Instrumentation ===============================

    # Instrumentation support for the heartbeat.
    #instrumentation:
    # Set to true to enable instrumentation of heartbeat.
    #enabled: false

    # Environment in which heartbeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server inventory.yml to report instrumentation results to.
    #inventory.yml:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

One more strange thing.
It shows the monitor uptime only from the "Last 24 hours" and above.

Meaning:

Today - not showing
This week - not showing
Last 15 minutes - not showing
Last 30 minutes - not showing
Last 1 hour - not showing.

Last 24 hours and above - working fine.

Hi @LiRi_Elgozi,

Thanks for bringing up the issue.

Can you tell us what indices are configured for Uptime documents and what indices/data views are used to query documents in Discover?

You can find the Uptime indices in Uptime -> Settings -> Heartbeat indices. For Discover, you can share a screenshot where you see the documents which you don't see in Uptime.

I am looking into the possibility of a similar scenario we discovered a few days back.

I re-created it so it would be more "generic" but if I remember correctly the indices was

%NODENAME%-%{[agent.version]}-%{+yyyy.MM.dd}
so it looked something like:
heartbeat-8.9.1-2023-08-28

regarding to the view - it looked something like this:


I now testing it with the default setting to check if it changes anything.

Looks like after using the default settings I commented out these:

# ======================= Elasticsearch template setting =======================

#output.elasticsearch.index: "filebeat-%NODENAME%-%{[agent.version]}-%{+yyyy.MM.dd}"
#setup.template.name: "filebeat-%NODENAME%-%{[agent.version]}-%{+yyyy.MM.dd}"
#setup.template.pattern: "filebeat-%NODENAME%-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

So now the configuration actually looks like this:

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

and my Indices looks "empty" now, which had this structure
%Service%-%NODENAME%-%{[agent.version]}-%{+yyyy.MM.dd}

Service = filebeat/heartbeat/metricbeat etc.
NODENAME = server name
agent.version = in mycase was 8.9.1
yyyy.MM.dd = in my case was 2023-08-28

So now, a day after, it is still working. So I'm happy:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.