Monitor uptime is not shown

Elastic Observability Synthetics
1

Hello!

Recently I started setting up a self hosted elastic stack (v8.6.1).
I implemented multiple browser monitors via the heartbeat yml config, and those monitor pings work like they should and show up in the bar chart under Observability. However, the total, up and down count of the monitors are always 0.
I only adjusted the bare minimum in the config for all of the elastic stack services, so I thought that I might have missed a setting. But sadly there are no errors or any other logs pointing me to a misconfiguration.

monitors
monitors1601×212 13.7 KB

I have set up elastic stack in docker containers, with each service running in a separate container:

  • Kibana
  • Elasticsearch
  • Logstash
  • Metricbeat
  • Heartbeat

I have a "Free and open - Basic" subscription.

Is this a known issue? Can anybody help me figure this issue out?
Thanks in advance!

2

Hello @Sam_Verhoeven this usually happens when clock is out of sync between the server where heartbeat is running vs the kibana.

Can you please check clock are in sync between kibana and heartbeat.

Can you share what config are you using?

Also try playing around with date picker and see if it makes a difference between selecting a relative time vs absolute.

Regards

3

Hello @shahzad31, thank you for your reply.
I checked the time on the Kibana, Elasticsearch & Heartbeat containers, but they are in sync (+- a second).
It doesn't make a difference switching the date picker to relative.

The config isn't anything special, mostly default config from the default elastic docker images.
(three dots mean I left out sensitive info)
Hearbeat config:

heartbeat.monitors:
  - type: browser
    ...

heartbeat.jobs:
  browser.limit: 5

output.elasticsearch:
  hosts: "..."

Elasticsearch config:

cluster.name: "docker-cluster"
network.host: ...
transport.host: ...

discovery.type: "single-node"

xpack.security.enabled: false

Kibana config:

server.name: kibana
server.host: "..."
elasticsearch.hosts: [ "..." ]
elasticsearch.username: ...
server.ssl.enabled: false
xpack.monitoring.ui.container.elasticsearch.enabled: true
xpack.encryptedSavedObjects.encryptionKey: "..."
xpack.reporting.encryptionKey: "..."
xpack.security.encryptionKey: "..."

It think this issue is also affecting alerts, because I would think no alert will be triggered if it thinks that we always have 0 down. (Haven't had any real downtime since setting the monitors up, so no way to know for sure yet.)

Let me know if you have any other clue towards a possible fix.

4

Another issue could be the mapping issue. Can you please share your heartbeat index mappings.

you can try troubleshooting guide provided here Troubleshoot mapping issues | Observability Guide [8.6] | Elastic

5

Thank you, I will take a look at that guide.
I can now also confirm alerts based on the uptime also don't work.

6

Based on the following line from the guide, I don't think the guide will solve my issue:

This can occur when the underlying heartbeat-VERSION ILM alias is deleted manually or when Heartbeat writes data through an intermediary such as Logstash without the setup command being run.

I have no manual indexes, only the ones that were automatically created. The one that holds the data is called ".ds-synthetics-browser-default-2023.04.14-000001" which is based on the "synthetics-browser-default" data stream.
The ".ds-heartbeat-8.6.1-2023.04.14-000001" index, based on the "heartbeat-8.6.1" data stream holds no data.
But from what I can tell the uptime should also take the synthetics index into account.

image
image823×264 16 KB

Here's the index mapping for ".ds-synthetics-browser-default-2023.04.14-000001" index:

{
  "mappings": {
    "_data_stream_timestamp": {
      "enabled": true
    },
    "dynamic_templates": [
      {
        "match_ip": {
          "match": "ip",
          "match_mapping_type": "string",
          "mapping": {
            "type": "ip"
          }
        }
      },
      {
        "match_message": {
          "match": "message",
          "match_mapping_type": "string",
          "mapping": {
            "type": "match_only_text"
          }
        }
      },
      {
        "strings_as_keyword": {
          "match_mapping_type": "string",
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      }
    ],
    "date_detection": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "agent": {
        "properties": {
          "ephemeral_id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "browser": {
        "properties": {
          "experience": {
            "properties": {
              "cls": {
                "type": "float"
              },
              "dcl": {
                "properties": {
                  "us": {
                    "type": "float"
                  }
                }
              },
              "fcp": {
                "properties": {
                  "us": {
                    "type": "float"
                  }
                }
              },
              "lcp": {
                "properties": {
                  "us": {
                    "type": "float"
                  }
                }
              },
              "load": {
                "properties": {
                  "us": {
                    "type": "float"
                  }
                }
              }
            }
          },
          "relative_trace": {
            "properties": {
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "score": {
                "type": "float"
              },
              "start": {
                "properties": {
                  "us": {
                    "type": "float"
                  }
                }
              },
              "type": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          }
        }
      },
      "data_stream": {
        "properties": {
          "dataset": {
            "type": "constant_keyword",
            "value": "browser"
          },
          "namespace": {
            "type": "constant_keyword",
            "value": "default"
          },
          "type": {
            "type": "constant_keyword",
            "value": "synthetics"
          }
        }
      },
      "ecs": {
        "properties": {
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "error": {
        "properties": {
          "code": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "message": {
            "type": "match_only_text"
          },
          "stack_trace": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "event": {
        "properties": {
          "dataset": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "host": {
        "type": "object"
      },
      "monitor": {
        "properties": {
          "check_group": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "duration": {
            "properties": {
              "us": {
                "type": "long"
              }
            }
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "project": {
            "properties": {
              "id": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "status": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "timespan": {
            "properties": {
              "gte": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "lt": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "num_journeys": {
        "type": "long"
      },
      "os": {
        "properties": {
          "platform": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "package": {
        "properties": {
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "state": {
        "properties": {
          "checks": {
            "type": "long"
          },
          "down": {
            "type": "long"
          },
          "duration_ms": {
            "type": "long"
          },
          "ends": {
            "properties": {
              "checks": {
                "type": "long"
              },
              "down": {
                "type": "long"
              },
              "duration_ms": {
                "type": "float"
              },
              "id": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "started_at": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "status": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "up": {
                "type": "long"
              }
            }
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "started_at": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "status": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "up": {
            "type": "long"
          }
        }
      },
      "summary": {
        "properties": {
          "down": {
            "type": "long"
          },
          "up": {
            "type": "long"
          }
        }
      },
      "synthetics": {
        "properties": {
          "error": {
            "properties": {
              "message": {
                "type": "match_only_text"
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "stack": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "index": {
            "type": "long"
          },
          "journey": {
            "properties": {
              "id": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "package_version": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "payload": {
            "properties": {
              "network_conditions": {
                "properties": {
                  "downloadThroughput": {
                    "type": "long"
                  },
                  "latency": {
                    "type": "long"
                  },
                  "offline": {
                    "type": "boolean"
                  },
                  "uploadThroughput": {
                    "type": "long"
                  }
                }
              },
              "source": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "status": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "text": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "type": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "url": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "step": {
            "properties": {
              "duration": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              },
              "index": {
                "type": "long"
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "status": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "url": {
        "properties": {
          "domain": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "full": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "path": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "port": {
            "type": "long"
          },
          "scheme": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      }
    }
  }
}
7

it is indeed mapping issue

this fields mappings are not correct

      "timespan": {
        "properties": {
          "gte": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "lt": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },

instead it should be like this

        "timespan": {
          "type": "date_range"
        },

Regards

8

I will also suggest using monitor management and private locations to add monitors via ui or project monitors in a self hosted environment.

9

I did the following steps:

  1. Stopped the heartbeat container.
  2. Removed the synthetics data streams and indexes.
  3. Altered the index template for the synthetics data stream.
  4. Restarted the heartbeat container.

I can now see that the timespan field is indexed differently, but still the amount of monitors, up or down is at 0. So I'm guessing that there is some other index mapping that needs to be done, that I'm unaware of. I sadly have no clue where to look as to what every field is expected to be mapped as for the uptime checking.

10

What do you mean by

  1. Altered the index template for the synthetics data stream.

Are you setting up index template manually?

you can use heartbeat command ./heartbeat setup to do template setup.

11

I didn't set up the index manually.
What I did was go to "Index Management" in Kibana and altered the mapping of the "synthetics" index pattern that was automatically created (probably because the heartbeat data stream was detected).
It's just a bit strange that the default synthetics data stream and index is not able to be interpreted by the uptime feature.

12

I will suggest stop heartbeat, remove all heartbeat indices and templates.
Set template with ./heartbeat setup. Verify timespan mapping.

Restart heartbeat. That should hopefully fix it.

Yeah unfortunately this can happen sometimes. This is why we moved to using elastic agent. Which prevents issues like this.

13

I will try the ./heartbeat setup. Thank you.

The reason I'm not keen on using the elastic agent, is because it only adds complexity to my implemention and has no real added value for now. But I will keep the approach in mind.

14

I sadly haven't been able to fix the issue.
It might be handy for me to take a look at the controller handling the "/internal/uptime/snapshot/count" api request. Then I could check which fields are expected in the index.
Could you perhaps point me towards the code in git that should handle this, because I wasn't able to find it by looking through the github repo?
Thanks in advance!

15

Sorry for the spam.
But I have figured out that this issue only occurs for browser monitors. So the heartbeat index is created properly (automatically).
I guess it's not unusual to have something that's still in beta (synthetics/browser monitors) to not be working properly. But that said, I hope the synthetics index generation gets fixed soon!

16

So your http/tcp monitors are showing properly in uptime ?

17

@Andrew_Cholakian1 Do you have any insight on this. Why mapping or index generation would be broken for browser monitors when using heartbeat.

18

Correct! I added an icmp and an http monitor for testing purposes, and those got picked up in the uptime counter immediately. A downtime alert based on a rule I created, was also triggered immediately.

19

Hello. Any update regarding this issue?
Thanks in advance.

20

@Sam_Verhoeven i am going to try to reproduce locally.