Duplicate Documents Generated by Kibana for ICMP Monitor Down and Recovery Statuses

[Adriann (Adrian)]

Hello,

I've been experiencing an issue where Kibana's monitoring is generating duplicate documents for both 'down' and 'recovery' statuses across all hosts. Below, I have provided the configuration of the rule and additional details. Currently, I am on Kibana version 8.11.4, but this issue has been occurring since version 8.7.0. I'm uncertain if this is a configuration error or a bug in Kibana.

There have been no changes made to Kibana's configuration during this period. The only significant change was relocating the Heartbeat module to a new server location. I do not observe any duplicate documents in the Heartbeat index itself. I'm at a loss as to where to look for a solution and would greatly appreciate any help.

I will attach screenshots of the configurations in Kibana and Heartbeat, as well as the indexed documents from Heartbeat and the monitoring alerts to provide further context.

DOCUMENTS INDEX BY KIBANA ALERT RULE

RULE CONFIGURATION

HEARTBEAT CONFIG

- type: icmp
  id: PolXXX.254
  name: PolXXX.254
  hosts: ["XXX.254"]
  enabled: true
  schedule: "@every 15s"
  ipv4: true
  ipv6: false
  mode: any
  timeout: 2s
  wait: 5s
  tags: ["poXXX", "network", "availability"]
  fields_under_root: true
  fields:  {host.ip: "XXX.254", host.bo: "PolXXX", "host.hostname": "PolXXX.254"}

Thank you in advance for any assistance or insights you can provide.

[Adriann (Adrian)]

I don't know how exactly this works but this is the only difference besides @timestamp that I can see in the duplicated documents.

[Adriann (Adrian)]

I found a "solution". Generally, by cloning the rule and removing the old one, duplication stopped. I think the issue might have been related to moving the Heartbeat, which somehow caused hosts with the same IDs to be perceived as different entities. Consequently, the same monitor was tracking the state of the same host twice. This process of cloning and removing the old rules seems to have reset the configuration, effectively resolving the issue of duplication. It's an interesting situation that suggests there might be a subtle bug or a configuration quirk related to the Heartbeat's relocation.