Synthetics Monitoring

Elastic Observability Synthetics
1

Hi All,

We are running the elk stack version 8.13.4 and have recently started using synthetics to monitor a few services. We could see that the data after each poll is being saved in indices backed by synthetics datastream. The issue we are facing is regarding the monitors which were deleted from the GUI but still report. We could see the data coming for these monitors under discover (synthetics data view). The documentation says that the API support is present from 8.14 onwards. Given this context I have the following two questions

  1. Why do deleted monitors keep on sending data? Is it a known bug?
  2. How can we permanently delete these monitors so that we do not see any data in the synthetics dataview
1 Like
2

@Ankita_Pachauri Deleted monitors should never report any data. can you please tell me if you are seeing data from private locations or elastic managed locations?

Maybe you have those monitors in difference kibana space?

if they are on private locations, can you see when you deleted those, maybe somehow related integeration policies didn't get deleted? you can check those in Synthetics Integeration in fleet app.

3

Hi,
I have deleted the monitor from private location but it still reports data.

Is there any way to check if any stale process left and how can it be deleted.

4

@Ankita_Pachauri can you please check if you have any integration policies in in this view

image
image3450×1360 214 KB

5

Hi,
Yes, I have checked and found nothing. Still, the data is coming but the tags field is coming as blank.

image
image1452×638 43 KB

6

Hello, have you checked if the configurations in heartbeat.yml are updated without the monitor?

7

Hi,
Where can I find heartbeat.yml file since I am using synthetics.

8

If you followed this tutorial (Create monitors with Project monitors | Elastic Observability [8.14] | Elastic), it will be in the project folder.

If in doubt, if you are running on Linux or Mac, search for the file using the command:

grep -r "heartbeat.monitors" /path/to/folder --include \*.yml

or

find /path/to/folder  -name "heartbeat.yml"
9

Hi,
Thanks for your response.
I am using Synthetics app via Kibana GUI and not the project monitors. Also, I tried using the find command and found the heartbeat.yml file, however it is the sample file that comes along with the beats package.

image
image956×27 1.77 KB

10

Hi,
Can anyone help on this?

11

Hi Team,
Is anyone aware of this?

12

Hi @Ankita_Pachauri,

I would recommend starting by locating the elastic agent that is sending the events for the deleted monitor. You should be able to do so by looking at the agent.* field family in discover:

image
image932×259 18.7 KB

Once you locate the agent, the first step is checking whether this is a Fleet-managed agent. If so, you should be able to find it in Kibana -> Fleet -> Agents.

Two options here; 1) the agent is not in the list, so it's not managed by Fleet. In this case, there's likely a standalone agent deployed with a static configuration. You'll need to find where the agent is deployed and update the configuration accordingly.
2) agent is managed by Fleet. I'd start by checking that the agent is enrolled in agent policy pertaining to a private location and it's status:

image
image765×138 9.44 KB

Check that the agent policy does not include any inputs of type: synthetics/<browser, http, tcp, icmp>. If it does not contain any synthetics inputs, request an elastic agent diagnostics package and make sure that the returned configuration does not contain any synthetics inputs as well.