Hi
I've added a monitor and its giving continuous false positive results
I want to know why it is checking 2 times at every interval time
A first its showing correct and then againg checks and shows FP results
Hi, by default Synthetics monitors will automatically retest if a test fails. This is to limit false positives because of transient issues, such that may occur over the internet.
In your screenshot, this looks like a test at 15:16:19 failed so the retest automatically ran 1 second later, and the retest was successful.
The same thing then happens 3 minutes later at 15:19:19 (suggesting a 3 minute frequency). The UI should represent this with an icon that depicts the test was run because it’s a retest (as opposed to a natural scheduled test):
This does not show in your screenshot, what version of Kibana are you using?
If you don’t want this behaviour (i.e. don’t retest on failure), you can change the default.
If you have configured your monitors via the UI, toggle the Enable retest on failure option:
If you are using Project Monitors, set this via the retestOnFailure parameter (docs).
One thing that stands out in your screenshot is that the failure is pretty quick (under 1 second), with the successful retest taking > 9 seconds. Perhaps this is indicative of a problem with the endpoint you are hitting e.g. if you tried something like curl, does the first test fail then a second in quick succession, pass?
The retest feature didn't come in until 8.11, so your screenshot aligns with that (you won't have the option for retest). In theory, your testing infrastructure shouldn't be retesting either (because it didn't exist as a feature in 8.8).
Are you running your monitors on Private Locations, or the Elastic Managed Testing Infrastructure?
If on Private Locations, what version of Elastic Agent are you running? In theory, if these are also on 8.8.x, they shouldn't be attempting to retest on failure.
I am unclear if you are running your own agents (i.e. private locations), or Elastic provided ones?
Is your Elastic stack self-managed or on ESS ? If self-managed, then these must be private locations. Make sure you are running the elastic-agent-complete docker image to match your Elastic stack (8.8.1). Don't try and use a newer version of the Elastic Agent.
@PaulB-Elastic
Running own elastic agents and Elastic stack self-managed.
elastic agent are also same version as elk
I have to admit I am a little unsure what's happening here.
I have tested an Elastic 8.8.1 stack with 8.8.1 Elastic Agent, and it's behaving as I would expect (a DOWN result does not automatically run another test).
As you can see here, my 3 minute monitor runs every 3 minutes (whether up or down).
Can you confirm you using the elastic-agent-complete docker image for your private location?
As I understand, you have set up an HTTP monitor to run every 3 minutes. Does every initial test (at the 3 minute mark) fail and then there is a (pretty much immediate) second test that is successful?
I wonder if for some reason Heartbeat is restarting during the failure, which results in another immediate test (which happens to be up)? Is your docker container continually running throughout this?
Hi @theacodes,
Are the IPs pinged in the duplicate tests different?
If so, you might have enabled ping-all mode and your endpoint might resolve to multiple IPs. You'll find it inside Edit Monitor -> Advanced options:
If that's not the case, can you make sure that there's only one agent enrolled in the private location policy? Having more than one agent enrolled in a private location policy can lead to duplicate runs. You'll find that in Fleet -> Agent Policies:
Hi @emilioalvap @PaulB-Elastic
I'm confused as well. I've set up a new instance on 8.12 on my local machine VM. (Not installed heartbeat only added the monitor in Synthetics app)
and it's working fine, Just like you showed @PaulB-Elastic
For my current installation of ELK v8.8.1
I've added monitors in Uptime and then later managed it under synthetics.
I have enabled ping-all mode and the endpoint resolves to single IPs.
Can you confirm you using the elastic-agent-complete docker image for your private location?
No. I've installed it via the deb package. No docker was used. both for my new VM instance & current v8.8.1 instance.
Hi @theacodes,
Can you please confirm that only one agent is running under the private location policy? You can find the instructions on how to find that information in my comment above:
If that's not the case, can you make sure that there's only one agent enrolled in the private location policy? Having more than one agent enrolled in a private location policy can lead to duplicate runs. You'll find that in Fleet -> Agent Policies :
Hi @theacodes,
Each agent enrolled in the private location policy will execute the configured checks independently. If you have two agents enrolled, you'll get duplicate checks.
Edit: Please read through our docs on private location scaling:
Do not run the same agent policy on multiple agents being used for Private Locations, as you may end up with duplicate or missing tests. Private Locations do not currently load balance tests across multiple Elastic Agents. See Scaling Private Locations for information on increasing the capacity within a Private Location.