Cloud elastic Alerting not working as expected

Elastic Stack Elasticsearch
1

Hi Team,

We have setup watcher on our monitoring elasticsearch server. We are monitoring on ".monitoring-es-*" indices with following mentioned watcher definition, but it always getting triggered so I believe compare stage is not working properly.

"trigger": {
"schedule": {
  "interval": "10s"
}
},
"input": {
"search": {
  "request": {
    "search_type": "query_then_fetch",
    "indices": [
      ".monitoring-es*"
    ],
    "types": ["doc"],
    "body": {
      "query": {
        "match": {
          "type": "cluster_stats"
        }
      },
      "sort": [
        {
          "timestamp": {
            "order": "desc"
          }
        }
      ],
      "_source": [
        "cluster_stats"
      ],
      "size": 1
    }
  }
}
},
"condition": {
"compare": {
  "ctx.payload.hits.hits.0._source.cluster_stats.indices.store.size_in_bytes": {
    "gt": 1000000000
  }
}
},
"actions": {
"send_email": {
  "email": {
    "profile": "standard",
    "to": [
      "XXXXXXXXX"
    ],
    "subject": "[Disk Usage Watcher]Elastic cloud Watcher Notification",
    "body": {
      "text": "You Elastic cloud cluster disk usage is more than 10GB 
 {{ctx.payload.hits.hits.0._source.cluster_stats.indices.store.size_in_bytes}}"
    }
   }
  }
},
"throttle_period_in_millis": 2160000
2

can you share the output of the execute watch API in a gist, please?

3

Hi Alexander,

I tried simulate but at that time I see "status":"success" and "met": falsedue to which action email is not getting triggered.

But once I set it got trigger it sends email even if the condition of 10GB is not satisfied.

4

can you also paste one of the entries from the watcher history index for this watch, then we can check if the condition was triggered or not.

5

I am seeing conditions where not met and triggering was not done for few. I am checking one of my other cluster where actually I am getting false trigger emails.

I will be able to share in sometime.

But mean while I also checked that I could not actually query fields which are added in compare section of watchers. Is there are specific reasons for them?

6

Hi,

Here is the link for the event where we are getting false trigger.

One thing which I noted is that condition which matches is in string while we have created watcher with integer value.

7

Hi Alexander,

Let me know your inputs for above issue.

8

Hey,

can you replace your condition with

    "condition": {
      "script": {
        "source": "return ctx.payload.foo > 150000000000L"
      }
    },

and report back if that works? You might have hit a bug in the compare condition.

--Alex

9

I think your last example fails because you put the number into ticks, making it a string and thus doing a string based comparison, not a number based one

"gt": "150000000000"
10

Thanks Alexander for your inputs issue has been resolved after removing quotes.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.