Elasticsearch indexes - watcher notifications

Hi All,

Iam using elastic stack 5.5 to monitor the netflow and sflow and I have two separate indexes , I view them in kibana , for search and dashboards.

If i stop receiving the logs in the indexes, from logstash or the source itself stop sending the logs, in real time monitoring how can I use watcher to get notify me in email that this particular indexe is not getting events or logs ,so that I could necessary steps to see if the source is offline or my logstash got crashed.

Since I cant monitor the kibana in real time 24/7.

Any help would be really helpful
Thanks in advance,
Raj

hey,

you could have a watch that executes a search, that returns only the newest timestamp (either by document search or by an max aggregation). If that timestamp is older than five minutes, you could trigger an alert.

--Alex

1 Like

Hi Alex,

Thank you for the reply,

lets say i have a feild says , host and its value is some ip like 10.10.10.101,how do i write the watch alerts , if there is no document host is equal 10.10.10.101, in last 5 mins ?

Raj

The watch you are looking for sounds similar in structure to the example provided here.

1 Like

Hi Dave,

Thank you for the reply,when am trying to execute it says

You need to define the scripts in someway as described here.

Or put insert them into the watch, which isn't as graceful, but works for testing. I just did something similar with the same watch as a starting point.

"condition": {
    "script": {
      "inline": "return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();",
      "lang": "painless"
    }
  },
1 Like

Hi Dave,

Thank you for the reply,when am trying to execute it says

"exception": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();",
" ^---- HERE"
],
"script": "return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();",
"lang": "painless",
"caused_by": {
"type": "null_pointer_exception",
"reason": null,
"stack_trace": "java.lang.NullPointerException\n\tat org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:202)\n\tat org.elasticsearch.painless.PainlessScript$Script.execute(return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();:25)\n\tat org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:133)\n\tat org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:94)\n\tat org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:84)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:391)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:492)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:748)\n"
},
"stack_trace": "ScriptException[runtime error]; nested: NullPointerException;\n\tat org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:101)\n\tat org.elasticsearch.painless.PainlessScript$Script.execute(return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();:8)\n\tat org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:133)\n\tat org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:94)\n\tat org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:84)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:391)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:492)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.NullPointerException\n\tat org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:202)\n\tat org.elasticsearch.painless.PainlessScript$Script.execute(return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();:25)\n\t... 10 more\n"
}
}

The example I gave had different aggregation names.

"condition": {
    "script": {
      "inline": "return ctx.payload.aggregations.periods.buckets.history.hosts.buckets.size() > ctx.payload.aggregations.periods.buckets.last_period.hosts.buckets.size();",
      "lang": "painless"
    }
  },
2 Likes

Thank you so muck for the reply :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.