Elasticsearch indexes - watcher notifications

Elastic Stack Elasticsearch
1

Hi All,

Iam using elastic stack 5.5 to monitor the netflow and sflow and I have two separate indexes , I view them in kibana , for search and dashboards.

If i stop receiving the logs in the indexes, from logstash or the source itself stop sending the logs, in real time monitoring how can I use watcher to get notify me in email that this particular indexe is not getting events or logs ,so that I could necessary steps to see if the source is offline or my logstash got crashed.

Since I cant monitor the kibana in real time 24/7.

Any help would be really helpful
Thanks in advance,
Raj

2

hey,

you could have a watch that executes a search, that returns only the newest timestamp (either by document search or by an max aggregation). If that timestamp is older than five minutes, you could trigger an alert.

--Alex

3

Hi Alex,

Thank you for the reply,

lets say i have a feild says , host and its value is some ip like 10.10.10.101,how do i write the watch alerts , if there is no document host is equal 10.10.10.101, in last 5 mins ?

Raj

4

The watch you are looking for sounds similar in structure to the example provided here.

5

Hi Dave,

Thank you for the reply,when am trying to execute it says

6

You need to define the scripts in someway as described here.

Or put insert them into the watch, which isn't as graceful, but works for testing. I just did something similar with the same watch as a starting point.

"condition": {
    "script": {
      "inline": "return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();",
      "lang": "painless"
    }
  },
7

Hi Dave,

Thank you for the reply,when am trying to execute it says

"exception": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();",
" ^---- HERE"
],
"script": "return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();",
"lang": "painless",
"caused_by": {
"type": "null_pointer_exception",
"reason": null,
"stack_trace": "java.lang.NullPointerException\n\tat org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:202)\n\tat org.elasticsearch.painless.PainlessScript$Script.execute(return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();:25)\n\tat org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:133)\n\tat org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:94)\n\tat org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:84)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:391)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:492)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:748)\n"
},
"stack_trace": "ScriptException[runtime error]; nested: NullPointerException;\n\tat org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:101)\n\tat org.elasticsearch.painless.PainlessScript$Script.execute(return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();:8)\n\tat org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:133)\n\tat org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:94)\n\tat org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:84)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:391)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275)\n\tat org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:492)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.NullPointerException\n\tat org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:202)\n\tat org.elasticsearch.painless.PainlessScript$Script.execute(return ctx.payload.hosts.aggregations.hosts.buckets.size() != ctx.payload.history_hosts.aggregations.hosts.buckets.size();:25)\n\t... 10 more\n"
}
}

8

The example I gave had different aggregation names.

"condition": {
    "script": {
      "inline": "return ctx.payload.aggregations.periods.buckets.history.hosts.buckets.size() > ctx.payload.aggregations.periods.buckets.last_period.hosts.buckets.size();",
      "lang": "painless"
    }
  },
9

Thank you so muck for the reply :slight_smile: