How to visualize/query hosts not sending logs in over an hour?

[Brandon_Reeves]

I am trying to build a query, dashboard, or something to solve an issue. We have multiple hosts sending data to elasticsearch via logstash.

What we would like to do is determine quickly if elasticsearch has NOT received logs from a specific source (host or hosts) for in the last hour, day etc. This would allow us to view in Kibana if a host has failed, services are stopped, network issues are occurring and help start the troubleshooting process.

Anyone have any feedback re: this?

Our indexes are in the format logstash-YYYY-MM-DD and we have hostname (host) and date in the fields collected.

Thanks in advance

[cjcenizal]

Hi Brandon, have you taken a look at our Watcher API? It's part of X-Pack, and it will allow you to get notified when your the state of your logs change in the way you describe.

What you'll do is set up a watch which queries against the index where your logs are stored. You can define a condition which will compare the time-based field from that index again the watch's execution time. If that difference is over some threshold, you can tell the watch to execute an action, which in your case would probably be to send you an email.

If you want to learn more about Watcher, we have a great webinar and also some examples online.

Hope this helps,
CJ

[Brandon_Reeves]

I havent looked at Watcher yet for this. Honestly didnt think about Watcher as a solution but am open to it. Really just want a simple solution to let us know if logs havent been received by a host or multiple hosts.

We are planning on using the elastic cloud once in production so we will have watcher.

Could you provide any specific example of how we could accomplish this as the demo examples seem pretty basic.

Thanks

[cjcenizal]

Hi Brandon, a really simple solution to your problem would be a visualization built in Visualize which shows you a date histogram of logs received over time. For example, you could create an Area Chart, lock the time period to be from "now" to "now minus {period of time}" and check it periodically. If there's a drop-off at the end of the chart then you'll know you haven't been receiving logs.

To create such a chart, go to the Visualize app, select Area Chart and the relevant index pattern, and add an X-axis Bucket set to "Date histogram" aggregation type. You can set the time picker in the top right corner of the screen to set the time range.

Take a look at this example about creating a watch that sends you an email. It seems to be very similar to your use case -- you'd just need to tweak the query in the watch to request all logs (instead of ones filtered on the response) and use ctx.execution_time to see if any logs fall into the window you're interested in (other variables you can access are listed here). Here's an example which checks for logs from the past day and sends an email if you have 0:

PUT _xpack/watcher/watch/check-for-logs
{
  "trigger" : {
    "schedule" : { "cron" : "0 0/1 * * * ?" }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [
          "your-index-pattern-*"
        ],
        "body" : {
          "query" : {
            "bool" : {
              "filter" : {
                "range": {
                  "@timestamp": {
                    "from": "{{ctx.trigger.execution_time}}||-1d",
                    "to": "{{ctx.trigger.execution_time}}"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition" : {
    "compare" : { "ctx.payload.hits.total" : { "eq" : 0 }}
  },
  "actions" : {
    "email_admin" : {
      "email" : {
        "to" : "your@email.com",
        "subject" : "No logs created in the past 24 hours"
      }
    }
  }
}

Does this help?

Thanks,
CJ