Creating a watcher

[ChristianOelsner (Christian Oelsner Hansen)]

Hello guys.
I am still very much in a learning process when it comes to the elastic stack.
Now i am trying to set up a watcher that can be user for some sort of allerting/notification, be it slack or mail.
We have a lot of different teams producing logs. When some of those logs are malformed or otherwise invalid they are given some tags while being processed in logstash.
I am trying to sort out a watch that can react to these tags, but the information is kinda overwheming for a newcomer.
Can anyone here perhaps offer some advise on how to go about it?

Best regards
Christian Oelsner

[gmmorris (Gidi Morris)]

Hi @ChristianOelsner,
Welcome to the community.

I'm sure we can help you figure out how to leverage the stack to achieve this .

What you're describing should definitely be achievable using either Watcher or the Kibana Alerting framework once the new Search Alert is implemented (which is aimed for some time soon in the 7.x release cycle).
I mention the Alerting Framework as it is designed to be a little easier for users to pickup and use from within Kibana, but the specific alert you're looking for isn't quite ready yet.

Assuming we go down the Watcher route, as it's available in the current release, it sounds like you should start by figuring out the query that should back your Watch.
I don't know what you mean by "given some tags", but assuming you can author a query that can find these documents in your index, then we should be able to build a watch on top of it.

For example, suppose the query below returned all these tagged documents:

{
  "query" : {
    "match" : { "tag": "log_malformed" }
  }
}

Then you could create a watch called log_malformed_watch that detects these documents on a schedule every 10 seconds:

PUT _watcher/watch/log_error_watch
{
  "trigger" : {
    "schedule" : { "interval" : "10s" } 
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "logs" ],
        "body" : {
          "query" : {
            "match" : { "tag": "log_malformed" }
          }
        }
      }
    }
  }
}

If you're not sure how to construct this query, it might help if you gave us a few concrete examples of the documents you like to be able to watch for.

[ChristianOelsner (Christian Oelsner Hansen)]

Hi @gmmorris
Thanks for the welcome i suspect you guys will see a lot more of me in the times to come!
And thank you so much for that example, that will certainly get me started.

If you dont mind i will return to this thread for advice later on how to actually aleart the projects/teams who produces the malformed logs.

Have a great weekend and thanks again.

Christian Oelsner