Adding/editing fields through watcher

Elastic Stack Elasticsearch
1

I have this nut that I've been trying to crack for a while now.

I'm wondering if it is possible to edit a field or add a tag of the ES documents that match a watcher query. Ideally, when a log comes into my pipeline that is urgent I want to do two things:

  1. Send an email to myself to alert that the error happened -- I can do this no problem

  2. Put into the "tags" of the log that it has been "AlertedToMe". I'd either like to do this or change an existing field of the log to say something along those lines. I want to do this because I'd like to remove the errors that I've already seen, from my visualizations by simply querying: -tags:AlertedToMe and hiding all of the logs that I've seen rather than typing each log I don't want to see into the query box (I have hundreds of different ones to go through).

Is there a way to have watcher do this type of active tagging on documents in an index? Can I do it through "transforms"?

2

Can anyone help me figure this out?

3

Hey,

how about using a webhook, that runs an update by query?

--Alex

4

Hmm yeah, that looks like a splendid way to solve this problem! Thanks!!

5

So I've made a watch that has an update by query, but the main issue with this is that I then have to have two queries in the same watch which is a bit too unwieldy for me (I have to be making these watches daily and it seems redundant to have to change several parts of the watch for the same change).In other word, for the update on query, I need to query a second time because it doesn't seem to interact with the input data to the watch at all.

In my head, I want to do two things: input documents that match a query then change one field in those documents. But this isn't as easy as I thought it would be. Especially because it feels like the documentation is all over the place with the different languages and the way to call things and insufficient descriptions of how things work. Here's what I've assembled so far. It works but seems redundant:

POST _xpack/watcher/watch/ActiveTagExample
{
  "trigger" : {
    "schedule" : { "interval" : "10s" } 
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "<logstash-{now/d}>" ],
        "body" : {
          "_source" : "Summary",
          "query" : {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1d/m"
                    }
                  }
                },
                {
                  "match_phrase" : { "Summary": "Message of error123"
                  }
                }
              ]
            }
          }
        }
      }
    }  
  },
  
  "condition" : {
    "compare" : { 
      "ctx.payload.hits.total" : { "gt" : 0 
      }
    }
  },
  "actions" : {
    "ActiveTagger" : {
      "webhook" : {
      "method" : "POST",
      "host" : "localhost",
       "port" :9200,
      "path" : "/{{ctx.payload.hits.hits.0._index}}/_update_by_query",
      "body" : "{\"script\" : {\"inline\": \"ctx._source.tags = 'TAGGED' \"}, \"query\": {\"bool\": {\"must\": [{\"query_string\": {\"query\": \"Summary:'Message of error123'\"}}]}}}"
      }
    }
  }
}

If I take away the query part of the _update_by_query, then every document in ES gets changed vs just the ones I've inputted. In other words if I take this away:

 \"query\": {\"bool\": {\"must\": [{\"query_string\": {\"query\": \"Summary:'Message of error123'\"}}]}}}

After reading a lot of documentation, I still have no idea the use of transform. Does it have a use here? Or what is it used for? Or maybe I can have one params go across the whole watch? Any suggestions to make this easier would be tremendously appreciated

6

Hey,

so the input of the watch (and the search result) can be used as input for the email, but the update by query needs to be its own query (and thus also its own definition). I havent understood why this is a problem exactly, maybe you could elaborate on that as well.

Thank you!

--Alex

7

Doing it this way is quite hard to work with for the system I'm creating and I would love to know if there is another way that I only need to type the query that I'd like to replace once.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.