No alert form array results

Elastic Stack Elasticsearch
1

I am trying to create a watcher condition, the result of my query is:
{ 0= {_type=test, _source= {timetaken=46, [...] } }, 1={_type=test, _source= {timetaken=1222, [...] } }, 2={_type=test, _source= {timetaken=5000, [...] } }, [...] }
I goal is to take all results with timetaken > 2000
I wrote this condition, and it is working :
"condition" : { "compare" : { "ctx.payload.hits.hits.0._source.timetaken" : { "gt" : 2000 }} }
but it is verifying only the first result.
then I wrote this :
"condition" : { "array_compare": { "ctx.payload.hits.hits": { "path": "timetaken" , "gt": { "value": 2000 } } } }
but this is not creating any alert.
what is wrong ?
someone have some clue?

Regards
Carmelo

2

Hey,

somehow the _source seems to be lost in the array_compare. If that doesnt work, maybe just paste the full watch here, so it is easier to reproduce.

--Alex

3

This is my first watcher, probably I can write it better ( if you have tips you are welcome)
below my watcher:
PUT http://localhost:9200/_watcher/watch/time_taken { "trigger" : { "schedule" : { "interval" : "10s" } }, "input": { "search": { "request": { "indices": ["logstash*"], "body": { "query": { "filtered": { "query": { "query_string": { "analyze_wildcard": true, "query": "*" } }, "filter": { "bool": { "must": [ { "query": { "query_string": { "analyze_wildcard": true, "query": "*" } } }, { "range": { "@timestamp": { "gte": 1452384000000, "lte": 1452470399999, "format": "epoch_millis" } } } ], "must_not": [] } } } } } } } }, "condition" : { "compare" : { "ctx.payload.hits.hits.0._source.timetaken" : { "gt" : 0 }} }, "actions" : { "log_error" : { "logging" : { "text" : " -Found {{ctx.payload.hits.total}} -- errors -- {{ctx.payload.hits.hits.0._source.timetaken}}" } } } }

And I would to take some specific fields in the response, non only the first like : {{ctx.payload.hits.hits.0._source.timetaken}}

4

Hey,

can you try this condition?

  "condition": {
    "array_compare" : {
      "ctx.payload.hits.hits": {
        "path": "_source.timetaken",
        "gt" : {
          "value" : 0,
          "quantifier" : "some"
        }
      }
    }
  },

worked for me locally.

--Alex

5

Thank you, for you time, I try it soon,
In while time I improved my query :

"query": {
    "filtered": {
         "query": {"query_string": { "query": "timetaken:>2000","analyze_wildcard": true }  },
         "filter":
              { "bool": {
                     "must": [{"range": { "@timestamp": {"gte": "now-10s"}}} ],
                     "must_not": []
                   }
               }
     }
  }

Then I am receiving this :

{
0= {_type=test, 
_source= {timetaken=46, [...] }
},
1={_type=test, 
_source= {timetaken=1222, [...] }
}, 
2={_type=test, 
_source= {timetaken=5000, [...] }
}, 
[...]
}

How can I read the single fields on the array?
Exist some kind of for/while?

Thank you for your time.
Carmelo

6

Hi @Alex,
thank you for your time

I tried your solution, but is not working. In my test I have change it like :

"gt" : { "value" : 2000, "quantifier" : "some" }

In my tests I have 20 logs :

  • 12 --> timetaken <= 2000
  • 8 --> timetaken > 2000

with your condition I am receiving this result : Total = 20 expected Total=8

Something is wrong, what ?

7

Hey,

I think you misinterpreted the functionality of the script condition here. First, conditions only exist, to decide if the action should be executed at all. Conditions dont change your search results or anything like that. The above condition simply matches if any of those array elements is greather than 2000.

This still means, that the hits you are getting back can contain anything based on your search.

Wondering why you dont change your query to only contain results with the timetaken threshold and change your condition to have the total hits being returned greater than 0.

--Alex

8

Thank you @Alex,
I understood what you mean in fact I changed my query, below, and it is working fine

`"query": {
         "query_string": {
            "query": "timetaken:>2000",
           "analyze_wildcard": true
         }`

and the condition:

 `"condition" : {
      "compare" : {
        "ctx.payload.hits.total" : {  "gte" : 1 }
      }
  }`

and I am taking the results I want.

Now I have a new problems:

  1. How can I read the array results?
  2. How can I get the fields in the array? Should I write a script?

Do you prefer I open a new thread ?

Regards
Carmelo

9

Hey,

The easiest way is to add a scripted transform, which reduced the hits you got to the ones you want. Or just even extract some fields out of that one. So writing a transformation script and then being able to easily process those results in your logging action should be the way to go.

--Alex

10

Thank you for your time,
I will do it.

11

I am trying to use transform
"condition" : { "compare" : { "ctx.payload.hits.total" : { "gte" : 1 } } }
but I am receiving this error:
"type": "script_exception", "reason": "failed to compile script [ScriptException[scripts of type [inline], operation [elasticsearch-watcher_watch] and lang [groovy] are disabled]] with lang [return ctx.payload.hits] of type [groovy]" }, "status": 500
my elasticseach.yml is:
script.engine.groovy.file.aggs: on script.engine.groovy.file.mapping: on script.engine.groovy.file.search: on script.engine.groovy.file.update: on script.engine.groovy.file.plugin: on script.engine.groovy.indexed.aggs: on script.engine.groovy.indexed.mapping: off script.engine.groovy.indexed.search: on script.engine.groovy.indexed.update: off script.engine.groovy.indexed.plugin: off script.engine.groovy.inline.aggs: on script.engine.groovy.inline.mapping: off script.engine.groovy.inline.search: off script.engine.groovy.inline.update: on script.engine.groovy.inline.plugin: off

What is wrong?

12

Hey,

is it possible, that your watch has somewhere else some scripting part? Can you post the complete watch? Maybe using markdown formatting features like ```, so it is easier to follow

--Alex

13

Good morning @Alex,
below an example of my watcher

{
  "trigger" : { "schedule" : { "interval" : "10s" }  },
  "input": {
     "search": {
       "request": {
           "indices": ["logstash*"],
	     "body": {
               "query": {
                 "filtered": {
                       "query": {
                             "query_string": { "query": "timetaken:>7", "analyze_wildcard": true }
                        }
                   }
                }
	      }
        }
	}
   },
   "condition" : {
         "compare" : { "ctx.payload.hits.total" : {  "gte" : 1 } }
     },
     "actions" : {
	 "log" : { 
            "transform" : { "script" : "return ctx.payload.hits" },
	    "logging"   : { "text" :"CTX : {{ctx}}"  }
	   }
       }
}

the errors is :
> "error": {

          "root_cause": [
             {
                "type": "script_exception",
                "reason": "failed to compile script [ScriptException[scripts of type [inline], operation [elasticsearch-watcher_watch] and lang [groovy] are disabled]] with lang [return ctx.payload.hits] of type [groovy]"
             }
          ],
          "type": "script_exception",
          "reason": "failed to compile script [ScriptException[scripts of type [inline], operation [elasticsearch-watcher_watch] and lang [groovy] are disabled]] with lang [return ctx.payload.hits] f type [groovy]"
       },
       "status": 500
    }`

What is wrong ?

14

Fixed error:
the problem was on my elasticsearch.yml

I added this :
script.inline: on
script.indexed: on

and now it is working