I am new to Watcher and will readily confess that I'm having difficulty figuring out how to create an alert when we're experiencing timeouts on our transaction processors.
Here's the goal: If we have 5 or more timeouts (which is exec_time greater than or equal to 3.5) in the last 30 seconds, send an email alert. Not worried about throttling yet, just proving out a concept.
Here's what I've tried:
{
"trigger": {
"schedule": {
"interval": "30s"
}
},
"input": {
"search": {
"request": {
"indices": [
"httplog-*"
],
"body": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-30s"
}
}
},
{
"match": {
"exec_time": "[3.5 TO *]"
}
}
]
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gte": 5
}
},
"actions": {
"send_email": {
"email": {
"to": "alerts@redacted.com",
"subject": "Timeout Notification",
"body": "{{ctx.payload.hits.total}} Timeouts occurred in the last 30 seconds."
}
}
}
}
}
It's not working, though. Here's what the logs say:
Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[httplog-2017.11.12, httplog-2017.11.13, httplog-2017.11.10, httplog-2017.11.21, httplog-2017.11.11, httplog-2017.11.20, httplog-2017.11.09, httplog-2017.11.07, httplog-2017.11.18, httplog-2017.11.08, httplog-2017.11.19, httplog-2017.11.16, httplog-2017.11.17, httplog-2017.11.14, httplog-2017.11.15], indicesOptions=IndicesOptions[id=7, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=false], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=128, source={
"size" : 0,
"query" : {
"bool" : {
"filter" : [
{
"range" : {
"@timestamp" : {
"from" : "now-1m",
"to" : null,
"include_lower" : true,
"include_upper" : true,
"boost" : 1.0
}
}
},
{
"match" : {
"exec_time" : {
"query" : "[3.5 TO *]",
"operator" : "OR",
"prefix_length" : 0,
"max_expansions" : 50,
"fuzzy_transpositions" : true,
"lenient" : false,
"zero_terms_query" : "NONE",
"boost" : 1.0
}
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [EMCTS499][90.21.65.98:8081][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.index.query.QueryShardException: failed to create query: {
"bool" : {
"filter" : [
{
"range" : {
"@timestamp" : {
"from" : "now-1m",
"to" : null,
"include_lower" : true,
"include_upper" : true,
"boost" : 1.0
}
}
},
{
"match" : {
"exec_time" : {
"query" : "[3.5 TO *]",
"operator" : "OR",
"prefix_length" : 0,
"max_expansions" : 50,
"fuzzy_transpositions" : true,
"lenient" : false,
"zero_terms_query" : "NONE",
"boost" : 1.0
}
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
Caused by: java.lang.NumberFormatException: For input string: "[3.5 TO *]"
I'm having a lot of difficulty finding good examples of how to format the query to return the results I seek. Any advice?