Creating a Watch with Nest difficulty

Elastic Stack Elasticsearch
1

Hi, I'm having a bit of trouble getting this watch setup correctly. I know my problem is in the Input lambda, it doesn't like how I'm using the aggregation.

What I'm trying to do is find the number of times my field "sc-status" equals 500 grouped by days so I get something like:

Monday : 150
Tuesday : 175
Wed : 225
Blah Blah Blah

Here's the code,(Using NEST)

        string query2 = "{ \"match\" : { \"sc-status\" : \"500\" } }";
        var client2 = new ElasticClient();           
        var result = client2.PutWatch("watch_name", w => w
            .ThrottlePeriod("10s")
            .Trigger(t => t
                .Schedule(s => s
                    .Cron("0 * * * * ?")))
            .Input(i => i
                .Search(se => se
                    .Request(r => r
                        .Indices("iislog")
                        .Body<object>(b => b
                            .Aggregations(a=>a
                                .DateHistogram("errors",d=>d
                                    .Interval("day")))
                                    .QueryRaw(query2)))))
            .Condition(c => c
                .GreaterThanOrEqualTo("ctx.payload.hits.total", 1000))
            .Actions(a => a
                .Add("send_email", new EmailAction
                {
                    To = "<my-email>",
                    Subject = "iislog errors",
                    Body = new Nest.EmailBody
                    {
                        Text = "At {{ctx.trigger.triggered_time}}\nThere were {{ctx.payload.hits.total}} errors in the iislogs"
                    },
                    AttachData = true,
                    Priority = Nest.EmailPriority.High
                })));

I've got the query working in an ElasticsearchClient like this: (Using Elasticsearch.Net)

        string query = @"{
""aggs"" : { 
    ""messages"" : { 
        ""filter"" : {
            ""bool"": {
                ""must"": [{
                    ""query"": {
                        ""match"": {
                            ""sc-status"": ""500""
                            }
                        }
                    }
                ]
            }
        },
        ""aggs"" : {
            ""errors"" : {
                ""date_histogram"" : {
                    ""field"" : ""date"",
                    ""interval"" : ""day"",
                    ""min_doc_count"" : 1
                }
            }
        }
    }
}
}";
ElasticsearchClient client = new ElasticsearchClient();
var response = client.Search("iislog", "logentry", query);

I definitely know it's something to do with the input lambda.
I just can't figure out how to get it in the watch correctly.

2

Hey @danielmoon, what exactly is the trouble that you're having? Are you getting an error of any sort, or is the watch simply not working as you expect?

The search input in your put watch request isn't the same as the query you posted using Elasticsearch.Net. Also, nesting the date histogram under the filter aggregation isn't really necessary in this case. Instead, I think you want the main query to filter on sc-status: 500, and then you want to execute the histogram agg over those results.

Try this as your Input instead:

.Input(i => i
  .Search(s => s
    .Request(r => r
      .Body<object>(b => b
        .Query(q => q
          .Filtered(fq => fq
            .Filter(f => f
              .Term("sc-status", 500)
            )
          )
        )
	.Aggregations(aggs => aggs
          .DateHistogram("errors", dh => dh
            .Field("date")
            .Interval(DateInterval.Day)
            .MinimumDocumentCount(1)
          )
	)
      )
    )
  )
)

Let me know if that helps.

3

Hey that worked!

To answer your question, I was getting an error trying my watch as I described above.

Could not find the appropriate value context to perform aggregation [errors]

In my testing both of the queries I wrote gave me the same result (In Sense) however, the second one had the extra "messages" bucket.

I'll post both of them for clarity:

POST /iislog/logentry/_search?search_type=count
{
"aggs" : { 
    "messages" : { 
        "filter" : {
            "bool": {
                "must": [{
                    "query": {
                        "match": {
                            "sc-status": "500"
                            }
                        }
                    }
                ]
            }
        },
        "aggs" : {
            "errors" : {
                "date_histogram" : {
                    "field" : "date",
                    "interval" : "day",
                    "min_doc_count" : 1
                }
            }
        }
    }
}
}

POST /iislog/logentry/_search?search_type=count
{
"aggs" : {
    "days" : {
        "date_histogram" : { 
            "field" : "date",
            "interval" : "day"
        }
    }
},
"query": {
    "match": {
       "sc-status": "500"
    }
}
}