Logstash - Elastic search input plugin - Aggregation query doesn't seem to work

Kyle_Ryan · March 25, 2019, 8:26pm

I have the same issue. It seems to be related the behavior of the scroll api with respect to aggregations.
It would appear from the docs that this plugin uses a scroll query, which does not play nice with aggregations. In 2.3 and earlier you could choose whether you wanted the query to be executed as an ordinary query, or as a scan and scroll via the "scan" parameter.

But support for that was dropped in 2.4.

It is worth noting that this question has been asked many times before.
2017:

2016:

2015:

There's even an open issue for it.

github.com/logstash-plugins/logstash-input-elasticsearch

Support Aggregations

opened 01:41PM - 22 Feb 17 UTC

PhaedrusTheGreek

status:needs-triage int-shortlist

Not sure if this would be better supported by a totally different plugin, but I …think it would make sense to support aggregation results. As an example, say we wanted to use a sum aggregation, by terms. ``` GET myserverlogs*/_search { "size": 0, "aggs": { "types": { "terms": { "field": "hostname" }, "aggs": { "connections": { "sum": { "field": "connections" } } } } } } ``` Results: ``` "aggregations": { "types": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "hostA", "doc_count": 90310, "total_connections": { "value": 2344 } }, { "key": "hostB", "doc_count": 485, "total_connections": { "value": 233 } }, { "key": "hostC", "doc_count": 485, "total_connections": { "value": 123 } } ] } } ``` Then in Logstash we could say something like: ``` input { elasticsearch { ... results_mode => "aggregation" source_array => "types.buckets" } } ``` This could even be used to create aggregation based summary indices on a regular interval. I remember @polyfractal was working on an ES plugin that did something similar last year.

For now your best bet is probably to use one of the workarounds suggested with bash/php in previous questions. Or you could try and write up a PR to address the issue.

Not sure if anyone else has news on support for elastic aggregations within the elastic input plugin, it's a sorely needed feature to be sure.