Logstash - Elastic search input plugin - Aggregation query doesn't seem to work

I'm trying to use logstash to source data from elastic search and index them in elastic search log index. Basically, I am trying to use an aggregate query from ES input and put them in another ES log index. When I use curl command results show correctly, however, when I use that in logstash query, it runs in a indefinite loop. Here's my logstash input configuration

input {
elasticsearch {
hosts => "localhost:9200"
index => "summary-dev"
user => xxxxx
password => xxxxx
query => '{
"aggs" : {
"genres1" : {
"terms" : { "field" : "customerId" },
"aggs": {
"terms1": {
"filter": {"bool": {"must": {"match" : {"activeFlg":"N"}}}}
schedule => "*/2 * * * *"

1 Like

I have the same issue. It seems to be related the behavior of the scroll api with respect to aggregations.
It would appear from the docs that this plugin uses a scroll query, which does not play nice with aggregations. In 2.3 and earlier you could choose whether you wanted the query to be executed as an ordinary query, or as a scan and scroll via the "scan" parameter.

But support for that was dropped in 2.4.

It is worth noting that this question has been asked many times before.



There's even an open issue for it.

For now your best bet is probably to use one of the workarounds suggested with bash/php in previous questions. Or you could try and write up a PR to address the issue.

Not sure if anyone else has news on support for elastic aggregations within the elastic input plugin, it's a sorely needed feature to be sure.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.