Logstash - Elastic search input plugin - Aggregation query doesn't seem to work

Elastic Stack Logstash
1

I'm trying to use logstash to source data from elastic search and index them in elastic search log index. Basically, I am trying to use an aggregate query from ES input and put them in another ES log index. When I use curl command results show correctly, however, when I use that in logstash query, it runs in a indefinite loop. Here's my logstash input configuration

input {
elasticsearch {
hosts => "localhost:9200"
index => "summary-dev"
user => xxxxx
password => xxxxx
query => '{
"aggs" : {
"genres1" : {
"terms" : { "field" : "customerId" },
"aggs": {
"terms1": {
"filter": {"bool": {"must": {"match" : {"activeFlg":"N"}}}}
}
}
}
}
}'
schedule => "*/2 * * * *"
}
}

1 Like
2

I have the same issue. It seems to be related the behavior of the scroll api with respect to aggregations.
It would appear from the docs that this plugin uses a scroll query, which does not play nice with aggregations. In 2.3 and earlier you could choose whether you wanted the query to be executed as an ordinary query, or as a scan and scroll via the "scan" parameter.

But support for that was dropped in 2.4.

It is worth noting that this question has been asked many times before.
2017:

2016:

2015:

There's even an open issue for it.

For now your best bet is probably to use one of the workarounds suggested with bash/php in previous questions. Or you could try and write up a PR to address the issue.

Not sure if anyone else has news on support for elastic aggregations within the elastic input plugin, it's a sorely needed feature to be sure.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.