Aggregation Query possible input ES plugin

Elastic Stack Logstash
1

I really want to know that aggregation is possible in input / output plugin ?

input {

Read all documents from Elasticsearch matching the given query

elasticsearch {
hosts => ["192.168.1.5"]
index => "matching_score_log"
type => "keyword_score"
port => 9200
size => 0
docinfo => true
scan => false
query => '{"aggs":{"terms_keyword":{"terms":{"field":"user_keyword","size":0},"aggs":{"key_score":{"terms":{"field":"inquiry_id","size":0},"aggs":{"key_score":{"sum":{"field":"inquiry_score"}}}}}}}}'

}
}

1 Like
2

I guess you want ES plugin reads aggregations part of query results.
But, it seems that logstash elasticsearch input plugin only watchs hits.hits array of elasticsearch search API response.
Note: https://github.com/logstash-plugins/logstash-input-elasticsearch/blob/master/lib/logstash/inputs/elasticsearch.rb#L168

Does anybody know the way to make aggregation result as hitted documents?

2 Likes
3

Thank you so much.
Finally i made a customized Elasticsearch input plugin. That would be good.

Here is sample config for aggregation query.
input {

Read all documents from Elasticsearch matching the given query

elasticsearch {
hosts => ["${ES_NODE_HOST}"]
index => "xxx_matching_score_log"
type => "keyword_score"
size => 0
scan => false
interval => 1
query => '{"aggs":{"terms_keyword":{"terms":{"field":"user_keyword","size":0},"aggs":{"key_score":{"terms":{"field":"inquiry_id","size":30},"aggs":{"key_score":{"sum":{"field":"inquiry_score"}}}}}}}}'
}
}
output {
elasticsearch {
host => "${ES_NODE_HOST}"
protocol => "http"
index => "oas_matching_score_app"
document_id => "%{[@metadata][keyword]}"
}
#stdout { codec => rubydebug }
}

4

Hi,
I am facing the same issue of extracting aggregation data from the input ES plugin. Seems that you have overcome this issue. Can you please share your customized ES input plugin. Thank you.

4 Likes