How to get ES aggregation data as Logstash input?

Elastic Stack Logstash
1

I have a Logstash configuration with the 'elasticsearch' input.

input {
  elasticsearch {
    hosts => "localhost"
    index => "logstash-*"
    type => 'aggregation_metric'
    size => 0
    query => '
      {
        "size": 0,
        "aggs": {
           ... here is multi-level aggregation without buckets
        }
      }'

I'm trying to send aggregation result as a new document in new index of ES. I don't need any documents from the root level of searching, but maybe some of documents from top hits aggregations

How can I get the response without buckets as Logstash output?

Response example with expected value 
{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 12,
    "successful": 12,
    "failed": 0
  },
  "hits": {
    "total": 198973,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "first": {
      "doc_count": 34924,
      "another_one": {
        "value": 18      <--- I need values like this one as a new documents 
                              (that I want to create with mutate filter)
      }
    }
  }
}
2

My trick:

  1. I wrote the php-utility that made multiple queries to ES with different console arguments tuning.
  2. In logstash I used the 'exec' input plugin

If someone interesting:

Utility response example (new line separated) 
{"id":1478231999,"created_at":"2016-11-03T23:59:59.000Z","one-chart-complex-value":17,"another-chart-value":3,"something-else-calculated":4}
{"id":1478145599,"created_at":"2016-11-02T23:59:59.000Z","one-chart-complex-value":14,"another-chart-value":2,"something-else-calculated":5}
Logstash exec plugin config

input {
exec {
command => '/etc/logstash/php/daily_chart 5'
interval => 14400
type => 'daily_chart'
codec => multiline {
pattern => "^\n"
what => "previous"
}
}
}
filter {
if [type] == "daily_chart" {
json {
source => "message"
}
mutate
{
convert => [ "created_at", "string" ]
remove_field => [ "command", "message", "host" ]
}
date {
match => ["created_at", "ISO8601"]
target => ["@timestamp"]
}
}
}
output {
if [type] == "daily_chart" {
elasticsearch {
# common ES output
}
}
}

1 Like
3

I am also building a php utility so that logstash can invoke and get the aggregated result, would you please tell me where the php utility writes the aggregated response?

4

@sharon.c, the php tool writing results in stdout and 'exec' input plugin is reading this output.

1 Like
5

Got it, thank you very much!