Aggregate Index Data Daily to another Index

Elastic Stack Logstash
1

We have tomcat access logs in one index called "access" (with fields like timeTaken and bytesSent) and we'd like to have this data aggregated into another index "stats" with, for example, the average timeTaken for certain pages or the sum of all 500 errors. We plan to have only daily granularity in the "stats" index.

We would like some tips and pointers on how to achieve this.

  • Is it possible within ELK w/o coding ruby or making an external script to query and push the data?
  • How does one pull the aggregated SUM and AVG data from ES?
  • If we wanted the data only daily, could we schedule such runs within ELK?

Thanks in advance!

2

Have a look at the rollup feature in ES 6.3, announced the other day.

3

Since we neither have XPack nor 6.3 I've looked into the Avg aggregation.

So we can request the average timeTaken for docs from yesterday:

{
    "from" : 0, "size" : 0,
    "aggs" : {
        "avgTimeTaken" : { "avg" : { "field" : "timeTaken" } }
    },
    "query": {
        "range" : {
            "timestamp" : {
                "gte" : "now-1d/d",
                "lt" : "now/d"
            }
        }
    }
}

We use size: 0 so that no actual docs are returned. Don't know if there's a better way.

The question then becomes: can we somehow use Logstash to a) make this query to ES and b) convert the response into a document it passes back to ES as a new document?

4

You should be able to use the elasticsearch input, possibly followed by one or more filters to get the search result in the shape you want.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.