# Aggregate Index Data Daily to another Index

**URL:** <https://discuss.elastic.co/t/aggregate-index-data-daily-to-another-index/136035>\
**Category:** Logstash\
**Created:** [June 15, 2018, 6:14am UTC](https://discuss.elastic.co/t/aggregate-index-data-daily-to-another-index/136035 "2018-06-15T06:14:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [June 15, 2018, 6:14am UTC](https://discuss.elastic.co/t/aggregate-index-data-daily-to-another-index/136035/1 "2018-06-15T06:14:13Z")

</div>

We have tomcat access logs in one index called "access" (with fields like timeTaken and bytesSent) and we'd like to have this data aggregated into another index "stats" with, for example, the average timeTaken for certain pages or the sum of all 500 errors. We plan to have only daily granularity in the "stats" index.

We would like some tips and pointers on how to achieve this.

- Is it possible within ELK w/o coding ruby or making an external script to query and push the data?
- How does one pull the aggregated SUM and AVG data from ES?
- If we wanted the data only daily, could we schedule such runs within ELK?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 15, 2018, 6:20am UTC](https://discuss.elastic.co/t/aggregate-index-data-daily-to-another-index/136035/2 "2018-06-15T06:20:32Z")

</div>

Have a look at the rollup feature in ES 6.3, announced the other day.

> **[Elasticsearch 6.3.0 Released
	  	 | Elastic](https://www.elastic.co/blog/elasticsearch-6-3-0-released)**
>
> Today we are pleased to announce the release of Elasticsearch 6.3.0, based on Lucene 7.3.0. This is the latest stable release, and is already available for deployment via our Elasticsearch Servic...

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [June 19, 2018, 4:19pm UTC](https://discuss.elastic.co/t/aggregate-index-data-daily-to-another-index/136035/3 "2018-06-19T16:19:58Z")

</div>

Since we neither have XPack nor 6.3 I've looked into the Avg aggregation.

So we can request the average timeTaken for docs from yesterday:

```
{
    "from" : 0, "size" : 0,
    "aggs" : {
        "avgTimeTaken" : { "avg" : { "field" : "timeTaken" } }
    },
    "query": {
        "range" : {
            "timestamp" : {
                "gte" : "now-1d/d",
                "lt" : "now/d"
            }
        }
    }
}

```

We use `size: 0` so that no actual docs are returned. Don't know if there's a better way.

The question then becomes: can we somehow use Logstash to a) make this query to ES and b) convert the response into a document it passes back to ES as a new document?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 19, 2018, 7:44pm UTC](https://discuss.elastic.co/t/aggregate-index-data-daily-to-another-index/136035/4 "2018-06-19T19:44:14Z")

</div>

You should be able to use the elasticsearch input, possibly followed by one or more filters to get the search result in the shape you want.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 17, 2018, 7:44pm UTC](https://discuss.elastic.co/t/aggregate-index-data-daily-to-another-index/136035/5 "2018-07-17T19:44:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.