# Timeseries data aggregation

**URL:** <https://discuss.elastic.co/t/timeseries-data-aggregation/86527>\
**Category:** Elasticsearch\
**Created:** [May 20, 2017, 2:38pm UTC](https://discuss.elastic.co/t/timeseries-data-aggregation/86527 "2017-05-20T14:38:41Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Soumitra](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@Soumitra](https://discuss.elastic.co/u/Soumitra)\
**Post date:** [May 20, 2017, 2:38pm UTC](https://discuss.elastic.co/t/timeseries-data-aggregation/86527/1 "2017-05-20T14:38:41Z")

</div>

Hello,

I have timeseries metrics about memory usage by processes. Here are the fields:

- timestamp when the sample was taken  
 - userid  
 - process id  
 - memory of process id at time

Here are few samples in CSV format:

1, foo, 1, 100  
1, foo, 2, 500  
2, foo, 3, 100  
2, bar, 4, 100

In general there are many processes and one user may have multiple processes running at the same time. At a particular time, memory usage by all processes are indexed into Elastic. What is the best way to find out peak memory usage by any user in last 15 days?

Here is the query with aggregation, which adds the memoryMB by user at every time instant, then I post process the output from ES to get the max memoryMB.

```
{
    "from":0,
    "size":0,
    "query":{"bool":{"must":[{"range":{"date":{"gte":"now-15d"}}}]}},
    "aggs": {
        "name": {
            "terms": {
                "field": "user",
                "size": 1000
            },
            "aggs": {
                "date": {
                    "terms": {
                        "field": "date",
                        "order" : { "_term" : "asc" },
                        "size": 15000
                    },
                    "aggs": {
                        "mb": {
                            "sum": { "field": "memoryMB" }
                        }
                    }
                }
            }
        }
    }
}

```

It looks like I am abusing terms aggregation, I am also exploring groovy plugin. What is the best way to do it?

Thanks  
-Soumitra.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 20, 2017, 10:17pm UTC](https://discuss.elastic.co/t/timeseries-data-aggregation/86527/2 "2017-05-20T22:17:47Z")

</div>

A good tip when you are getting started is to try building something that looks, visually, close to what you are after in Kibana and then copy the agg that it generates and adjust to your needs 🙂

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 21, 2017, 6:16am UTC](https://discuss.elastic.co/t/timeseries-data-aggregation/86527/3 "2017-05-21T06:16:20Z")

</div>

Is your memory usage data captured with a certain frequency, e.g. every minute, so that you can use a [date histogram](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/search-aggregations-bucket-datehistogram-aggregation.html) aggregation with the interval set to the periodicity of the data?

---

<div class="post-metadata">

**Author:** ![Soumitra](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@Soumitra](https://discuss.elastic.co/u/Soumitra)\
**Post date:** [May 21, 2017, 3:02pm UTC](https://discuss.elastic.co/t/timeseries-data-aggregation/86527/4 "2017-05-21T15:02:49Z")

</div>

I have periodicity of 90 seconds, but it is not guaranteed.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 21, 2017, 3:14pm UTC](https://discuss.elastic.co/t/timeseries-data-aggregation/86527/5 "2017-05-21T15:14:58Z")

</div>

Can you have stats coming in at different times that need to be aggregated together, e.g. from different hosts?

---

<div class="post-metadata">

**Author:** ![Soumitra](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@Soumitra](https://discuss.elastic.co/u/Soumitra)\
**Post date:** [May 23, 2017, 1:14pm UTC](https://discuss.elastic.co/t/timeseries-data-aggregation/86527/6 "2017-05-23T13:14:40Z")

</div>

There may be stats coming at different time from different hosts.

But for this use case, I want to find out the peak memory usage at any given time. So, the memory usage for every usage, for every timestamp has to added, then need to find out the max.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 20, 2017, 1:15pm UTC](https://discuss.elastic.co/t/timeseries-data-aggregation/86527/7 "2017-06-20T13:15:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.