# Aggregating using 'Dynamic Fields'

**URL:** <https://discuss.elastic.co/t/aggregating-using-dynamic-fields/21228>\
**Category:** Elasticsearch\
**Created:** [December 12, 2014, 1:44am UTC](https://discuss.elastic.co/t/aggregating-using-dynamic-fields/21228 "2014-12-12T01:44:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Perryn\_Fowler](https://avatars.discourse-cdn.com/v4/letter/p/82dd89/32.png) [@Perryn\_Fowler](https://discuss.elastic.co/u/Perryn_Fowler)\
**Post date:** [December 12, 2014, 1:44am UTC](https://discuss.elastic.co/t/aggregating-using-dynamic-fields/21228/1 "2014-12-12T01:44:46Z")

</div>

Hello

I have a use case that feels like a good fit for ElasticSearch except for  
one problem. I'm hoping someone might be able to suggest an approach for  
overcoming it using ElasticSearch.

I have a lot of time-series data from sensors. Extremely simplified, a  
reading looks a bit like this

{ "sensor\_id": 12345678, "timestamp": 10203454354, "value": 5643 }

I want to do things like calculate the average value for each sensor within  
date buckets for recent history.

Thus far ElasticSearch seems like an excellent fit (using an approach  
similar to that described here:  
[http://www.elasticsearch.com/guide/en/elasticsearch/guide/current/time-based.html](http://www.elasticsearch.com/guide/en/elasticsearch/guide/current/time-based.html))

The problem is that I need the end user to be able to dynamically group  
sensors into 'categories' via a UI and then do aggregations and filtering  
based on that.  
( eg 1: calculate the average value for each category of sensor within date  
buckets for recent history)  
( eg 2: as above but filtered to only calculate for category A & B)

If the user moves a particular sensor from one category to another, then  
the system should reflect that when calculating aggregations across  
previous readings.

Some approaches I could take

1. re-index every time a user changes the category structure. This doesn't  
really seem feasible.

2. Resolve categories to sensor\_ids in the application and use them to  
filter and bucket in ElasticSearch. Take the result from ElasticSearch and  
re-aggregate in the application.  
This seems problematic because  
A) There may be 1000s of sensor\_ids in a category. The request  
payload could get quite large.  
B) It seems a shame to have to implement bucketing and  
aggregation in the app when I have ElasticSearch

3. Filter and Aggregate using a function that can map a sensor\_id to a  
category for each reading.  
This would address problem B from approach 2, but  
a) the function would still be large if there are 1000s of  
sensor ids, and  
b) I am unsure of the performance implications of using  
functions this way.

Has anyone done something like this with ElasticSearch? How?

Cheers  
Perryn

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1a2acbc4-e72e-488a-8ef3-36846d290b4c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1a2acbc4-e72e-488a-8ef3-36846d290b4c%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:44am UTC](https://discuss.elastic.co/t/aggregating-using-dynamic-fields/21228/2 "2017-07-06T00:44:00Z")

</div>