Aggregating using 'Dynamic Fields'

Elastic Stack Elasticsearch
1

Hello

I have a use case that feels like a good fit for ElasticSearch except for
one problem. I'm hoping someone might be able to suggest an approach for
overcoming it using ElasticSearch.

I have a lot of time-series data from sensors. Extremely simplified, a
reading looks a bit like this

{ "sensor_id": 12345678, "timestamp": 10203454354, "value": 5643 }

I want to do things like calculate the average value for each sensor within
date buckets for recent history.

Thus far ElasticSearch seems like an excellent fit (using an approach
similar to that described here:
http://www.elasticsearch.com/guide/en/elasticsearch/guide/current/time-based.html)

The problem is that I need the end user to be able to dynamically group
sensors into 'categories' via a UI and then do aggregations and filtering
based on that.
( eg 1: calculate the average value for each category of sensor within date
buckets for recent history)
( eg 2: as above but filtered to only calculate for category A & B)

If the user moves a particular sensor from one category to another, then
the system should reflect that when calculating aggregations across
previous readings.

Some approaches I could take

  1. re-index every time a user changes the category structure. This doesn't
    really seem feasible.

  2. Resolve categories to sensor_ids in the application and use them to
    filter and bucket in ElasticSearch. Take the result from ElasticSearch and
    re-aggregate in the application.
    This seems problematic because
    A) There may be 1000s of sensor_ids in a category. The request
    payload could get quite large.
    B) It seems a shame to have to implement bucketing and
    aggregation in the app when I have ElasticSearch

  3. Filter and Aggregate using a function that can map a sensor_id to a
    category for each reading.
    This would address problem B from approach 2, but
    a) the function would still be large if there are 1000s of
    sensor ids, and
    b) I am unsure of the performance implications of using
    functions this way.

Has anyone done something like this with ElasticSearch? How?

Cheers
Perryn

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1a2acbc4-e72e-488a-8ef3-36846d290b4c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.