I have a problem I would like to discuss with you; let me explain.
- I have an application which ingest data read from some sensors.
- The readings came in groups of 400 points (i.e. every time the sensors read the data they run 400 points, one every 3 ms, then the reading stop for few seconds and then again an another reading).
- every reading is identified by an ID so group the readings is pretty easy.
- I would like to compute custom metrics form each reading (e.g. distance from two significant point in the reading, integrals of the readings, etc. );
- on the customs readings I would like to compute statistical analysis with Timelion or Kibana.
- get the readings from ES with a Java app, compute the new metrics and re-ingest the new data in ES;
Using external code to compute the metrics works for sure but doesn't seem to me really scalable and modern; is this the only solution?
Take into account that using custom fields is not an option because for computing the new metrics I need to work on all the 400 points not just a single doc (e.g. for computing a distance).
Note that these points are time series read from sensors.