Operations over indexed documents

Elastic Stack Elasticsearch
1

Hi,

Is it possible to compute variables taking the documents from an index as input? I will describe my current situation and my objective.

I have data indexed on an Elasticsearch cluster. My data contains a timestamp and a set of numeric values and identifiers. I'm currently running operations over my data using Python and the Eland library. I perform operations mainly selecting time windows and checking the existence of certain data to compute numeric values. As I run my script and compute my metrics I'm indexing this computed values on a new index on the cluster.

I want to know if it is possible to do something similar using Elasticsearch and/or Kibana. Is it possible to run a script over the data of my index and store the results in a new index. I think about it as a reindexing process, running the computation over the complete index to get the results and store them in a new index.

The ideal case would be to compute my metrics as I index the data in the index. However, I also don't know if it is possible.

Best regards,

2

Hi @tmslara.a

Perhaps look at transforms... sound a lot like what you are trying to do.

3

Thank you. I will read about it

4

I checked transformations. I need to do more complex operations than aggregations. More like selecting data an run scripts.

5

Understood... Did you see that you can run painless scripts as part of the transformation... probably still not enough but just in case you missed it.

Many users use the data tool/language of their choice like python to do large-scale / complex processing.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.