Does Elasticsearch process data sequentially?
I want to use elasticsearch with filebeat as no need to filter any kind of data.
But will it process sequentially or non-sequentially.
I just want to process data sequentially. So please suggest the approach.
If ES cannot process sequentially, what is the solution approach for sequential processing (to calculate consumption = diff between two consecutive records)?
Thanks
Elasticsearch will process data as it receives it. If Filebeat doesn't send in order then Elasticsearch cannot do anything about that.
How are you doing the calculation between the two records?
Okay. Will it work same when logstash is in picture? Actually I have to use logstash for performing calculations. Like addition of values from a particular column of all records of a day and subtracting the sum from previous day's sum. Input will be from json file.
It should help, yes. Take a look at https://www.elastic.co/guide/en/logstash/6.2/plugins-filters-elapsed.html
Doing this type of processing in Logstash often means that all events need to go through the same thread, which is inefficient and limits performance. Another way to do this may be to create a process that runs periodically and processes data that has been indexed recently. This approach will naturally lag a bit and requires custom code but should scale better.
Actually where does it have limitations? While subtracting a value from last day's value or while adding values of all records of a day?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.