Maintaining "time delta" between events when reindexing

Elastic Stack Elasticsearch
1

I'm trying to work out a process for maintaining the time difference between event records when re-indexing data from one index to another but with a new "t0" using only Elasticsearch with ingest pipelines and/or logstash (if needed). An external program+cursor approach is not possible in this use case unfortunately.

I'll try to explain this further:

I have an index with log events which have been pre-collected and indexed into a master index. We want to re-use the content of this master index by populating new indices using the re-index api. However, and this is the tricky part, we want to maintain the time-difference between event records although they're indexed at another time - a new "t0".

Say we have a new re-indexing "t0": 2024-02-22 10:45:09

and given the following original timestamps
2023-11-19 12:34:56
2023-11-19 12:36:22
...

The difference of the above is 00:01:26 which, using a new "t0" of 2024-02-22 10:45:09, would result in:
2024-02-22 10:45:09
2024-02-22 10:46:35
...

I really hope this makes sense :thinking:

Is this possible at all given the tools at hand?

2

I don't think this is possible without using an external tool.

In both Logstash and Elasticsearch the events are independent from each other, what you want is to have the timestamp of one event dependent of the timestamp of the previous event.

3

Thanks for your reply. I was/am afraid that this'd be the case :blush: I'll re-iterate and see how to approach it.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.