Maintaining @timestamp order of docs when using the reindex api

Elastic Stack Elasticsearch
1

There's a deprecated feature in the reindex api which makes (made?) it possible to maintain the timestamp order between docs being reindexed using the reindex api. However, the documentation states that this is a deprecated feature and should not be used and/or used with caution.

What I'd like to accomplish is to retrieve all documents in @timestamp order and reindex them in that order but rewrite the @timestamp with an ingest pipeline. This seems(?) to work up to the version we're using currently using (v8.10.4) but it will be broken in the near future I assume... :frowning:

Is there a possibility to emulate this behavior using plain vanilla Elastisearch functionality without relying on any external scroll-api based solutions? We have a number of use cases where we're not able to give users the ability to access ES using any programmatic (python, java whatever) means except the standard ES api:s using the Kibana dev tool.

The way I've approached this so far is:

POST _reindex
{
  "source": {
    "index": "<some index>",
    "query": {
      "query_string": {
        "query": "@timestamp:*"
      }
    },
    "sort": [
        { "@timestamp" : {"order" : "asc"}}
    ]
  },
  "dest": {
    "index": "<some new index>",
    "pipeline": "set-timestamp"
  }
}

Is this a "proper" approach given my use case?

Many thanks in advance.

2

Your approach of using the _reindex API with sorting by @timestamp in ascending order, combined with an ingest pipeline to rewrite the @timestamp , is quite effective for your objective of maintaining document order during reindexing in Elasticsearch. This method aligns well with your current constraints and version (v8.10.4). However, be mindful of potential performance impacts due to the resource-intensive nature of this process. Also, keep an eye on future Elasticsearch updates, as deprecated features may eventually be removed, possibly affecting this method. Testing and validation are crucial to ensure data integrity in the new index. While this approach works for now, consider alternative, more adaptable methods like scripting if your constraints change in the future. :+1::bar_chart:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.