Automate re-index process

[gerardgorrion]

We are using:

elastic operator 1.8
Elasticsearch 7.15.2

Current problem: we have 2 shards for every index and 7 data/masters nodes on Elasticsearch (impossible to scale it by license number), we need to retain 6 days the index data and this limit increase the number of index over 3000 (over 6000 shards), that produce that every data node store up to 800 shards.

Our index stores as: solution_service_date, and is not possible to change it. To reduce index number, we think that reindex solution could be a good solution (we have a lot of solution index with -1mb), but the reindex api seem like is no possible to automate it.

We try some solutions, like elasticsearsh reindex tool.

There're some solution to automate reindex to index up to +3 days by index "some_name"?

Thanks

[warkolm (Mark Walkom)]

There's nothing native to the stack that can currently do that for you, no.

It should be pretty simple to script though.

[gerardgorrion]

Thanks! You know some way to run this script into k8s deploy?

[gerardgorrion]

Hi again, we try to make an script to do this and when try to reindex a big index into a new index, the curl to reindex stop at 60 seg of execution:

    reindex=$(curl -X POST "https://$ES_HOST:9200/_reindex" -u user:pass -H 'Content-Type: application/json' -d'
    {
      "source": {
        "index": "'$index'"
      },
      "dest": {
        "index": "'$destination'"
      }
    }
    ')

The output:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   169    0     0  100   169      0      2  0:01:24  0:01:00  0:00:24     0
curl: (52) Empty reply from server

When the reindex operation can complete before 60 seg, the process continue correcly.

Try to use "?wait_for_completion=true" in the reindex petition but the problem persist. Some idea? if we not verify if the reindex was finished is no possible to continue and delete older index...

Thanks!

[gerardgorrion]

More tests, now firing similar reindex from kibana dev tools console:

POST /_reindex
    {
      "source": {
        "index": "metricbeat-origen"
      },
      "dest": {
        "index": "metricbeat-dest"
      }
    }

Timeout:

Failed to connect to Console's backend.
Please check the Kibana server is up and running

And console timeout at 60s:

[gerardgorrion]

Hi, someone know how to upgrade the Elasticsearch timeout to wait the response from reindex? thanks