Reindex in progress, not able to access data by then

Elastic Stack Elasticsearch
1

Hi Team

I am reindexing from a remote cluster a dataset of around 180GB. The complete source index is rather bigger (around 1.5tb) but i am using time query while reindexing data of 3 months only.

The task is still in progress and going on slowly. Till now it's reached about 20gb in the new index. The new destination index status shows green if i check the health of the index.

While creating the new index, i have defined a new mapping set to disallow dynamic mapping for certain fields. Also, there are the settings which i have provided

"settings" : {
        "number_of_shards" : 10,
        "number_of_replicas" : 0,
        "refresh_interval": -1
    },

Can i not use the new index data until it's completed? Because when i am checking in kibana, index patterns it says:

"Couldn't find any Elasticsearch data
You'll need to index some data into Elasticsearch before you can create an index pattern.."

Although, the elasticsearch index management shows the index with health green and status open.

Both the cluster stacks are on same version: 7.4.2

Can you please help with this? Also, is the method used to reindex data is correct or can be more optimized?

Thanks in advance
Dhruv

2

Update: I created a new blank index with similar settings and put some docs in it. It was not visible in kibana until i added a replica to the index. So, kibana would require a replica (which is by default for a new index) to create index patterns.

So, if i create a replica of my new index which is in process of reindexing, will that further slower down the process? Should i create it when the reindexing completes?

Please suggest.

Thanks
Dhruv

3
4

Thanks for referring to this. I have already set my index refresh
-interval to -1. My reindex is not yet complete. It has still reached to 92gb only as I post this.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.