Temporarily disable ID lookup

Elastic Stack Elasticsearch
1

Hi,

I am trying to index an initial workload of ~1 billion documents into ES 5.2. The documents are pulled from an MYSQL database and they already contain a unique ID and I want to use the same IDs in ES for a quick reference.

I was reading about how to tune indexing and I have tried disabling index refresh and replicas and it sped up the process so now I am wondering if there is a way to disable ID lookup as it might speed up things more.

So is there a way to do this?

Regards

2

I don't think so.

But is it really slow? What are you using to do the extraction?

3

Its not slow per say, however as I move forward with the actual process I will have the shutdown the system to prevent new data from being inserted in the db untill the process finish so I would like to minimize the downtime as much as possible.

4

There is no need for downtime or prevent new data from being inserted if you use timestamps.

5

We would like to do the whole tranistion in a single shot without having to worry about new records.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.