Parallel indices in Elastic search

[radhika]

Hi All,

Is it possible to store the fields of a document in two or more separate parallel indices and later retrieve all the fields together while doing a search. The reason for asking this is we have an elastic index with about 50 million documents each having around 40 fields. So if we split the index into two and have the frequently updating fields in one of the parallel indices, we could easily recreate the new partial index than updating the huge document which has 40 fields.

I have read that updating a document in lucene is painfully slow ~ 100k times slower compared to an insert, hence my question.

Thanks a lot in advance.

[Christian_Dahlqvist]

@radhika

I have read that updating a document in lucene is painfully slow ~ 100k times slower compared to an insert, hence my question.

Where did you get this from? Updating requires more work than inserting a new document with an autogenerated ID but I doubt the difference generally is that much.

@radhika

Is it possible to store the fields of a document in two or more separate parallel indices and later retrieve all the fields together while doing a search. The reason for asking this is we have an elastic index with about 50 million documents each having around 40 fields. So if we split the index into two and have the frequently updating fields in one of the parallel indices, we could easily recreate the new partial index than updating the huge document which has 40 fields.

Frequently updating the same document can be very inefficient and slow as it can result in lots of small segments being created. How frequently do you expect to update a single document?

For scenarios like this you can potentially use parent-child relationship within a single index, but this changes how you query the data and is not supported by Kibana (not sure if that matters to you). You can not join different indices the way you described though.