One index or multiple index for exact same mapping, but the data are clustered based on a field

Elastic Stack Elasticsearch
1

Hi there, community!

I have a short question.
I have around 100k documents, each ranging from a few hundred KB to a maximum of 1MB. The documents have exactly the same mapping but can be grouped into three different clusters of equal size, based on a field in each document (let's say doc_type). Each query will target only one cluster at a time.

Will I achieve better performance if I split the documents into three different indexes?
My initial thought is that performance will improve, since each search would involve a smaller number of documents. However, I'm new to Elasticsearch and would like to know if there’s a flaw in this reasoning. I assume I won’t run into an oversharding problem (only three indexes/shards), right? :slightly_smiling_face:

My setup is a single machine with 4 cores and 12 GB of RAM.

Thank you in advance!

2

Hello,

As per my understanding, you should be able to see some performance improvement as the document will be available in 3 different indices & query will be against required doc_type.
But the question is if there is any performance issue for current setup with the document count & search against this single index?

Thanks!!

3

Thanks for your quick response!

Just to clarify: each document is stored in only one index, and I don’t filter by doc_type — instead, I query directly against the index that holds documents of that specific type. Is that what you meant?

To be honest, I already set things up with 3 separate indices from the start, so I don’t know if there were any performance issues with a single-index approach — that’s on me :grinning_face_with_smiling_eyes:. But since the document count might eventually grow to around 300k, I wanted to take a more proactive approach rather than having to migrate and reindex everything later.