We have a use case where we need to create around 4000 different indices in a cluster. Every index has a different mapping. We were benchmarking elasticsearch 6.6.1 and observed the following.
When first created a single index and indexed around 32k documents and ran getByID for stored documents. The 99.9% latency was under 100ms.
We then went ahead to just create another 1500 indices (with one replica and without any data) and ran the benchmark again for the earlier index. The 99.9% latency shot upto 700-800ms.
When we go ahead and delete these empty 1500 indices, the latency goes back to under 100ms.
Note: Our cluster contains 12 data nodes having 1TB disk and 16gb memory allocated to ES.
Does number of indices in a cluster affect the getByID call in anyway?