This is not a hard limit - just a recommendation. Having several thousand shards per node will at some point cause problems and when it does it might be difficult to make changes to the cluster due to this. In version 7.x there is an enforced limit of 1000 shards per node, although I personally think it should be set lower.
Thanks @Christian_Dahlqvist @dadoonet
Both of your suggestions really helps me a lot to understand the concept in detail.
I am going to update the shards in Production from 5 Primary and 1 replica to 2 Primary and 1 replica. Do you think there will be any impact with the performance like querying , or while fetching data for Kibana dashboards as we will be having lot of dashboards in Kibana ?
Also if you can suggest how to rename the hourly index to daily index with the changes in the shards from 5 primary to 2 primary and 1 replica ?
Hi @Christian_Dahlqvist @dadoonet
Can you please suggest me anything on the above queries ?
How you rename indices to instead get daily ones depend on how you are indexing data. Change the pattern to not include hour.
Yes @Christian_Dahlqvist I will change the index pattern to daily ones however my question is with the performance as I am resetting the shards from 5 primary shards to 2 primary shards with 1 replica ?
I do not know what will limit performance at what point within your cluster so this is something you need to test.
thanks @Christian_Dahlqvist
My only concern is that changing the shards setting should not impact the performance. It should not happen that more shards means better performance?
Querying one small shard is typically faster that querying one large shard. If querying the same data volume havinglots of very small shards can often be slower than a smaller number of larger shards. As this depends on data as well as queries you need to test. It is often recommended to have shards that are between 10GB and 50GB in size so that is what I would aim for if I had time-based indices.
thanks a lot @Christian_Dahlqvist I will test the approach on Test Instance and see if it works fine.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.