How many shards should I create in terms of an index across 20 data node?

Hi Robin,

The average size of one elastic shard (primary or replica) should not be more than 80 GB max. It helps you in quick initialise and allocation of shards among nodes when you restart any node or in case of any failure.

The number of shards depend on how much data you have right now and what would be data growth rate in next 2 years.

EG: if your data volume is 1.2 TB and data growth is 20 percent an year. that makes your total planned volume as 1.7 TB. Hence you should go with 20 primary nodes and 1 replica at least.

If you have ample resources and memory you can offer yourself 2 replicas. The right guess of primary shards at start of index creation is vital as you cannot change this number once index is created.

Another suggestion : Always use elastic index alias name in your query, so that you can change underlying index name as per your wish, It gives you flexibility of re-indexing data at ease.

You can have a good read at below link. Hope it helps