# Index with few shard or index with many shards?
**URL:** https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985
**Category:** Elasticsearch
**Created:** [December 24, 2018, 8:50am UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985 "2018-12-24T08:50:41Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@kamal](https://discuss.elastic.co/u/kamal)
#### Post date: [December 24, 2018, 8:50am UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/1 "2018-12-24T08:50:41Z")
Hi
I have 3 nodes and 55TB data to index, I can split it to 10 index, each one contain 155 shard or 110 index each contain 10 shards.
I don't know which one is the best?
Can any one help?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 24, 2018, 9:04am UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/2 "2018-12-24T09:04:23Z")
What is the use case? What kind of data do you have? How are you going to query/use it?
---
### Author:  [@kamal](https://discuss.elastic.co/u/kamal)
#### Post date: [December 24, 2018, 11:01am UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/3 "2018-12-24T11:01:14Z")
The data is log, most of the fields are structured, just need to search on one field, but I want to use kibana to draw different visualizations.
Each record is max 500 bytes.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 24, 2018, 11:21am UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/4 "2018-12-24T11:21:00Z")
In that case the recommended best practice is to use [time-based indices](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/time-based.html). Make sure you follow [these guidelines on shard sizes and sharding practices](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster). The following resources may also be useful:
[https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch](https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch)
[https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right](https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right)
[https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)
> **[Sizing Hot-Warm Architectures for Logging and Metrics in the Elasticsearch...](https://www.elastic.co/blog/sizing-hot-warm-architectures-for-logging-and-metrics-in-the-elasticsearch-service-on-elastic-cloud)**
>
> Want to learn more about the differences between the Amazon Elasticsearch Service and our official Elasticsearch Service? Visit our AWS Elasticsearch comparison page.These are exciting times! Elastics...
Having said that, having only 3 nodes for 55TB of raw data sounds a bit small, especially if you intend to have replicas in order to get high availability. I would however recommend running some tests to see how much data you can hold on your particular hardware.
---
### Author:  [@kamal](https://discuss.elastic.co/u/kamal)
#### Post date: [December 24, 2018, 1:02pm UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/5 "2018-12-24T13:02:55Z")
Each node has 40 cores, 128GB ram and 12 hdd (6TB) which are raid 10 in three arrays.
I read the documents, but still can't decide which one is better.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 24, 2018, 1:18pm UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/6 "2018-12-24T13:18:44Z")
What is not clear? Which options are you considering? Unsure how to apply time-based indices?
---
### Author:  [@kamal](https://discuss.elastic.co/u/kamal)
#### Post date: [December 24, 2018, 1:50pm UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/7 "2018-12-24T13:50:54Z")
Sorry to ask again, I am not professional in that (although I read all elastic docs).
I don't know which way to go:
1. Many indexes, each index few shards.
2. Few indexes, each index many shards.
In both architectures, each shard size is at max 30GB.
Thanks
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 24, 2018, 1:54pm UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/8 "2018-12-24T13:54:16Z")
It depends on your data. How many different types of data? For each type, how much data do you have per day? What is the total time period covered by this data set?
---
### Author:  [@kamal](https://discuss.elastic.co/u/kamal)
#### Post date: [December 24, 2018, 2:05pm UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/9 "2018-12-24T14:05:19Z")
All data are same type, belongs to 24hours, after indexing, new data will not append.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 24, 2018, 4:30pm UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/10 "2018-12-24T16:30:13Z")
Then time-based indices may not be applicable. Try to align the indices with how you query the data. The feeer shards you need to query the better performance I would expect. If you are always going to query the full data set the total shard count may be more important that exactly how these divide into indices.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 21, 2019, 4:30pm UTC](https://discuss.elastic.co/t/index-with-few-shard-or-index-with-many-shards/161985/11 "2019-01-21T16:30:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.