# Setting total shard per node **URL:** \ **Category:** Elasticsearch\ **Created:** [November 10, 2018, 1:36am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120 "2018-11-10T01:36:28Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Rizal\_Trenggono](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@Rizal\_Trenggono](https://discuss.elastic.co/u/Rizal_Trenggono)\ **Post date:** [November 10, 2018, 1:36am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/1 "2018-11-10T01:36:28Z")
Hi, i have 30 data node,3 master node and 1 coordinator node, i want to create 1 index with 30k shard (each node gets 1k shard). i try put mapping shard on coordinator node, response from node is error "Failed to parse value [30000] for setting [index.number\_of\_shards] must be \<= 1024", so please give me advices for this problem ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 10, 2018, 1:49am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/2 "2018-11-10T01:49:22Z")
> [@Rizal\_Trenggono](#): > > i want to create 1 index with 30k shard (each node gets 1k shard). Why would you want it that big? ---
**Author:** ![Rizal\_Trenggono](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@Rizal\_Trenggono](https://discuss.elastic.co/u/Rizal_Trenggono)\ **Post date:** [November 10, 2018, 1:55am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/3 "2018-11-10T01:55:21Z")
i will put the data 2 PB, please give me advice for node topology how many data node, master node and coordinator node? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 10, 2018, 1:55am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/4 "2018-11-10T01:55:47Z")
You would be much better off having multiple indices. ---
**Author:** ![Rizal\_Trenggono](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@Rizal\_Trenggono](https://discuss.elastic.co/u/Rizal_Trenggono)\ **Post date:** [November 10, 2018, 2:06am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/5 "2018-11-10T02:06:04Z")
how to combine/append indices? to get all the data when I query ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 10, 2018, 2:06am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/6 "2018-11-10T02:06:52Z")
What sort of use case is this? What is the data source, what sort of queries are you doing? ---
**Author:** ![Rizal\_Trenggono](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@Rizal\_Trenggono](https://discuss.elastic.co/u/Rizal_Trenggono)\ **Post date:** [November 10, 2018, 2:17am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/7 "2018-11-10T02:17:07Z")
the data source from log, sum of the data 1k per second, my query sql like this (select \* from index union all select \* from index2), sorry i am newbie in elasticsearch ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 10, 2018, 2:18am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/8 "2018-11-10T02:18:10Z")
You should use time based indices then, with one index per hour/day/week. not one big index. ---
**Author:** ![Rizal\_Trenggono](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@Rizal\_Trenggono](https://discuss.elastic.co/u/Rizal_Trenggono)\ **Post date:** [November 10, 2018, 2:31am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/9 "2018-11-10T02:31:32Z")
how to retrieve all data from indices? ---
**Author:** ![Rizal\_Trenggono](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@Rizal\_Trenggono](https://discuss.elastic.co/u/Rizal_Trenggono)\ **Post date:** [November 10, 2018, 2:48am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/10 "2018-11-10T02:48:29Z")
Thank Mark, i can get retrieve all data use GET /index\*/\_search ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 10, 2018, 7:16am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/11 "2018-11-10T07:16:59Z")
Yes you can. May I suggest you look at the following resources about sizing: [https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing) > **[How many shards should I have in my Elasticsearch cluster? | Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet... > **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)** > > Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type « Black Friday » (Vendredi noir) ou période de Noël peuvent pousser... And [https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right](https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 10, 2018, 9:10am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/12 "2018-11-10T09:10:13Z")
Once you have gone through the resources David linked to, you may also find [this webinar on storage optimisation](https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch) useful. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2018, 9:10am UTC](https://discuss.elastic.co/t/setting-total-shard-per-node/156120/13 "2018-12-08T09:10:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.