# Correct sharding and indexes for Logging System **URL:** \ **Category:** Elasticsearch\ **Created:** [November 22, 2018, 8:44am UTC](https://discuss.elastic.co/t/correct-sharding-and-indexes-for-logging-system/157826 "2018-11-22T08:44:15Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![CamTheMan](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@CamTheMan](https://discuss.elastic.co/u/CamTheMan)\ **Post date:** [November 22, 2018, 8:44am UTC](https://discuss.elastic.co/t/correct-sharding-and-indexes-for-logging-system/157826/1 "2018-11-22T08:44:15Z")
Hello All, I was wondering if anyone can provide any guidance on the following points. I have a 5 server cluster running each with: 24 CPU Cores 64 GB RAM (30 for elastic the rest for file caches) Lots of SSD disk The system is being using as a logging system SIEM etc, it's been running for about 4 months now, but I'm not sure about the sharding. Already we have: Nodes: **5** Indices: **1636** Memory: **65.9 GB / 154.5 GB** Total Shards: **16165** Unassigned Shards: **0** Documents: **1,845,342,366** Data: **1.0 TB** I believe we are running the default sharding of 5 per index, across 7 index patterns, with each index-pattern generating a new index each day. Question: 1). Is this sharding and index creation just plain wrong, given the use case (logging) 2). At what point do I require a dedicated master nodes? (at present they are just 5x data+master nodes) Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 22, 2018, 8:58am UTC](https://discuss.elastic.co/t/correct-sharding-and-indexes-for-logging-system/157826/2 "2018-11-22T08:58:38Z")
> [@CamTheMan](#): > > 1). Is this sharding and index creation just plain wrong, given the use case (logging) Yes, that does not look good. You seen to have far too many indices and shards for a cluster that size. Please read [this blog post on shards and sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) for practical guidance. I would recommend rethinking how you approach sharding and reduce this dramatically. > [@CamTheMan](#): > > 2). At what point do I require a dedicated master nodes? (at present they are just 5x data+master nodes) It often depends on the load of the system. If nodes are too busy and you start seeing long GC, master elections or cluster instability, it is probably time to add dedicated master nodes. ---
**Author:** ![CamTheMan](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@CamTheMan](https://discuss.elastic.co/u/CamTheMan)\ **Post date:** [November 22, 2018, 8:58am UTC](https://discuss.elastic.co/t/correct-sharding-and-indexes-for-logging-system/157826/3 "2018-11-22T08:58:54Z")
Also current document count for one index (as an example) is 17million This will only grow perhaps by a factor of 10? as more systems are onboarded and send data... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 22, 2018, 9:00am UTC](https://discuss.elastic.co/t/correct-sharding-and-indexes-for-logging-system/157826/4 "2018-11-22T09:00:44Z")
If you are looking to handle larger data volumes this webinar may also be useful: [https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch](https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 20, 2018, 9:00am UTC](https://discuss.elastic.co/t/correct-sharding-and-indexes-for-logging-system/157826/5 "2018-12-20T09:00:49Z")
Closed on December 20, 2018, 9:00am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.