# Optimizing our Elastic Stack? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 8, 2019, 8:36pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555 "2019-03-08T20:36:48Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)\ **Post date:** [March 8, 2019, 8:36pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/1 "2019-03-08T20:36:48Z")
We're starting to witness a heavy CPU Load on our stack and were hoping to get some help in understanding how we can lessen these. Our current topology is as follows. _Hosted on AWS EC2_ 5 Master/Data Nodes of M5x4x Large 2 Coodinating Nodes of M4x4x Large Java heap set to less than 50% of max of 64GB Total no of documents / day ! 100,000 Total no of documents overall ! 2 million The cluster stats is extremely large so I've collated our Stats on this page: [Cluster Stats](https://github.com/titan1978/elastic/wiki/ClusterStats) The nodes stats are listed here: [Node Stats](https://github.com/titan1978/elastic/wiki/Nodes) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 9, 2019, 10:20am UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/2 "2019-03-09T10:20:35Z")
The links to your stats are broken. Indexing 100k documents per day sounds fairly light. Are you searching them very heavily? How many indices do you have, and how many shards? ---
**Author:** ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)\ **Post date:** [March 9, 2019, 11:35am UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/3 "2019-03-09T11:35:13Z")
we have about 40 indices (rotating monthly) and around 250 Shards. Thats just the thing - we did some analysis in our logs and audit to see what could be causing the spikes. The only thing that could stand out was wildcard search in our indexe's free form field. For example searching for "Susan" In a column called "conversation". But those are one-off spikes. We're seeing a continuous CPU Load of 70-80% on the Data Nodes which eventually brings down elastic (Red Indices state) Cluster Stats: > **[Cluster Stats | Computer Hardware | Data](https://www.scribd.com/document/401445504/Cluster-Stats)** > > cluster stats Node Stats: > **[NodeStats | Java (Programming Language) | Virtual Machine](https://www.scribd.com/document/401445546/NodeStats)** > > node stats ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 9, 2019, 12:11pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/4 "2019-03-09T12:11:11Z")
> [@titan1978](#): > > For example searching for "Susan" In a column called "conversation" This should not be a resource-intensive search. One thing to try is to look at `GET _nodes/hot_threads` to see what the nodes are busy doing. > [@titan1978](#): > > We're seeing a continuous CPU Load of 70-80% on the Data Nodes ... I can't see any nodes reporting CPU load of 70-80% in the stats you've shared, although the website you've used to share your stats seems to have a very broken in-page search function so it's hard to be sure. Can you name a specific node that you think has such high CPU in those stats? Could you use something better-suited to sharing text documents like [https://gist.github.com](https://gist.github.com) in future? > [@titan1978](#): > > ... which eventually brings down elastic Normally when something "brings down Elastic" there are copious logs describing what went wrong. Have you looked at them? What do they say? ---
**Author:** ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)\ **Post date:** [March 11, 2019, 1:27pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/5 "2019-03-11T13:27:57Z")
> [@DavidTurner](#): > > \_nodes/hot\_threads we dont have the current node stats. but when It hits those CPUs again - I'll post back here with both hot threads and node\_stats. I'm curious - is the Master/Data Node structure we've chosen in the OP - does that sound like an optimal setup? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 11, 2019, 1:44pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/6 "2019-03-11T13:44:08Z")
> [@titan1978](#): > > I'm curious - is the Master/Data Node structure we've chosen in the OP - does that sound like an optimal setup? It looks like you have ~40GB of data and ~5M documents on each node, so 200GB and 20M documents in total. You're indexing 100k documents per day, and it's not clear what your search load is. In the stats you shared most of the nodes are using less than 10% CPU. It's hard to say for sure without knowing a lot more about your needs, but my initial impression is that this cluster is oversized for its current workload. ---
**Author:** ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)\ **Post date:** [March 11, 2019, 1:47pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/7 "2019-03-11T13:47:29Z")
thanks...I'll keep that in perspective for our coming iteration. Another question here is - would it make sense to separate our Master/Data Nodes into 2 Master and 5 Data Nodes (just to keep current BAU even if oversized as you've indicated) from a performance perspective? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 11, 2019, 5:01pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/8 "2019-03-11T17:01:01Z")
I wouldn't recommend going below 3 master-eligible nodes, because you need at least three for fault tolerance. On busy or large clusters we recommend dedicated master nodes instead of mixed master/data nodes, but the threshold is not clearly defined. Really the only way to answer this kind of question is to perform some [benchmarks](https://esrally.readthedocs.io/en/stable/) of your own workload. ---
**Author:** ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)\ **Post date:** [March 11, 2019, 5:38pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/9 "2019-03-11T17:38:03Z")
Capital! So 3 X Master Node and 5 X Data Nodes to keep in line our current BAU? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 11, 2019, 6:22pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/10 "2019-03-11T18:22:34Z")
> [@titan1978](#): > > So 3 X Master Node and 5 X Data Nodes to keep in line our current BAU? I don't really understand what you're trying to achieve with this change. ---
**Author:** ![titan1978](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@titan1978](https://discuss.elastic.co/u/titan1978)\ **Post date:** [March 11, 2019, 6:24pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/11 "2019-03-11T18:24:24Z")
Just to break out our existing 5 MASTER/DATA cluster into a MASTER specific and DATA specific re-architecture without reducing the number of DATA nodes ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 11, 2019, 6:30pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/12 "2019-03-11T18:30:52Z")
Your cluster seem oversized so I would recommend going with 3 master/data nodes. Just because you CAN have dedicated node types does not mean you SHOULD. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 8, 2019, 6:30pm UTC](https://discuss.elastic.co/t/optimizing-our-elastic-stack/171555/13 "2019-04-08T18:30:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.