# Cluster from virtual machines

**URL:** <https://discuss.elastic.co/t/cluster-from-virtual-machines/52035>\
**Category:** Elasticsearch\
**Created:** [June 7, 2016, 7:57am UTC](https://discuss.elastic.co/t/cluster-from-virtual-machines/52035 "2016-06-07T07:57:48Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![maigel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maigel/32/81104_2.png) [@maigel](https://discuss.elastic.co/u/maigel)\
**Post date:** [June 7, 2016, 7:57am UTC](https://discuss.elastic.co/t/cluster-from-virtual-machines/52035/1 "2016-06-07T07:57:48Z")

</div>

I'm currently working on a system that gathers IDS events and stores them in an ELK stack.  
For the IDS and event storage we have one big server. (128 GB ram, 8 core Xeon, 12 Disk (10K RPM) raid 5 array)

To give you an idea on the size, it stores about 6 billion documents on average.

At the moment our ES cluster consists of only 1 single node. However, the performance of ES is underwhelming.  
Would it be beneficial to set up virtualisation and have the 1 physical server run multiple nodes?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 7, 2016, 8:09am UTC](https://discuss.elastic.co/t/cluster-from-virtual-machines/52035/2 "2016-06-07T08:09:12Z")

</div>

Yep!

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 7, 2016, 9:29am UTC](https://discuss.elastic.co/t/cluster-from-virtual-machines/52035/3 "2016-06-07T09:29:00Z")

</div>

What do you mean by underwhelming performance? Is it related to indexing or query performance or perhaps both? What does disk usage and CPU usage look like? Do you see a lot of GC and heap pressure?

Running multiple nodes on the server might help, but probably depends on what is limiting performance.

---

<div class="post-metadata">

**Author:** ![maigel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maigel/32/81104_2.png) [@maigel](https://discuss.elastic.co/u/maigel)\
**Post date:** [June 7, 2016, 9:53am UTC](https://discuss.elastic.co/t/cluster-from-virtual-machines/52035/4 "2016-06-07T09:53:32Z")

</div>

I should've probably mentioned this indeed.  
The main problem we have right now is with indexing.  
Logstash is spitting out a lot of ES 429 error, which if I understand it correctly, means that ES can't keep up with indexing.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 7, 2016, 10:29am UTC](https://discuss.elastic.co/t/cluster-from-virtual-machines/52035/5 "2016-06-07T10:29:20Z")

</div>

Which version of Elasticsearch are you on? Have you got any non-default configuration settings? What indexing throughput are you seeing? What is the size of bulk requests and documents? How many parallel indexing threads/connections do you have? What does CPU utilisation and disk I/O statistics look like during indexing?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:45pm UTC](https://discuss.elastic.co/t/cluster-from-virtual-machines/52035/6 "2017-07-05T22:45:39Z")

</div>


