Cluster from virtual machines

Elastic Stack Elasticsearch
1

I'm currently working on a system that gathers IDS events and stores them in an ELK stack.
For the IDS and event storage we have one big server. (128 GB ram, 8 core Xeon, 12 Disk (10K RPM) raid 5 array)

To give you an idea on the size, it stores about 6 billion documents on average.

At the moment our ES cluster consists of only 1 single node. However, the performance of ES is underwhelming.
Would it be beneficial to set up virtualisation and have the 1 physical server run multiple nodes?

2

Yep!

3

What do you mean by underwhelming performance? Is it related to indexing or query performance or perhaps both? What does disk usage and CPU usage look like? Do you see a lot of GC and heap pressure?

Running multiple nodes on the server might help, but probably depends on what is limiting performance.

4

I should've probably mentioned this indeed.
The main problem we have right now is with indexing.
Logstash is spitting out a lot of ES 429 error, which if I understand it correctly, means that ES can't keep up with indexing.

5

Which version of Elasticsearch are you on? Have you got any non-default configuration settings? What indexing throughput are you seeing? What is the size of bulk requests and documents? How many parallel indexing threads/connections do you have? What does CPU utilisation and disk I/O statistics look like during indexing?