Logstash hardware requirements

[perezdev]

Hello,

I'm going to install logstash in a Linux VM. I'm trying to find out the best hardware requirements for this VM so that logstash could handle the amount of data expected as smooth as possible.

The amount of data to handle is the following:

• Events per second: 3000 EPS

• Size: 100 GB/day (data won't be stored in logstash)

• Data format: Most of the data should be able to be parsed using csv filters or grok patterns and sending the output directly to an Elasticsearch cluster in other VM.

According to this requirements, what could be a good configuration as starting point for my logstash server?

Thank you in advance for the help.
Regards.

[mikewillis]

In the set up I work with we have servers running Logstash with 8 CPU cores and 10GB of RAM and they're each processing more than 3000 events per second most of the time and more than 100GB per day, sending it to Elasticsearch. Some of the processing is grok filter some csv, some nasty custom ruby. I couldn't explain how we arrived that amount of CPU and RAM but evidently is is sufficient in our case.

Assuming you are able to easily change CPU/RAM on the virtual machine it might be worth doing some experimenting. Also with grok filter I've found it can be possible to improve the performance by spending some time thinking about how it works. E.g. if you need to match against multiple patterns and one of those patterns is going to be matched much more frequently than the others, put that pattern first.
I once managed to make some Logstash config process data 50x faster than my first attempt. I used config like this along with some large log files.

input {
    stdin {
        codec => plain
    }
}

your grok filters or whatever go here


filter {
    metrics {
        meter => "documents"
        add_tag => "metric"
        flush_interval => 60
    }
}

output {
    if "metric" in [tags] {
       stdout {
           codec => line {
                format => "rate: %{[documents][rate_1m]}"
            }
        }
    }
}

invoke with
$ /path/to/logstash -f /path/to/filter.conf < /path/to/logfile