Dedicated Logstash Cluster

Elastic Stack Logstash
1

Hi there,

currently I am using Logstash and the GELF plugin to send logfiles to Graylog.

So: Logstash -> GELF -> Graylog

Basically it works, but sometimes Logstash goes crazy and consumes almost all CPU resources - which is particularly bad on the production system. One reason for sure are very complex grok filters.

To take some load off the production machines and make them immune to Logstash hickups I’m thinking about makind a dedicated Logstash cluster.

So something like this:
Filebeat -> Logstash Cluster -> GELF -> Graylog

Question is: Shoud I put a message broker, e.g. Kafka or RabbitMQ, inbetween?

For example:
Filebeat -> Kafka -> Logstash Cluster -> GELF -> Graylog

Is there kinda reference architecture out there? It should be a fairly common problem and I don’t wanna reinvent the wheel.

2

Shoud I put a message broker, e.g. Kafka or RabbitMQ, inbetween?

That's a good idea if you're planning on running multiple Logstash instances. That way each Logstash instance can all pull messages at its own pace.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.