ELK architecture choice

Elastic Stack Elasticsearch
1

Hello,

I read a lot of docs about ELK architecture and I saw different designs. My team has to deploy a new ELK stack and we are facing with some choices.

  1. beats ==> kafka ==> logstash ==> elasticsearch
  2. beats ==> kafka ==> elasticsearch (where beats have the role of logstash)
  3. beats ==> logstash ==> elasticsearch
  4. beats ==> elasticsearch

All kind of logs should be parsed and sent to ELK stack : firewall, proxy (nginx) , LB (haproxy, kemp), syslog, windows logs, journal logs from linux machines, ....

The workload is around 250Go / day. We want to use Kafka as a buffer.

It seems that logstash is becoming replaced by beats ? Any word about this ?

Any recommandations ?

Regards

2

If you can do option 4, I'd go for it as there are much less things to maintain :slight_smile:

Let me answer to:

It seems that logstash is becoming replaced by beats ? Any word about this ?

and

(where beats have the role of logstash)

About beats/logstash, that's not the same thing. You can compare logstash inputs vs beats and compare logstash filters vs elasticsearch node ingest feature though.

In short: if you can do all the parsing/processing of your data in elasticsearch with ingest node, just use beats and Elasticsearch.
If you have a more advanced pipeline or if you want to send the data collected by beats to elasticsearch AND to another output such as a storage or kafka or whatever, then use Logstash.

We want to use Kafka as a buffer.

So if this is mandatory, then go for option 2.

3

I would recommend option 1 if Kafka is required as Elasticsearch is not able to pull from Kafka.

4

Hello,

Thx for your reply.

Option 4 should be the easiest way but I am not sure it will cover all our "uses cases"

Just another question, what's the most used , logstash or beats ?

If we use beats with modules, logs will go to kafka then logstash, but logstash won't do anything else butsending them to ES.. So not added value.

Regards

5

I don't have numbers at hand but I'd say that:

For collecting data, beats is widely used now.

If we use beats with modules, logs will go to kafka then logstash, but logstash won't do anything else but sending them to ES.. So not added value.

Using beats with modules is unrelated to using kafka or logstash.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.