Kubernetes - Logstash or Fluentd?

Elastic Stack Logstash
1

Hi Team!
I have started working with ELK since last year and has been using a lot Beats, Elasticsearch, Kibana and a bit about Logstash.
I have started POC case using ELK on-premise supporting OCP 4.x and customer asked about use Fluentd instead Logstash.
Then I have research about it and found some articles saying it's better for orchestration cases.... However, since I don't know much about both one, I'd like hear some extra information from experts or someone already worked with both to get more feedback about it.
If could get a matrix comparing features between both would be great.

Below articles I have read about it:
https://platform9.com/blog/kubernetes-logging-comparing-fluentd-vs-logstash/
https://medium.com/tensult/the-log-battle-logstash-and-fluentd-c65f2f7c24b4

Thanks, Mauricio

2

I would recommend looking at Filebeat as well.

3

Thanks Christian!

4

LogStash is part of the popular ELK stack. Fluentd is built by Treasure Data and is part of the CNCF. Fluentd also has excellent support for Elastic. For CNCF hosted project (e.g. Kubernetes , OpenTracing or Prometheus), Fluentd could be a better choice.

5

When an elastic index moves into a read only state, it can prevent additional data being written to the cluster. Typically a read only state occurs when an index in on a node that reaches 85% disk space used.

would you please help me out with the below challenge ?
The cluster does have an API to get details for each index, so it may be possible to scan each index and check if it's in read only state.

  • how to know if one or more indexes are marked as read only, would you please let me know?
  • how can we generate an alert somewhere - likely grafana - when this happens
    Note: we use cerebro ES admin tool as well
6

LogStash is part of the popular ELK stack. Fluentd is built by Treasure Data and is part of the CNCF. Fluentd also has excellent support for Elastic. For CNCF hosted project (e.g. Kubernetes , OpenTracing or Prometheus), Fluentd could be a better choice.

7

Hi @Mauricio_Borges,

better for orchestration cases

That probably depends on what orchestration tools you use.

In my organization we use Puppet and Elastic provides official Puppet modules for e.g. Logstash. That being said. Logstash only needs a config file and JAVA (plus jvm.options conig) which could be easily orchestrated with any tool. I would expect Fluentd to be similar...

Elastic have ever expanding support for Kubernetes. We use Filebeat, running as a DaemonSet inside of Kubernetes to send logs via Logstash to Elasticsearch. You could cut out Logstash in the middle if you do not need extra log parsing. If you logs are well tokenized JSON then you could probably go straight to Elasticsearch.

For shipping logs from Kubernetes fluent-bit is another option.

We decided to go with Filebeat -> Logstash because the protocol they use to communicate lets Filebeat know if Logstash is under pressure and Filebeat can back down etc.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.