Logstash As Side Car Container


(Java Padawan) #1

I've been using logstash 2.x as a side car for a microservice container. And I facing an issue where it consumes a lot of resources. 1.4GB Ram and almost 2000 millicores for one of the most active microservice.

Not giving up on the approach, I tried to upgrade to 5.3, which is even worse. It slows down to a crawl on default settings.

Should I keep using the approach or should I sue filebeat side car container and a cluster of logstash containers?

And any suggested design that is already battle tested?

Updated:

Filebeat is the last option, as it requires us to create another infrastructure for logstash that supports different types of filters.

Also, I think the logstash 5.3 crashing might be caused by this exception. Running it in openshift

[2017-04-20T01:28:58,248][DEBUG][logstash.instrument.periodicpoller.cgroup] Erro
r, cannot retrieve cgroups information {:exception=>"Errno::ENOENT", :message=>"
No such file or directory - /sys/fs/cgroup/cpuacct/system.slice/docker-d53ade74e
71e495bfafd276db5e2cb30f3e80674436937452a77868711f5f833.scope/cpuacct.usage"}


(Magnus Bäck) #2

If Logstash consumes too much resources and you're only using it to read files, use Filebeat instead.


(Java Padawan) #3

Filebeats is an option, but we do use some transformation and a bit of processing (clone, drops and mutates).

Any thoughts on how to calculate the optimum logstash configuration based on input rate?

I don't want to do trial and errors as each POD I am managing have different behaviors.


(Magnus Bäck) #4

Filebeats is an option, but we do use some transformation and a bit of processing (clone, drops and mutates).

Right, but that could be done in a single Logstash instance, possibly on another machine.

Any thoughts on how to calculate the optimum logstash configuration based on input rate?

There are probably a number of JVM options to tweak, but apart from that here's isn't that much you can do.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.