Logstash As Side Car Container

[Java_Padawan]

I've been using logstash 2.x as a side car for a microservice container. And I facing an issue where it consumes a lot of resources. 1.4GB Ram and almost 2000 millicores for one of the most active microservice.

Not giving up on the approach, I tried to upgrade to 5.3, which is even worse. It slows down to a crawl on default settings.

Should I keep using the approach or should I sue filebeat side car container and a cluster of logstash containers?

And any suggested design that is already battle tested?

Updated:

Filebeat is the last option, as it requires us to create another infrastructure for logstash that supports different types of filters.

Also, I think the logstash 5.3 crashing might be caused by this exception. Running it in openshift

[2017-04-20T01:28:58,248][DEBUG][logstash.instrument.periodicpoller.cgroup] Erro
r, cannot retrieve cgroups information {:exception=>"Errno::ENOENT", :message=>"
No such file or directory - /sys/fs/cgroup/cpuacct/system.slice/docker-d53ade74e
71e495bfafd276db5e2cb30f3e80674436937452a77868711f5f833.scope/cpuacct.usage"}

[magnusbaeck (Magnus Bäck)]

If Logstash consumes too much resources and you're only using it to read files, use Filebeat instead.

[Java_Padawan]

Filebeats is an option, but we do use some transformation and a bit of processing (clone, drops and mutates).

Any thoughts on how to calculate the optimum logstash configuration based on input rate?

I don't want to do trial and errors as each POD I am managing have different behaviors.

[magnusbaeck (Magnus Bäck)]

Filebeats is an option, but we do use some transformation and a bit of processing (clone, drops and mutates).

Right, but that could be done in a single Logstash instance, possibly on another machine.

Any thoughts on how to calculate the optimum logstash configuration based on input rate?

There are probably a number of JVM options to tweak, but apart from that here's isn't that much you can do.