Is there a reason not to have logstash on each host

Elastic Stack Logstash
1

Hello,

i am curious if there is a reason that I should use file beat to ship the logs to a logstash agent rather than having a logstash agent on each host and eliminating the middle man?

Thanks,
Kenneth

2

You mean aside from the fact that Logstash requires Java, and can therefore ask for a 1G to 2G heap size to operate, while filebeat is super-lightweight by comparison?

Logstash should be your heavy lifter for doing regular expression and other parsing at a centralized point, rather than on each edge node. I mean, if that's what you want to do, you can. But it's kind of like swatting a fly with a sledgehammer. More power than the job calls for.

3

All of our servers already have Java installed since we are a java based company. I am using filebeat at the moment but it seems by the time it gets to ES , i am losing about 4-6 min of real time analysis. I would want to try and cut that time in half by chance. Any suggestions?

4

It may not have anything to do with whether you use Filebeat or Logstash on client side. What is the refresh interval of your indexes? 4-6 minutes are big unless you made changes to some settings.

5

Where can i check that? As far as I know , no changes have been made to that.

6

I am assuming it is the default (1s) . No one on the team has changed that setting.

7

Hi Kenneth
We have skipped filebeat and it works well.

We have 50-70 nodes and each of them has a logstash agent that pushes
log4j-logs to several ES indices. Each node has around 5-10 different logs
that goes to their own index. It involves some parsing of course.

They consume some cpu, roughly 1
core, and around 1GB of RAM. The main issue we have had is that the agents
pushed with too small batches to ES which hammered ES with connection
refused errors as result. However, when we'd identified the problem we
could easily increase the batch size (
https://gist.github.com/RickardCardell/341b00dc97c841e249b603c580c03fcf)
and now it has been really stable for a few months.

All in all we haven't really seen a need for using filebeat yet since it
would be more complex using a middle-man and having to learn one more tool.

//Rickard

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.