Logstash like forwarder of beats. Why? Benefits?

Elastic Stack Logstash
1

Hello,
I'm newby in ELK world.
I'm using beats (filebeat and others) with output to Logstash. LS only forward to ES ingest node (https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html):

input {
beats {
port => 5044
}
}

output {
if [@metadata][pipeline] {
elasticsearch {
hosts => "ES_hostname:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
pipeline => "%{[@metadata][pipeline]}"
user => "elastic"
password => "pass"
}
} else {
elasticsearch {
hosts => "es_hostname:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
user => "elastic"
password => "pass"
}
}
}

I've loaded first beats pipelines at ES, of course, so all seems to works well.

But I have doubts. With this configuration, are there any benefit using Logstash? or it's better to point beats direct to Elasticsearch.
I understand that this configuration are using ES ingest node (with de pipelines from beats preloaded) to make all the job and Logstash only formward the events but no processing events.

I'm true? What is the best architecture thinking that now I'm not doing any aditional processing to the events but.
I've 300 virtual machines in my infraestructure.

Thanks all!!

2

I see no reason to use logstash if it is not modifying the events. Having beats forward directly to logstash simplifies your infrastructure.

3

Thanks Badger.
I've found one, buffering.

And looking statics recopiled by metricbeats I'm seen 350MB incoming traffic and 3GB outgoing traffic. So it seems like Logstash its doing a lot of job.
I'm missing something...

Can anyboy help me?

Thanks

4

Beats can spoil to file so I am not sure buffering is a differentiator. If you are just sending data and not going any processing Filebeat generally has less overhead and was developed do do just this task well.

5

Hi,
Reading againg documentation I think I've messed with the filbeat pipeline. Whether a pipeline is defined in logstash or if a pipeline is loaded from the ingest nodes, it is logstash who does the workjob and not the ES ingest nodes themselves as initially understood in the second case. So, for filebeat logstash it's interesting since we take workload outside elasticsearch.
For the other beats I'm still thinking logstash its only a follower and I can't see adventages.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.