Mutiple filebeat servers to Single logstash server

Elastic Stack Logstash
1

Hi,
New to ELK and I am doing a production setup . I have 3 servers and installed filebeat on each of them.
now i am sending these 3 filebeat inputs to single logstash using below config.

In all 3 servers Filebeat output is configured as :

type: log
enabled: true
paths:
- /wls_domains/Microservices/logs/app/*.log

logstash output to x.x.x.x:5044

x.x.x.x is ip of logstash server

logstash config:

input {
beats {
port => 5044
}
}

output {
elasticsearch {
hosts => ["http://host:port"]
index => "service-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
stdout { codec => rubydebug }
}

The problem now is my logstash can listen to beat on one server at any time,. logs sent by filebeat from all 3 servers are not read simultaneously by logstash.

can you please help here!

Thank you!

2

I would expect logstash to read events from any number of beats that write to port 5044, not just one. I do not run filebeat so I cannot help with debugging the issue.

3

I agree, that config should read from any number of filebeat senders. If this is your logstash config, your problem is elsewhere.

Do you eventually get events from all 3 filebeat servers, just delayed?

4

@Badger and @rugenl

Thank you for the response guys!!

I have figured it now looks like we need one port per one server so. i used pipelines.yml configuration , implemented collector pattern to achieve this so filebeats in 3 servers can connect to logstash on one server using 3 different ports.

here the config.

pipeline.id: beats1
config.string: |
input { beats { port => 5044 } }
output { pipeline { send_to => [commonOut] } }

  • pipeline.id: beats2
    config.string: |
    input { beats { port => 5045 } }
    output { pipeline { send_to => [commonOut] } }
  • pipeline.id: beats3
    config.string: |
    input { beats { port => 5046 } }
    output { pipeline { send_to => [commonOut] } }
  • pipeline.id: partner

    This common pipeline enforces the same logic whether data comes from any number of Beats

    config.string: |
    input { pipeline { address => commonOut } }
    filter { mutate { remove_field => ["agent","input","ecs"] } }
    output { elasticsearch { hosts => ["http://host:port"] index => "%{[fields][logtype]}-%{+YYYY.MM.dd}" } }
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.