Most Efficient Way to Add filebeats to Logstash

Currently I have added beats to my logstash with there own port numbers and created new configs in logstash for each one of them. Is this the most efficient way to add filebeats to logstash when creating my stack.

Ex:

Filebeat1 output{"LSservername:5001"}
Filebeat2 output{"LSservername:5002"}
Filebeat3 output{"LSservername:5003"}
Filebeat4 output{"LSservername:5004"}
Filebeat5 output{"LSservername:5005"}

logstash(LS) configs
config1input

input {
beats {
type => "***"
port => 5001
}
}
config2input

input {
beats {
type => "***"
port => 5002
}
}

config 3 input

input {
beats {
type => "***"
port => 5003
}
}

config 4 input

input {
beats {
type => "***"
port => 5004
}
}
config 5 input

input {
beats {
type => "***"
port => 5005
}
}

and so on ... We have many servers that will be sending all logs through the stack something like 300 so is there a better way to do this ?

Are you using different ports so that you can assign a different type to each one? Why not use the source host as the field to identify the source?

Filebeat logs of a "common format" can and probably should all go the same port/ in logstash, Each of these would be best in a logstash pipeline (of multi pipeline logstash) The only reason for seperate ports is if you need different processing, such maybe general iis logs vs those from Exchange.

What do you need to be different about each of these logstash configs?

I was using different ports because when I tried to send all of the filebeats to 5044 it seemed like all of the logs were not going through .. but when I separated the filebeats by ports it all worked better

so what it sounds like your saying is that this should work fine ?

Ex:

Filebeat1 output{"LSservername:5044"}
Filebeat2 output{"LSservername:5044"}
Filebeat3 output{"LSservername:5044"}
Filebeat4 output{"LSservername:5044"}
Filebeat5 output{"LSservername:5044"}

logstash(LS) configs
config1input

input {
beats {
port => 5044
}
}

I just did a quick check

  "aggs": {
    "1": {
      "cardinality": {
        "field": "agent.hostname"
      }
    }
  },

for my winlogbeat shows 353 unique hosts sending to the same port, but I'm running 4 logstash servers. The beats config lists all of them to load balance.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.