is it possible to use filebeat>logstash>eleasticsearch in this way.
Logstash is running three ports, 5045, 5046, 5047.
I have a filebeats config that is picking up logs from 3 different locations and redirecting each log to its own logstash port for parsing.
each logstash "port" has its own grok expression tied to its own template etc being fed into ES.
Is this possible?
We've being experiementing with this today and it appears filebeat will only send to the last listed type:log entry with port number in the list inside filebeat.yml.
Filebeat.yaml excerpt
type: log
enabled: True
paths:
/var/logs/syslog/logs/*json
fields: {log_type: syslog}
---- Logstash output -----
output.logstash:
enabled: true
hosts: ["localhost:5045"]type: log
enabled: True
paths:
/var/logs/apache/logs/*json
fields: {log_type: apache}
---- Logstash output -----
output.logstash:
enabled: true
hosts: ["localhost:5046"]type: log
enabled: True
paths:
/var/logs/sec/logs/*json
fields: {log_type: security}
---- Logstash output -----
output.logstash:
enabled: true
hosts: ["localhost:5047"]logs1.yml from etc/logstash/conf.d
input {
beats{
port => 5045
}
}
filter {
grok {
match => { "message" => "some grok pattern" }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "syslog-%{+YYYY.MM.dd}"
document_id => "syslog"
manage_template =>true
}
stdout {
codec => rubydebug
}
}logs2.yml from etc/logstash/conf.d
input {
beats{
port => 5046
}
}
filter {
grok {
match => { "message" => "some grok pattern" }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "apache-%{+YYYY.MM.dd}"
document_id => "apache"
manage_template =>true
}
stdout {
codec => rubydebug
}logs3.yml from etc/logstash/conf.d
input {
beats{
port => 5047
}
}
filter {
grok {
match => { "message" => "some grok pattern" }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "security-%{+YYYY.MM.dd}"
document_id => "security"
manage_template =>true
}
stdout {
codec => rubydebug
}