Hi everyone,
I have a setup that looks like this:
(The data nodes also have the master role)
With Rsyslog used as a lightweight forwarder between the rest of my network and my ELK setup, storing the logs in Kafka so my Logstash cluster can pull them at its pace.
The Rsyslog/Kafka part works well, with logs being pushed to the desired topic, but Logstash doesn't seem to be pulling anything.
Here is my logstash config:
input {
kafka {
topic_id => "logstash-logs-apps"
group_id => "logstash-apps-cluster"
zk_connect => "XX.XX.XX.XX:2181"
type => "logs-apps"
codec => "line"
}
##
# a (winlog)beat input working fine here
##
}
output {
if [type] == "logs-apps" {
elasticsearch {
hosts => ["XX.XX.XX.XX","XX.XX.XX.XX"]
index => "logs-apps-%{+YYYY.MM.dd}"
}
}
##
# an elasticsearch output for the beat input working fine here
##
}
All the machines are Ubuntu 14.04 VMs, with the latest Elasticsearch and Logstash versions installed (2.2.x), and Kafka 0.9.0.0
The connections between my logstash servers and my kafka server are established (as shown with "netstat -uta") but it appears as if Logstash wasn't doing anything regarding this input and no logs are generated by Logstash, ES, or Kafka.
Any thoughts on this issue would be highly appreciated!
Thanks,
Nicolas