I'm running the latest stable Logstash version and Kafka v0.10.0.1 (however I've observed the same behavior on 0.8.x and 0.9.x versions).
My data flow is:
- Syslog input -> Kafka output (topic: raw)
- Kafka input (topic: raw) -> Kafka output (topic: processed)
- Kafka input (topic: processed) -> Elasticsearch output
My problem is that when I start up Kafka without creating any topics (I'm using the default kafka configuration which means that when you try and write to a non-existing topic it will create it automatically) and send syslog events I can see that only a small portion of the events reach the Elasticsearch output.
If I send a single event, I can see it reaching the raw topic, but not into the processed topic, then after sending the second event, I can see it in the raw topic and the processed topic but its not going into the Elasticsearch output and on the 3rd event, it goes all the way through.
When I pre-create my topics, events are reaching to all topics correctly, no events are missing.
Here is my inputs/outputs:
step1:
input {
tcp {
port => 7070
}
}
output {
stdout { codec => "rubydebug" }
kafka {
topic_id => "raw"
bootstrap_servers => "kafka:9092"
}
}
step2:
input {
kafka {
topic_id => "raw"
zk_connect => "zookeeper:2181"
group_id => "ls_process"
}
}
filters {
...some filters
}
output {
stdout { codec => "rubydebug" }
kafka {
topic_id => "processed"
bootstrap_servers => "kafka:9092"
}
}
step3:
input {
kafka {
topic_id => "processed"
zk_connect => "zookeeper:2181"
group_id => "ls_output"
}
}
output {
stdout { codec => "rubydebug" }
elasticsearch {
hosts => ["elastic"]
}
}
Any ideas on how to solve this?