# Some events are lost when using Kafka automatic topic creation

**URL:** <https://discuss.elastic.co/t/some-events-are-lost-when-using-kafka-automatic-topic-creation/60270>\
**Category:** Logstash\
**Created:** [September 12, 2016, 9:09am UTC](https://discuss.elastic.co/t/some-events-are-lost-when-using-kafka-automatic-topic-creation/60270 "2016-09-12T09:09:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Or\_Weinberger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/or_weinberger/32/11850_2.png) [@Or\_Weinberger](https://discuss.elastic.co/u/Or_Weinberger)\
**Post date:** [September 12, 2016, 9:09am UTC](https://discuss.elastic.co/t/some-events-are-lost-when-using-kafka-automatic-topic-creation/60270/1 "2016-09-12T09:09:40Z")

</div>

I'm running the latest stable Logstash version and Kafka v0.10.0.1 (however I've observed the same behavior on 0.8.x and 0.9.x versions).

My data flow is:

1. Syslog input -\> Kafka output (topic: raw)
2. Kafka input (topic: raw) -\> Kafka output (topic: processed)
3. Kafka input (topic: processed) -\> Elasticsearch output

My problem is that when I start up Kafka without creating any topics (I'm using the default kafka configuration which means that when you try and write to a non-existing topic it will create it automatically) and send syslog events I can see that only a small portion of the events reach the Elasticsearch output.

If I send a single event, I can see it reaching the raw topic, but not into the processed topic, then after sending the second event, I can see it in the raw topic and the processed topic but its not going into the Elasticsearch output and on the 3rd event, it goes all the way through.

When I pre-create my topics, events are reaching to all topics correctly, no events are missing.

Here is my inputs/outputs:

step1:

```auto
input {
  tcp {
    port => 7070
  }
}

output {
  stdout { codec => "rubydebug" }
  kafka {
    topic_id => "raw"
    bootstrap_servers => "kafka:9092"
  }
}

```

step2:

```auto
input {
  kafka {
    topic_id => "raw"
    zk_connect => "zookeeper:2181"
    group_id => "ls_process"
  }
}
filters {
  ...some filters
}
output {
  stdout { codec => "rubydebug" }
  kafka {
    topic_id => "processed"
    bootstrap_servers => "kafka:9092"
  }
}

```

step3:

```auto
input {
  kafka {
    topic_id => "processed"
    zk_connect => "zookeeper:2181"
    group_id => "ls_output"
  }
}

output {
  stdout { codec => "rubydebug" }
  elasticsearch {
    hosts => ["elastic"]
  }
}

```

Any ideas on how to solve this?

---

<div class="post-metadata">

**Author:** ![Or\_Weinberger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/or_weinberger/32/11850_2.png) [@Or\_Weinberger](https://discuss.elastic.co/u/Or_Weinberger)\
**Post date:** [September 12, 2016, 1:11pm UTC](https://discuss.elastic.co/t/some-events-are-lost-when-using-kafka-automatic-topic-creation/60270/2 "2016-09-12T13:11:14Z")

</div>

I was able to resolve this by upgrading my logstash-inputs-kafka and logstash-outputs-kafka plugins, strangely I had to specify the version I wanted to install (5.0.4) since running the normal update didn't choose that version but a much older version (2.0.9).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:38am UTC](https://discuss.elastic.co/t/some-events-are-lost-when-using-kafka-automatic-topic-creation/60270/3 "2017-07-06T04:38:57Z")

</div>