# Logstash index keeps increasing all the time

**URL:** <https://discuss.elastic.co/t/logstash-index-keeps-increasing-all-the-time/86343>\
**Category:** Logstash\
**Created:** [May 19, 2017, 1:21am UTC](https://discuss.elastic.co/t/logstash-index-keeps-increasing-all-the-time/86343 "2017-05-19T01:21:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![John\_06](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@John\_06](https://discuss.elastic.co/u/John_06)\
**Post date:** [May 19, 2017, 1:21am UTC](https://discuss.elastic.co/t/logstash-index-keeps-increasing-all-the-time/86343/1 "2017-05-19T01:21:40Z")

</div>

I am experiencing the following ES behavior: I copied a sample log file to a particular directory so Filebeat was able to read it and forward to Kafka. Then Logstash read that topic with logs data and sent it to ES.

I can see new index `logstash-2017.05.18` appeared in ES. But the size of the index keeps increasing even when Filebeat doesn't read any other files and Kafka topic doesn't have new messages, as of now the index size is 20 times more the size of the log file.

Moreover, when I tried to check logs in Kibana I was not able to see any of them for any of the period, even though Monitoring section displayed a big number of received and the same number of emitted events (those numbers keep increasing as well).

Log files didn't show any errors.

This is logstash.conf

```
input {
  kafka {
    bootstrap_servers => "server:9092"
    topics => ["logstash"]
    codec => "json"
  }
}

filter {
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
  }
}

```

So, the result is `logstash-2017.05.18` index size keep increasing constantly even though Kafka topic doesn't get any new messages.

Does anybody have any ideas why it happens?

---

<div class="post-metadata">

**Author:** ![John\_06](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@John\_06](https://discuss.elastic.co/u/John_06)\
**Post date:** [May 20, 2017, 12:20am UTC](https://discuss.elastic.co/t/logstash-index-keeps-increasing-all-the-time/86343/2 "2017-05-20T00:20:48Z")

</div>

Filebeat and Logstash work good without Kafka though. But after adding Kafka - Logstash receives messages all the time even when Kafka doesn't get new messages, though I cannot see any messages in Kibana.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 17, 2017, 12:20am UTC](https://discuss.elastic.co/t/logstash-index-keeps-increasing-all-the-time/86343/3 "2017-06-17T00:20:55Z")

</div>

Closed on June 17, 2017, 12:20am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.