Logstash index keeps increasing all the time

Elastic Stack Logstash
1

I am experiencing the following ES behavior: I copied a sample log file to a particular directory so Filebeat was able to read it and forward to Kafka. Then Logstash read that topic with logs data and sent it to ES.

I can see new index logstash-2017.05.18 appeared in ES. But the size of the index keeps increasing even when Filebeat doesn't read any other files and Kafka topic doesn't have new messages, as of now the index size is 20 times more the size of the log file.

Moreover, when I tried to check logs in Kibana I was not able to see any of them for any of the period, even though Monitoring section displayed a big number of received and the same number of emitted events (those numbers keep increasing as well).

Log files didn't show any errors.

This is logstash.conf

input {
  kafka {
    bootstrap_servers => "server:9092"
    topics => ["logstash"]
    codec => "json"
  }
}

filter {
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
  }
}

So, the result is logstash-2017.05.18 index size keep increasing constantly even though Kafka topic doesn't get any new messages.

Does anybody have any ideas why it happens?

2

Filebeat and Logstash work good without Kafka though. But after adding Kafka - Logstash receives messages all the time even when Kafka doesn't get new messages, though I cannot see any messages in Kibana.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.