Kafka (input) > Logstash (1.5) > Elasticsearch 1.7.3 (output)
i had one index per month, 3 shards with 1 replica in 3 data nodes*.
It's the third time logstash doesn't unstack any message. In bigdesk we had only one data server that reached 100% in CPU (upload image) (it happens once a month).
Even after restarting Elasticsearch the problem continues. I reduced the refresh interval (-1), changed indices.store.throttle.type to none and removed the replica trying to increase the speed indexation without reaching to index new documents.
The only way to index/update new documents with logstash is to remove the index and create a new one.