We recently did a proof-of-concept project for a customer which ingested log events from various sources into a Kafka – Logstash – Elasticsearch – Kibana stack. For various reasons resources were tight. One problem which we ran into several times was running out of disk space on the Elasticsearch nodes (this was despite setting up Curator to delete older indexes, and increasing the available storage as much as possible). Like most software, Elasticsearch does not always handle this situation gracefully, and we often had to ssh in and manually delete index files to get the system working again.
We've built a clever open source proxy to help - it works with Solr and Elasticsearch http://www.flax.co.uk/blog/2016/04/21/running-disk-space-elasticsearch-solr/