I've setup a local lab for testing and killed my elasticsearch cluster for a few days and restarted it and noticed all that data seems to be gone after restarting it. I wasn't sure if that data just get's dropped if elasticsearch is unreachable. If I want and/or need store and forward capability do I need to have logstash as part of the data pipeline? I understand it won't persist unless I use something like rabbitmq with disk persistence as my transport but I thought that logstash did indeed store and forward. Please correct me if I'm wrong and what I need to do to make it work in this fashion.
packetbeat and topbeat will drop events if elasticsearch is not available. filebeat retries until data could be send (as long as log files are available).
logstash has no full persistent queue support yet (about to come). In the future logstash would have helped buffering these events (speak disk persistence). For now you can use logstash to push all data to some intermediate queue like rabbitmq or kafka.
Great, this is the exact answer I was looking for, I was mostly concerned with filebeat anyway. Glad to hear they are adding persistence as well.