Sorry I was inaccurate. Yes, Logstash handles back pressure from Elasticsearch by suspending indexing requests.
And that is why I recommended Kafka. Logstash persistent queues are GA for just a few weeks, so it hasn't been an option for product teams.
These scenarios have nothing to do with log files. My customer makes the router and programs the embedded Linux VM. As the router does its job, it also sends HTTP Posts with event info such as "connection established".
We could have used the Logstash HTTP Input Plugin, but we wanted buffering. Until otherwise proven, Kafka is still the leading fire-hose drinker.
My customers have built many non-log Event systems using Elasticsearch:
- Enterprise desktops, servers, phones and other devices send Security Events.
- QA Automation uses hundreds of VMs in Build, Test, Destroy. The testing VMs send QA Events.
- Network appliances send Network Events.
- Advertising widgets send Ad Events (impression, click, action, ...)