Alternative outputs to run alongside Elasticsearch

We have 300-400 machines piping a volume of roughly 20 million log messages a day through our ELK stack.

We’ve recently had some stability issues and taken the opportunity to upgrade all the components and explore some changes to the topology of the cluster.

While Elasticsearch & Kibana are serving us well for some use-cases, one area where we’re struggling to make it work well is providing real-time (or near-realtime) access to logs—essentially just simple grep / tail -f functionality without requiring SSH access.

Network conditions & surges in log volumes—especially in dev/test environments, can cause a slow down in indexing time, leading to frustrations from developers trying to access their logs for debugging purposes.

This has lead us to start discussing other potential outputs from Logstash to run alongside Elasticsearch, that could provide alternative windows into the river of logs.

Anyone else explored similar configurations? I’ve love to hear what has & hasn’t worked for you.

What does your architecture look like now, how have you scaled, are you using brokers?