Best way to reliably forward logs from Elasticsearch to multiple destinations using Logstash

Elastic Stack Logstash
1

Hi all,

I am collecting logs from multiple sources (Syslog, SNMP, Windows EventLog,) and storing them in Elasticsearch.

Now I need to forward these stored logs from Elasticsearch to multiple external destinations using Logstash.

Logstash Configuration Overview Diagram
Logstash Configuration Overview Diagram1536×1024 81.1 KB

For example:

  • Kafka forward
  • EventLog forward
  • Syslog forward
  • Secondary Syslog forward

To implement this, I pull events from Elasticsearch then forward them to each output destination.

My challenge is about reliable multi-destination forwarding.

Problem

If I use a field like isForwarded or forward_status to mark a log as "forwarded", then what happens in this situation:

  • The log successfully forwards to 2 destinations
  • Forwarding fails for the other 2 destinations
  • But Logstash still marks the event as forwarded because the pipeline completed

This risks data loss, because failed destinations will never receive those logs again.

Questions

  1. What is the recommended pattern to reliably forward logs from Elasticsearch to multiple independent destinations?
  2. Is there a proper way to track per-destination forward status?
  3. Should I use separate pipelines per destination with pipeline { } input/output?
  4. What is the best practice to ensure that failure of one destination does not affect processing of others, but also does not lose logs?
  5. Does Logstash provide any built-in at-least-once guarantee when forwarding from Elasticsearch input?

Kindly guide me for the following process...

Thanks
Mahesh Kumar S

2

I don't think there is a recommended pattern as this is unrelated to Elasticsearch, it depends on each user case and what tools they are going to use and what are the requirements.

If you use Logstash to query Elasticsearch and send the data to multiple destinations, if you use just one pipeline, no matter what output configuration you use, even using pipeline-to-pipeline communication, a failure in one destination may impact the others sooner or later.

To have true isolation for each destination you would need to have one separated pipeline to each destination, which could impact your Elasticsearch as you would query the data multiple times, unless each destination is receiving just one type of data.

A better approach would be using Kafka as a buffer, you would use a logstash pipeline to query the data from Elasticsearch and put it into Kafka, then you would use Logstash to query this from Kafka and send to the destinations.

2 Likes