If/else within Logstash output plugin

Elastic Stack Logstash
1

Hello,
I'm having a Logstash configuration similar to the below:

input {
  beats {
    port => 5044
  }
}
filter {
  clone {
    clones => ["local-dc"]
    add_tag => ["cloned"]
  }
}
output {
  if "cloned" in [tags] {
    elasticsearch { hosts => ["elastic-01:9200"] }
  }
  else {
    elasticsearch { hosts => ["elastic-02:9200"] }
  }
}

Assuming elastic-01 is temporarily unreachable, I would expect only the messages tagged with "cloned" to fail.
Nevertheless, all messages fail.

Is that normal?

2

Yes. Logstash require all outputs to succeed before the batch is considered complete do that is expected behaviour.

3

@Christian_Dahlqvist: Thank you for your feedback on this!
Any workarounds (besides using distinct Logstash instances per Elasticsearch destination) ?

4

What are you trying to accomplish? If you want to clone events and send them to the same server, just send them to a different index in the elasticsearch output.

5

What I'm trying to accomplish is to send logs to two distinct Elasticsearch clusters.
If that requires two distinct Logstash instances, then so be it.

6

You can have two different output pipelines (distributor pattern) within a single Logstash instance, both backed by separate persistent queues.

7

@Christian_Dahlqvist: That is an interesting approach (albeit a beta feature).

What is not clear from the documents though is whether all filtering now needs to take place within the contents of config/pipelines.yml.
Do we need to move filtering logic away from the .conf file?

8

The pipelines feature is just a different way of organizing your config.

9

That seems like an odd design decision, what's the rationale behind it?

10

It is designed to prevent data loss. If not all outputs were required to succeed any of them could fail and drop data at any time.

11

Wouldn't that be where DLQ comes in to pick up failed entries?

12

That would depend on why it failed, I believe. DLQ does not queue everything that fails.

1 Like
13

DLQ is only supported by the Elasticsearch output plugin as far as I know and only queues documents where Elasticsearch reported an error, not when Elasticsearch was not available.

1 Like
14

Thanks everyone for the responses.

Given that I had a spare Logstash server doing nothing, I ended up duplicating everything:

  • Two Filebeat instances residing on the host that produces the logs.
  • Each Filebeat instance ships logs to a dedicated Logstash node.
  • Each Logstash node pushes documents to a dedicated Elasticsearch cluster.

Not the most elegant approach, but it was easy to configure without spending much time converting an existing (and huge) Logstash configuration into pipelines.
The regression test would have lasted weeks for no reason.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.