Output after a health check for destination?

Elastic Stack Logstash
1

Hello Everyone,

When the output destination container is slower to start or unavailable, the message dequeued from RabbitMQ input is lost by Logstash.

Is it possible to have the Logstash wait for a health check to be OK, before attempting output?

Cheers,
Eddie.

2

What output are you using? Are you using persistent queues? Please provide more detail.

3

Output is OpenSearch.

Re persistence:
Are you asking about the RabbitMQ queues? They are durable.
Or are you asking if the messages in the queue are published as persistent?
They are not.

5

I am asking if you are using persistent queues in logstash. It is unclear under what circumstances you think logstash is losing messages. The at-least-once delivery model, along with persistent queues, should prevent message loss under most circumstances.

6

No we are not using persistent queues in Logstash.

My apologies. I had the wrong impression of what was happening with our set up.

When OpenSearch was down, I saw the Logstash "in" event incrementing while "filtered" and "out" did not.

EG:

/_node/stats/events

{
  ..  ..  
  },
  "events": {
    "in": 1,
    "filtered": 0,
    "out": 0,
    ..  ..  
  }
}

But what I did not realize was this did increment when OpenSearch was available again.

Some more thorough testing has shown that the messages are getting through.

Sorry about that.

Thanks very much for your help.

Cheers,
Eddie.

1 Like