Influx Plugin: Data loss during Output server down-time

Elastic Stack Logstash
1

Hi everyone!
I would like to know how Logstash handles when your output server is down.
In my case, I'm using the Influxdb output plugin.
From what I'm seeing it will keep trying according to the max_retries parameter, but I don't know if configuring it to retry indefinitely could block logstash since my pipelines are triggered every minute .
This makes me think that the data will be lost even though we use a persistent queue configuration since the Influx Connection error is not native to logstash and it seems that no ACK signal is sent.

Could anyone confirm for me what will happen?

Thank you very much
Aurelio

2

logstash has an at-least-once delivery model. It will queue events, either in-memory or in a persistent queue. If the queues fill up it will stop reading the inputs.

3

Thank you very much for your answer

But in order to ensure that data is delivered, should I set the number of retries to infinite?
At the moment, when we reach the maximum number of retries, we would obtain a warning rather than an ERROR. So I don't know if Logstash will guarantee "at-least-once delivery" in this situation

[WARN][logstash.outputs.influxdc][pipeline_id]Connection Error while writint to InfluxDB {:exception=>#InfluxDB::ConnectionError: Tried 3 times to reconnect but failed.>}

Moreover, will this action impact the performance of Logstash?

4

I have continued testing where my inputs come from querying a MySQL Server and I found:

  1. As you said, if the queues fill up it will stop reading inputs.
    2)If the initial input events fill up the queue, the pipeline will freeze just providing partial results. It won't continue event thought partial events has been processed
  2. Retries to export data to influx are scheduled according to the next sequence: 2s,4s,8s,16s,30s,30s...

However, if Logstash is trying to export data to influx during Influx Server restart, the pipeline will die and won't be restarted. Any suggestion?

[2022-05-26T10:32:38,760][WARN ][logstash.outputs.influxdb][pipeline_id][plugin_id] Connection Error while writing to InfluxDB {:exception=>#<InfluxDB::ConnectionError: The server has sent incomplete data (insufficient resources are a possible cause).>}
logstash    

[2022-05-26T10:32:40,781][WARN ][logstash.outputs.influxdb][pipeline_id][plugin_id] Non recoverable exception while writing to InfluxDB {:exception=>#<Errno::EPIPE: Broken pipe - No message available>}
5

No, I am not familiar with what the InfluxDB client considers retryable.

6

I have continued testing and Logstash will exit, meaning data will be lost, in two situations:
-It arrives at the maximum limit of retries, considering it a valid outcome since it was designed for it. (Solvable erasing the limit+Persisten Queue)
-Connection has been established, but Influx didn't manage to ingest the data.

The second point happened to me when I restarted the Influx server simultaneously with one of these retries. Logstash has stored the logs presented in my previous comment and exited. However, Influx has kept just partial information.

I don't know how to solve this situation. Any idea?

P.D. The pipeline froze on my previous post because I configured queue--page_capacity bigger than queue.max_bytes, so the queue wasn't flushed properly.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.