Apologies if this is not the way to continue the discussion on a closed topic. But has Logstash 5.6.1 inifinite loop? Encountered a retryable error been resolved?
We are seeing the same issue running Logstash 5.6.5. We see repeated 400 errors with the message 'Encountered a retryable error. Will Retry with exponential backoff ' that accumulate until the Logstash instance fails and no longer processes logs. We are unsure why Logstash stops processing, possibly file handlers, as in IOException: Too many open files - TCP input. There are no OOM errors.
Enabling debug on the elasticsearch output we get:
{"level":"ERROR","loggerName":"logstash.outputs.elasticsearch","timeMillis":1517840567573,"thread":"[main]>worker2","logEvent":{"message":"Encountered a retryable error. Will Retry with exponential backoff ","code":400,"url":"http://127.0.0.1:9200/_bulk","body":"{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}"}}
The errors pile up over time and are never dropped, which appears to be the expected behaviour.
We have DLQ enabled and do see some messages in it but none we believe are related to this issue.
The only way we have found to prevent Logstash from failing is to intermittently restart the process.
The machines have plenty of memory and -xms and -xmx are set at 4g.
Wer'e running Ubuntu 16.04.3