Should A Full DLQ Queue Effect Logstash Performance Signifigantly?

When an event is sent to an elasticsearch there are two status codes returned, and the retry handling for them is different.

If the _bulk request returns an error, unless it is a 429 it is retried indefinitely. In some cases this makes no sense (e.g. a 413 error will always recur if retried and thus it will shut down the flow of events).

Each request within the _bulk payload also returns a status. For these, if the code is 400 or 404 then the event is sent to the DLQ if configured, otherwise, if a 409 it is dropped, otherwise it is retried indefinitely (or "for everything else there is Mastercard" as the comment says).

logstash has an at-least-once delivery model, so it leans towards retries. The documentation says "If the DLQ is not enabled, and a mapping error happens, the problem is logged as a warning, and the event is dropped". That is specific to 409 errors, everything else is retried indefinitely.

1 Like