Still trying to wrap my head around a few things regarding the Dead Letter Queue feature:
I sometimes see my DLQ log files contain just a
1. What does this mean?
I am running one config for my main pipeline and another to process the errors with DLQ. I am also adding a
dead_letter_queuetag for everything processed in my DLQ pipeline so I can see in Kibana which issues there have been. This setup was working for a bit, but then I redid some config settings, deleted the index, and restarted Logstash and now I am not seeing any docs with the
dead_letter_queuetag. However, my
fixesDLQ directory has a ton of files in it, but my main DLQ directory has very few. Does this mean that they aren't getting indexed into Elasticsearch at all? I haven't seen any logs that say ES indexing failed. I would have expected some kind of error or something if ES couldn't index it. I've also noticed none of the other tags I've been adding in the
fixesDLQ config haven't been added either, but I'm not sure why since I'm not getting any indexing errors.
Is the DLQ input plugin's
commit_offsetsetting the equivalent of the JDBC input plugin's
last_runsetting? If they are the same thing, could we perhaps name them the same things so that there is less of a learning curve? I think the concept of a "last run" is something that can be easily applied to many aspects of Logstash.