Dead Letter Queue questions

[arisbanach]

Still trying to wrap my head around a few things regarding the Dead Letter Queue feature:

1. I sometimes see my DLQ log files contain just a 1. What does this mean?

2. I am running one config for my main pipeline and another to process the errors with DLQ. I am also adding a dead_letter_queue tag for everything processed in my DLQ pipeline so I can see in Kibana which issues there have been. This setup was working for a bit, but then I redid some config settings, deleted the index, and restarted Logstash and now I am not seeing any docs with the dead_letter_queue tag. However, my fixes DLQ directory has a ton of files in it, but my main DLQ directory has very few. Does this mean that they aren't getting indexed into Elasticsearch at all? I haven't seen any logs that say ES indexing failed. I would have expected some kind of error or something if ES couldn't index it. I've also noticed none of the other tags I've been adding in the fixes DLQ config haven't been added either, but I'm not sure why since I'm not getting any indexing errors.

3. Is the DLQ input plugin's commit_offset setting the equivalent of the JDBC input plugin's last_run setting? If they are the same thing, could we perhaps name them the same things so that there is less of a learning curve? I think the concept of a "last run" is something that can be easily applied to many aspects of Logstash.

Thank you.

[guyboertje]

I will invite Rob Bavey to answer your questions.

[RobBavey]

@arisbanach

1. A new log 'segment' is created whenever Logstash is started up, which initially consists of a 'header' value containing internal information about the DLQ, the '1' in this case indicates that it version 1 of the DLQ file format. This may change if and when the file format changes.
2. Can you attach copies of the configuration (with any sensitive information removed), so we can take a look.
3. They are somewhat similar in their effect - determining whether or not to store 'checkpoints', and I tend to agree that having different nomenclature for different plugins can be confusing - although commit_offsets is probably a more accurate term for what the DLQ input does than last_run and vice versa.

Thanks,

Rob