DLQ - Managing Dead Letter Queue Size- To /dev/null?

[scott_stash]

Looking through the DLQ docs https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html it looks like I can set a maximum size for each DLQ file, but it sounds like the rollover will just create a new file.

My main concern is filling up my drive, and having to manage the storage location for data I generally do not review.

Would it just make sense to set the file path to /dev/null? Or is there a better way to manage this?

[Badger]

@scott_stash

Would it just make sense to set the file path to /dev/null?

/dev/null is not a directory, so I would expect that if logstash tried to create a file inside it then it would get an exception.

However, my understanding is that each file in the DLQ is limited by the maximum segment size (fixed at 10MB) and the overall queue size is limited by the maximum queue size (1GB) which you can change.

[scott_stash]

Thanks a lot Badger,

I tried 30MB, and get this: (there are 3 10MB files as you suggested)

Jun 24 12:02:00 hostname logstash[7379]: message repeated 3 times: [ [2020-06-24T12:02:00,143][ERROR][org.logstash.common.io.DeadLetterQueueWriter] cannot write event to DLQ: reached maxQueueSize of 31457280]
Jun 24 12:02:00 hostname logstash[7379]: [2020-06-24T12:02:00,144][ERROR][org.logstash.common.io.DeadLetterQueueWriter] cannot write event to DLQ: reached maxQueueSize of 31457280

I'm not sure how deadletterqueue tracks, but it took me a restart and deleting the files to get it to write again to the queue again. I'm not sure if the events are just discarded at this point, or if its trying 5 times then giving up for each event? I was hoping it would just over write older events, but I guess thats not the case.

Also it appears its 30MB(max size) per pipeline.

You were right about /dev/null as well, Logstash refused to start up.
I think this is better then permanent 400's looping because things seems to be processing, even though its slow. With the 400 it was also a lot more difficult to track down my problem entries, where with a DLQ they are readible in the 1.log, 2.log.

[Badger]

My reading of the code is that it discards events if it cannot write them to the DLQ.