Is it redundant to use file input plugin with persistent queue?

[danielyahn]

is it redundant to use file input plugin with persistent queue? Does persistent queue provide any additional delivery guarantee?

Let's say file input plugin uses

• read mode
• sinceDB

[Badger]

I don't think it is redundant. A file input could read a small file, flush it to the pipeline, and then delete the file. That is not dependent upon the outputs writing the events to their destinations.

[danielyahn]

Good point about small file scenario.

But I still think that sincedb and persistent queue are playing similar roles: providing durability by writing to disk. I see the differences as well, sincedb keeps track of offset and PQ has actual data. But essentially, you're reading the data from disk (File Input) and writing back to disk (PQ), and reading it again (Filter/Output plugin).