Query on Logstash S3 input

Elastic Stack Logstash
1

Hi Team,
I wanted some clarification on the Logstash S3 input plugin behaviour.
There is an option of "sincedb_path" where as per documentation, it defines where to write the since database (keeps track of the date the last handled file was added to S3).

I am not sure what happens when S3 lifecycle policies move the files around, say from standard storage to S3 Standard-IA storage class or S3 Glacier Flexible Retrieval storage class.

Will logstash consider the files state to have changed and read them back again?
If so I will put in a mitigation of deleting files after they have been read.
But if the implementation assures that the incoming files will be read only once (despite the changes in the storage class of the files) then it makes my life much simpler.

2

Thread bump.
If someone can chime in it will be great.

3

I don't think the date of when a file was added to s3 will change when it is moved to a different storage class, so it would make no difference.

Also, this input does not read anything that is on Glacier.

4

Thanks for the reply. I was a bit worried as long time ago filebeat had this issue of getting confused when reading from windows network/shared drives (it was not a recommended way of using filebeats anyways).

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.