S3 input plugin. Parse again an S3 bucket

Elastic Stack Logstash
1

Hello,

I have an S3 bucket filled with kinesis logs.
I want to re scan entirely this bucket but I didn't manage to do it.

I've tried to delete the sincedb file ( /var/lib/logstash/plugins/inputs/s3/sincedb_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx), but after restarting logstash, it regenerate it, start re rescan from the first file found in S3 bucket (2 weeks ago) BUT it only send to elasticsearch logs starting from the current date/time.

Can anyone tell me why logstash don't send to ES old logs but only logs from its starting time ?

Thank you !

Adrien

2

Was logstash running when you deleted the sincedb? If so, it will have persisted the in-memory sincedb when it shut down. Shut logstash down before you remove the sincedb.

3

Hi,

Yes logstash is stopped when I remoce the sinceDB.

  • I stop Logstash
  • I remove sincedb
  • I start logstash.

If I watch my sinceDB file when logstash just recreate it, I can see that it parse all the S3 logs . But the problem is that logstash send to elasticsearch ONLY logs from the date/time at which it was started.

Thanks

Adrien