How is the state (last read file or position in a file) is maintained for multiple pods running logstash

Elastic Stack Logstash
1

Hi Everyone,
I just have a query regarding last read s3 file or the last read line in the s3 file.
How or where is that data stored? and how is that shared among multiple pods running logstash?
What if the pod is replaced? How does the new pod know the last read location in s3?

input plugin: s3
output plugin Elasticsearch

2

The plugin stores the time of the last read inside a file in the configured sincedb_path.

It is not shared, the plugin was not build to have multiple inputs consuming from the same bucket, so to avoid duplication you can have only one input per bucket or prefix.

3

Thanks for the reply @leandrojmp .
A follow up question on that.
If the logstash container restarts, the file might get deleted from the configured or default sincedb_path. How is that handled in case of logstash running as a container?

4

Check the documentation for the sincedb_path setting.

You will need to configure this setting to a persistent volume that would be reused by a new container.

5

Got it. Thanks @leandrojmp

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.