Hi, the question is if I have multi logstash servers and set the same conf to collect the s3 bucket, is the result become duplicate? And how to setup load balance for s3 input plugin? Thank you for your time.
Hi, the question is if I have multi logstash servers and set the same conf to collect the s3 bucket, is the result become duplicate?
Yes.
And how to setup load balance for s3 input plugin?
If you ship everything to Elasticsearch (and not anywhere else) you can set the document id of the events that's stored in ES to a checksum of the event contents so that any duplicate(s) will overwrite the any existing document. However, this is obviously very inefficient. The s3 input plugin or Logstash itself has no clustering support to deal with this situation.
Thank you for your reply, I decided to download logs in s3 bucket to local file and then use filebeat to collect it
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.