However, these 2 .conf files each reference a different bucket. What I don't understand is, upon running Logstash, it seems the indices are sharing data even though each index should correspond to an S3 bucket. As one gets bigger, so does the other.
How do I keep this from happening and ensure that files from their respective buckets end up in their corresponding index?
WIth this configuration you have just one pipeline, when logstash starts it will merge all the files in the /etc/logstash/conf.d path as it was just one file.
Since you are not using conditionals in your output, the data from both inputs will be sent to all the outputs, you need to change the pipelines.yml file to use multiple pipelines.
You would need to stop logstash, remove the sincedb files created by the s3 input and start it again, the sincedb file for the s3 input basically just stores the date of the last object that was read from the bucket.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.