I seem to be having a strange problem but quite possibly be a misconfiguration on my side.
We are trying to monitor AWS S3 logs which have the following structure
s3://bucket/projectA s3://bucket/projectB s3://bucket/projectC s3://bucket/projectN etc..
We have logstash configured for each of these projects in a different configuration file as well as some of the grok expressions are different based on the type of Load Balancer that the project has. For example we have,
/etc/logstash/conf.d/projectA.conf /etc/logstash/conf.d/projectB.conf /etc/logstash/conf.d/projectC.conf /etc/logstash/conf.d/projectN.conf etc..
If I start logstash service it reads all the conf files and populate the indices but sometimes data from ProjectA is seen on ProjectC and so on.
I did a fresh start and only started one configuration file at a time and that seems to house the data to it's own indices.
Do we need to configure in a different way for a requirement like this?