Hi,
I've successfully configured logstash and was able to run the pipeline and abled to deliver the data from s3 to elasticsearch. But, Logstash is not continuously delivering the data from S3. Data is getting delivered only when I run the "sudo bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash/" command.
Moreover, right now, if I'm running the same command, I'm getting the following error:
[FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
I'm not using multiple pipelines or conf files. Still I'm getting the above error. Can you please let me know the mistake I'm performing? Also, Can you please tell me a way such that logstash will deliver the logs to elasticsearch as soon as there is a log file stored in S3 bucket.
Thank you!
@magnusbaeck can you please help?