We are using systemctl for running 3 logstash processes, one for each environment : dev, qa and prod on AWS centos box. We are unable to figure out how to configure logging for these 3 different logstash processes as well as using the in-built monitoring api provided by logstash to monitor pipeline, jvm usage, etc. Below are the commands that we are using for running each logstash along with log config.
sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/dev --path.data /tmp/dev/ --path.logs /var/log/logstash-dev.log --log.level error
sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/qa --path.data /tmp/qa/ --path.logs /var/log/logstash-qa.log --log.level error
sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/prod --path.data /tmp/prod/ --path.logs /var/log/logstash-prod.log --log.level info
Additionally, whenever we recreate multiple indices from scratch in an environment, logstash crashes pretty quickly before reindexing all the data. We need to reindex manually for each index one by one and once new index is created logstash process for that environment is stable.
Please help us understand the correct way of handling such a scenario.