# Configuring logs and monitoring different logstash running on same AWS instance **URL:** \ **Category:** Logstash\ **Created:** [January 11, 2018, 11:55pm UTC](https://discuss.elastic.co/t/configuring-logs-and-monitoring-different-logstash-running-on-same-aws-instance/115192 "2018-01-11T23:55:34Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![kseth](https://avatars.discourse-cdn.com/v4/letter/k/e0b2c6/32.png) [@kseth](https://discuss.elastic.co/u/kseth)\ **Post date:** [January 11, 2018, 11:55pm UTC](https://discuss.elastic.co/t/configuring-logs-and-monitoring-different-logstash-running-on-same-aws-instance/115192/1 "2018-01-11T23:55:34Z")
Hi, We are using systemctl for running 3 logstash processes, one for each environment : dev, qa and prod on AWS centos box. We are unable to figure out how to configure logging for these 3 different logstash processes as well as using the in-built monitoring api provided by logstash to monitor pipeline, jvm usage, etc. Below are the commands that we are using for running each logstash along with log config. `sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/dev --path.data /tmp/dev/ --path.logs /var/log/logstash-dev.log --log.level error` `sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/qa --path.data /tmp/qa/ --path.logs /var/log/logstash-qa.log --log.level error` `sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/prod --path.data /tmp/prod/ --path.logs /var/log/logstash-prod.log --log.level info` Additionally, whenever we recreate multiple indices from scratch in an environment, logstash crashes pretty quickly before reindexing all the data. We need to reindex manually for each index one by one and once new index is created logstash process for that environment is stable. Please help us understand the correct way of handling such a scenario. Thanks! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 16, 2018, 11:04am UTC](https://discuss.elastic.co/t/configuring-logs-and-monitoring-different-logstash-running-on-same-aws-instance/115192/2 "2018-01-16T11:04:44Z")
Isn't `--path.logs` supposed to point to a directory rather than a file? > We are unable to figure out how to configure logging for these 3 different logstash processes as well as using the in-built monitoring api provided by logstash to monitor pipeline, jvm usage, etc. You need to use different logstash.yml files so you can run the monitoring API on different ports (and/or network interfaces). > Additionally, whenever we recreate multiple indices from scratch in an environment, logstash crashes pretty quickly before reindexing all the data. Crashes how? What's the error message? ---
**Author:** ![kseth](https://avatars.discourse-cdn.com/v4/letter/k/e0b2c6/32.png) [@kseth](https://discuss.elastic.co/u/kseth)\ **Post date:** [January 20, 2018, 2:04am UTC](https://discuss.elastic.co/t/configuring-logs-and-monitoring-different-logstash-running-on-same-aws-instance/115192/3 "2018-01-20T02:04:04Z")
> Isn't --path.logs supposed to point to a directory rather than a file? As per the documentation [Logging | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/logging.html) - You can specify the log file location using --path.logs setting. We would want to have all the good features that come with logstash using the log4j. Please give example of how this should be done considering the example that I gave you above. > You need to use different logstash.yml files so you can run the monitoring API on different ports (and/or network interfaces). We are using systemctl to run 3 different logstash process, can you give an example of content of logstash.yml as well as naming convention to distinguish which process should refer to which logstash.yml file. > Crashes how? What's the error message? Usually, a heap dump file is generated like java\_pid5883.hprof. We do not see the logs as we are running these logstash process in background, without any logging in place. Using below: _ExecStart=/bin/sh -c 'sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/dev --path.data /tmp/dev/ --path.logs /var/log/logstash-dev.log --log.level error &'_ ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 23, 2018, 1:02pm UTC](https://discuss.elastic.co/t/configuring-logs-and-monitoring-different-logstash-running-on-same-aws-instance/115192/4 "2018-01-23T13:02:24Z")
> As per the documentation [Logging | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/logging.html) - You can specify the log file location using --path.logs setting. Yes, and according to [Running Logstash from the Command Line | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html) it should be a directory and not a file. > We are using systemctl to run 3 different logstash process, can you give an example of content of logstash.yml as well as naming convention to distinguish which process should refer to which logstash.yml file. You can use the existing file as an example. Just adjust the HTTP monitoring settings. Or, use the same settings file for all instances but use the command line options for adjusting the HTTP monitoring. See the page above. ---
**Author:** ![kseth](https://avatars.discourse-cdn.com/v4/letter/k/e0b2c6/32.png) [@kseth](https://discuss.elastic.co/u/kseth)\ **Post date:** [January 27, 2018, 1:39am UTC](https://discuss.elastic.co/t/configuring-logs-and-monitoring-different-logstash-running-on-same-aws-instance/115192/5 "2018-01-27T01:39:03Z")
Hi Magnus, thanks for your reply. I followed this link and I was able to configure logging for the environments we are running logstash separately for. Cheers! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 24, 2018, 1:39am UTC](https://discuss.elastic.co/t/configuring-logs-and-monitoring-different-logstash-running-on-same-aws-instance/115192/6 "2018-02-24T01:39:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.