Logstash service restarts constantly and does send data to elasticsearch

[bradfordaemorton (Brad)]

I have a problem with the Logstash service constantly restarting and never sending data elasticsearch. There must be a bad config or a file missing in a location that it is set to read on startup. If I run manually, Logstash starts and sends data to elasticsearch correctly.

I did do a bit of reading and found that the .conf files are meant to be in /etc/logstash/conf.d not /etc/Logstash which is where I had them, but I still have issues even after moving the files. Is there a log file to check for errors? Also, is there a place to download the default .yml and .conf files located in the /etc/logstash/ directory so I copy replace any that I might have updated incorrectly.

[balumurari1]

Hello, if you could provide the input code, which is stopping you to sent data to ES, then it will help you to get resolved quickly.

[bradfordaemorton (Brad)]

Thanks balumurari1,

I am not sure what changed but logs are now arriving at Elastic and starts after a reboot. The logstash services still reboots after a few minutes but doesn't appear to impact log shipping.

The problem I am still having issues with is monitoring Logstash in the elastic/kibana monitoring dashboard.

As you can see I have been able to colelct Elasticsearch, Kibana, Filebeat and Winlogbeat monitring data from different hosts but have not had any success with Logstash.

Can anyone suggest some solutions - my configuration file with the monitoring section is attached.

Cheers,
Brad

------------ X-Pack Settings (not applicable for OSS build)--------------

X-Pack Monitoring

https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html

xpack.monitoring.enabled: true
#xpack.monitoring.elasticsearch.username: logstash_system
#xpack.monitoring.elasticsearch.password: password
xpack.monitoring.elasticsearch.host: ["http://elastic.x.info:9200"]
xpack.monitoring.elasticsearch.url: ["http://elastic.x.info:9200"]
#xpack.monitoring.elasticsearch.ssl.ca: [ "/path/to/ca.crt" ]
#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
#xpack.monitoring.elasticsearch.ssl.truststore.password: password
#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.monitoring.elasticsearch.ssl.keystore.password: password
#xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.elasticsearch.sniffing: false
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true

X-Pack Management

https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html

xpack.management.enabled: true
#xpack.management.pipeline.id: ["main", "apache_logs"]
#xpack.management.elasticsearch.username: logstash_admin_user
#xpack.management.elasticsearch.password: password
xpack.management.elasticsearch.url: ["http://elstic.x.info:9200"]
#xpack.management.elasticsearch.ssl.ca: [ "/path/to/ca.crt" ]
#xpack.management.elasticsearch.ssl.truststore.path: /path/to/file
#xpack.management.elasticsearch.ssl.truststore.password: password
#xpack.management.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.management.elasticsearch.ssl.keystore.password: password
#xpack.management.elasticsearch.ssl.verification_mode: certificate
xpack.management.elasticsearch.sniffing: true
xpack.management.logstash.poll_interval: 5s