I posted a similar post at about a month ago but was not able to get any answers. Please read that link before reading the rest of this post.
The short version of my issue is ES output filter hangs at health check
Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2017-05-02T18:18:12,719][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2017-05-02T18:18:12,821][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"48b96e83-de69-4f77-a213-274b4d43d7d4", :path=>"/usr/share/logstash/data/uuid"}
[2017-05-02T18:18:14,901][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://<username>:xxxxxx@cluster-id.us-east-1.aws.found.io:9243/]}}
[2017-05-02T18:18:14,903][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://<username>:xxxxxx@cluster-id.us-east-1.aws.found.io:9243/, :path=>"/"}
I upgraded my version of logstash and labeled it resolved, but I was back to having the same issue a week later. I have no idea if my logs are processing or not.
Running Logstash 5.2.2 using elastic.co image, not the one hosted on dockerhub, using Docker for Mac
Docker Version 17.0.1-ce-mac5 (16048), channel: Stable, 2 CPUs, 4GB memory
Mac version 10.12.4 (Sierra)
Here's a heavily sanitized version of my logstash.conf
input {
stdin { type => stdin }
file {
path => [<path to file as string>]
codec => multiline {
patterns_dir => [<path to patterns directory as string>]
pattern => "%{TIMESTAMP_ISO8601} "
what => "previous"
negate => true
}
}
}
filter {
grok {
patterns_dir => [<path to patterns directory as string>]
match => {
"message" => <pattern as string">
}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["${ELASTICSEARCH_HOST}"]
user => "${ELASTIC_CLOUD_USER}"
password => "${ELASTIC_CLOUD_PASSWORD}"
index => "%{level}-%{+YYYY.MM.dd}"
}
}
}
When trying to curl I am successful.
$ curl --user $ELASTIC_CLOUD_USER:$ELASTIC_CLOUD_PASSWORD $ELASTICSEARCH_HOST/test_index/logs -d '{
"title": "One", "tags": ["ruby"]
}'
{"_index":"test_index","_type":"logs","_id":"AVvKXcXdOU-C_fihO7_D","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}