Logstash noisy and throwing errors about no elastic connection


(Will B) #1

I am running logstash from a docker container built from elastic.co's 5.3.2 container (latest at time of post). As part of testing I launch this container manually.

docker run --rm -it --name=testy --volume="$(pwd)/logtest/:/settings/:ro" xxxxxxx/logstash:5.3.2 -f /settings/conf

where i use stdin and stdout with ruby debug to test filter configs that I have defined in ./logtest/conf this has previously been a very convenient way to quickly perfect filters.

Nowhere in my pipeline config am I using elasticsearch, yet now (since dockerhub's logstash:5.0.1), I have a huge amount of noice from logstash trying to contact elasticsearch to write monitoring data on a default defined hostname, failing of course, and throwing noisy errors, which clogs up my stdout from the interactive docker container.

[2017-05-02T16:21:24,431][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x409fb6a9 URL:http://logstash_system:xxxxxx@elasticsearch:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Temporary failure in name resolution"}
[2017-05-02T16:21:24,431][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s][Manticore::ResolutionFailure] elasticsearch {:url=>http://logstash_system:xxxxxx@elasticsearch:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s, :error_message=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2017-05-02T16:21:24,432][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s][Manticore::ResolutionFailure] elasticsearch", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2017-05-02T16:21:26,434][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2017-05-02T16:21:26,434][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
[2017-05-02T16:21:29,433][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@elasticsearch:9200/, :path=>"/"}
[2017-05-02T16:21:29,439][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x61737333 URL:http://logstash_system:xxxxxx@elasticsearch:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}```

I also cannot seem to find a way here https://www.elastic.co/guide/en/logstash/5.3/logstash-settings-file.html to prevent logstash doing this.

How do I make it shut up/where would I properly configure this in production where it would be nice to have it write monitoring data to elasticsearch?

(Will B) #2

Welp, no responses


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.