Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}

Hello, I am running Logstash and Elasticsearch both in docker. Elasticsearch takes some time to set up after very first build. Logstash at some moment starts to log errors like the one in topic title and even if Elasticsearch is up, it won't stop, even though everything is running and works. I can request Elasticsearch and my data is there, so pipelines have to work. How can i get rid of this error? Thank you.

Example of errors which occurs every 5 seconds:

[2021-09-26T12:38:50,849][ERROR][logstash.outputs.elasticsearch][users_pipeline_delete][22a9ebe307fdd5a326b9f2dd14f4b953ce1bcd55ee4079220239a8ad983368f0] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2021-09-26T12:38:50,874][ERROR][logstash.outputs.elasticsearch][users_pipeline_upsert][cd3d623ffd8a303ab013ad152a605bcefc818b61a962451a4ec2eef3ac57fd7a] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2021-09-26T12:38:55,848][ERROR][logstash.outputs.elasticsearch][users_pipeline_upsert][cd3d623ffd8a303ab013ad152a605bcefc818b61a962451a4ec2eef3ac57fd7a] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2021-09-26T12:38:55,849][ERROR][logstash.outputs.elasticsearch][users_pipeline_delete][22a9ebe307fdd5a326b9f2dd14f4b953ce1bcd55ee4079220239a8ad983368f0] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2021-09-26T12:39:00,301][INFO ][logstash.inputs.jdbc     ] (0.001245s) SELECT version()
[2021-09-26T12:39:00,306][INFO ][logstash.inputs.jdbc     ] (0.001616s) SELECT id_row AS id FROM User_log
                      WHERE activity = 'delete' AND time > '2021-09-26 12:38:01'
[2021-09-26T12:39:00,324][INFO ][logstash.inputs.jdbc     ] (0.001869s) SELECT version()
[2021-09-26T12:39:00,330][INFO ][logstash.inputs.jdbc     ] (0.003389s) SELECT * FROM User
                      WHERE id IN (SELECT id_row FROM User_log
				   WHERE activity != 'delete' AND time > '2021-09-26 12:38:01')
[2021-09-26T12:39:00,432][ERROR][logstash.outputs.elasticsearch][users_pipeline_upsert][cd3d623ffd8a303ab013ad152a605bcefc818b61a962451a4ec2eef3ac57fd7a] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2021-09-26T12:39:00,846][ERROR][logstash.outputs.elasticsearch][users_pipeline_delete][22a9ebe307fdd5a326b9f2dd14f4b953ce1bcd55ee4079220239a8ad983368f0] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2021-09-26T12:39:00,875][ERROR][logstash.outputs.elasticsearch][users_pipeline_upsert][cd3d623ffd8a303ab013ad152a605bcefc818b61a962451a4ec2eef3ac57fd7a] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2021-09-26T12:39:05,852][ERROR][logstash.outputs.elasticsearch][users_pipeline_upsert][cd3d623ffd8a303ab013ad152a605bcefc818b61a962451a4ec2eef3ac57fd7a] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}

One of my pipeline's configuration:

input {
    jdbc {
        jdbc_driver_library => "/usr/share/jdbc/mysql-connector-java-8.0.21.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_connection_string => "jdbc:mysql://db:3306/instamini"
        jdbc_user => "zlackbra"
        jdbc_password => "password"
	schedule => "* * * * *"
        statement => "SELECT * FROM User
                      WHERE id IN (SELECT id_row FROM User_log
				   WHERE activity != 'delete' AND time > :sql_last_value)"
     }
}

filter {

}

output {
    elasticsearch {
        hosts => ["elasticsearch:9200"]
        action => update
        doc_as_upsert => true
        document_id => "%{id}"
        index => "users"
    }
}

Every time the pipeline sends a batch of events to the output, it will log this if the output was not initialized correctly. The exception it is logging is the one returned by finish_register which checks the Elasticsearch version, installs the template if necessary, etc. All of that requires a working Elasticsearch connection, and it is telling you it did not have one.

I suspect the error that prevented a connection being established was logged once at startup, so you should look earlier in your logs.

I checked logs from beginning and everywhere it is just saying that connection is not available. For example:

[2021-09-26T17:47:02,636][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2021-09-26T17:47:04,029][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2021-09-26T17:47:04,069][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused) {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2021-09-26T17:47:04,071][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

But I get it. These errors are because of starting it in docker, so it takes time to startup Elasticsearch and logstash must wait. So tI am thinking that these errors are normal, after time connection is restored, but this error in title will not disappear in first build, in second build is everything okay. Logs of restoring connection:

[2021-09-26T17:50:35,710][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://elasticsearch:9200"]}
[2021-09-26T17:50:35,753][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2021-09-26T17:50:35,778][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2021-09-26T17:50:35,792][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (7.14.0) {:es_version=>7}
[2021-09-26T17:50:35,793][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}

The main thing which is confusing me is, that even i got exceptions, because of not available connections, pipeline and Elasticsearch are working. I can successfully do request to Elasticsearch and all data from database are moved to Elasticsearch frequently as you can see in very first logs of this topic, when pipeline gets data from jdbc source. All this happens only in first build of docker compose, when i have no containers and volumes created. If you find something helpful in these logs, I would be glad. Thank you for your fast answer. :slight_smile:

I am adding here logs from second build, when containers, volumes,... are already created and logstash is not logging errors anymore:

[2021-09-26T18:26:39,213][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-09-26T18:26:40,138][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2021-09-26T18:26:40,409][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2021-09-26T18:26:40,446][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused) {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2021-09-26T18:26:40,448][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2021-09-26T18:26:40,464][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2021-09-26T18:26:40,734][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-09-26T18:26:41,962][INFO ][org.reflections.Reflections] Reflections took 248 ms to scan 1 urls, producing 120 keys and 417 values 
[2021-09-26T18:26:42,913][WARN ][deprecation.logstash.inputs.jdbc] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-09-26T18:26:42,935][WARN ][deprecation.logstash.inputs.jdbc] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-09-26T18:26:43,050][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-09-26T18:26:43,092][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-09-26T18:26:43,099][INFO ][logstash.outputs.elasticsearch][users_pipeline_delete] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
[2021-09-26T18:26:43,110][INFO ][logstash.outputs.elasticsearch][users_pipeline_delete] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2021-09-26T18:26:43,126][INFO ][logstash.outputs.elasticsearch][users_pipeline_upsert] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
[2021-09-26T18:26:43,128][WARN ][logstash.outputs.elasticsearch][users_pipeline_delete] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2021-09-26T18:26:43,177][INFO ][logstash.outputs.elasticsearch][users_pipeline_upsert] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2021-09-26T18:26:43,205][WARN ][logstash.outputs.elasticsearch][users_pipeline_upsert] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2021-09-26T18:26:43,205][INFO ][logstash.javapipeline    ][users_pipeline_delete] Starting pipeline {:pipeline_id=>"users_pipeline_delete", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/usr/share/logstash/pipeline/users_delete.conf"], :thread=>"#<Thread:0x72fcbe5d@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[2021-09-26T18:26:43,246][INFO ][logstash.javapipeline    ][users_pipeline_upsert] Starting pipeline {:pipeline_id=>"users_pipeline_upsert", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/usr/share/logstash/pipeline/users_upsert.conf"], :thread=>"#<Thread:0x6d857eb4@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[2021-09-26T18:26:45,423][INFO ][logstash.javapipeline    ][users_pipeline_upsert] Pipeline Java execution initialization time {"seconds"=>2.18}
[2021-09-26T18:26:45,425][INFO ][logstash.javapipeline    ][users_pipeline_delete] Pipeline Java execution initialization time {"seconds"=>2.22}
[2021-09-26T18:26:45,544][INFO ][logstash.javapipeline    ][users_pipeline_delete] Pipeline started {"pipeline.id"=>"users_pipeline_delete"}
[2021-09-26T18:26:45,562][INFO ][logstash.javapipeline    ][users_pipeline_upsert] Pipeline started {"pipeline.id"=>"users_pipeline_upsert"}
[2021-09-26T18:26:45,669][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:users_pipeline_delete, :users_pipeline_upsert], :non_running_pipelines=>[]}
[2021-09-26T18:26:48,167][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2021-09-26T18:26:48,242][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2021-09-26T18:26:58,664][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2021-09-26T18:26:58,663][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2021-09-26T18:26:59,201][INFO ][logstash.outputs.elasticsearch] Elasticsearch version determined (7.14.0) {:es_version=>7}
[2021-09-26T18:26:59,201][INFO ][logstash.outputs.elasticsearch] Elasticsearch version determined (7.14.0) {:es_version=>7}
[2021-09-26T18:26:59,202][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-09-26T18:26:59,203][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-09-26T18:27:10,461][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2021-09-26T18:27:10,784][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2021-09-26T18:27:10,805][INFO ][logstash.licensechecker.licensereader] Elasticsearch version determined (7.14.0) {:es_version=>7}
[2021-09-26T18:27:10,806][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-09-26T18:27:13,175][INFO ][logstash.outputs.elasticsearch] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2021-09-26T18:27:13,235][INFO ][logstash.outputs.elasticsearch] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2021-09-26T18:27:13,238][INFO ][logstash.outputs.elasticsearch] Installing Elasticsearch template {:name=>"logstash"}
[2021-09-26T18:27:13,278][INFO ][logstash.outputs.elasticsearch] Installing Elasticsearch template {:name=>"logstash"}
[2021-09-26T18:27:40,497][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2021-09-26T18:27:40,497][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2021-09-26T18:27:40,936][WARN ][deprecation.logstash.outputs.elasticsearchmonitoring] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-09-26T18:27:40,947][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://elasticsearch:9200"]}
[2021-09-26T18:27:40,958][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2021-09-26T18:27:40,973][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2021-09-26T18:27:40,984][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (7.14.0) {:es_version=>7}
[2021-09-26T18:27:40,984][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-09-26T18:27:41,024][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[2021-09-26T18:27:41,025][WARN ][logstash.outputs.elasticsearchmonitoring] Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[2021-09-26T18:27:41,026][WARN ][logstash.javapipeline    ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2021-09-26T18:27:41,031][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x577a9cc5@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[2021-09-26T18:27:41,052][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.02}
[2021-09-26T18:27:41,063][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2021-09-26T18:27:41,091][INFO ][logstash.agent           ] Pipelines running {:count=>3, :running_pipelines=>[:users_pipeline_delete, :users_pipeline_upsert, :".monitoring-logstash"], :non_running_pipelines=>[]}
[2021-09-26T18:28:00,212][INFO ][logstash.inputs.jdbc     ] (0.002265s) SELECT version()
[2021-09-26T18:28:00,221][INFO ][logstash.inputs.jdbc     ] (0.003648s) SELECT id_row AS id FROM User_log
                      WHERE activity = 'delete' AND time > '2021-09-26 18:27:02'
[2021-09-26T18:28:00,225][INFO ][logstash.inputs.jdbc     ] (0.002492s) SELECT version()
[2021-09-26T18:28:00,241][INFO ][logstash.inputs.jdbc     ] (0.010021s) SELECT * FROM User
                      WHERE id IN (SELECT id_row FROM User_log
				   WHERE activity != 'delete' AND time > '2021-09-26 18:27:02')