# Marking url as dead. Last error: \[LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError\] Elasticsearch Unreachable **URL:** \ **Category:** Logstash\ **Created:** [January 3, 2019, 8:00am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761 "2019-01-03T08:00:50Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 3, 2019, 8:00am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/1 "2019-01-03T08:00:50Z")
Hi I am not able to send the data to my AWS -ES . I am getting the below error [logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://search-mb-production-app-\*\*\*\*\*\*\*\*\*.us-west-2.es.amazonaws.com:80/][Manticore::SocketTimeout] Read timed out {:url=\>http://search-mb-production-app-\*\*\*\*\*\*[us-west-2.es.amazonaws.com:80/](http://us-west-2.es.amazonaws.com:80/), :error\_message=\>"Elasticsearch Unreachable: [http://search-mb-production-app-\*\*\*\*\*\*.us-west-2.es.amazonaws.com:80/][Manticore::SocketTimeout] Read timed out", :error\_class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} . ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 3, 2019, 8:04am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/2 "2019-01-03T08:04:29Z")
I believe you need to use the amazon\_es output plugin to send data to AWS ES. ---
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 3, 2019, 8:06am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/3 "2019-01-03T08:06:47Z")
Yes I have used the AWS es output plugin . Below is the output plugin used output { file{ path =\> "/var/log/logstash/alb-accesslogs.log" } ``` elasticsearch { hosts => "http://search-mb-production-app- *******.us-west-2.es.amazonaws.com:80" index => "alb-accesslog-%{+YYYY.MM.dd}" } ``` } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 3, 2019, 8:11am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/4 "2019-01-03T08:11:53Z")
That is the wrong plugin. You need to install and use the `amazon_es` plugin supplied by AWS. ---
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 3, 2019, 8:14am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/5 "2019-01-03T08:14:26Z")
Same output plugin is used for another logstash server and its working fine. Even this was working fine till morning and from past 1hr its not working and getting error. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 3, 2019, 8:20am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/6 "2019-01-03T08:20:50Z")
I am not very familiar with AWS ES so will probably not be able to help much in that case. You may want to take it up with their support. ---
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 3, 2019, 10:21am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/7 "2019-01-03T10:21:59Z")
I changed output plugin . output { file{ path =\> "/var/log/logstash/alb-accesslogs.log" } ``` amazon_es { hosts => "http://search-mb-production-app- *********.us-west-2.es.amazonaws.com:80" region => "us-west-2" index => "alb-accesslog-%{+YYYY.MM.dd}" } ``` } but it failed to execute the plugin itself. '''[2019-01-03T10:20:40,342][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"alb-accesslogs", :plugin=\>"#LogStash::OutputDelegator:0x55486400", :error=\>"Explicit value for 'port' was declared, but it is different in one of the URLs given! Please make sure your URLs are inline with explicit values. The URLs have the property set to '80', but it was also set to '443' explicitly", :thread=\>"#\"} [2019-01-03T10:20:41,662][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"askadoc-accesslogs", :plugin=\>"#LogStash::OutputDelegator:0x7448ae6c", :error=\>"undefined method `match' for nil:NilClass\nDid you mean? catch", :thread=>"#"} [2019-01-03T10:20:41,671][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline_id=>"askadoc-accesslogs", :exception=>#, :backtrace=\>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/endpoint\_provider.rb:72:in `block in partition_matching_region'", "org/jruby/RubyEnumerable.java:643:in`find'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/endpoint\_provider.rb:71:in `partition_matching_region'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/endpoint_provider.rb:60:in`get\_partition'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/endpoint\_provider.rb:14:in `signing_region'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/endpoint_provider.rb:89:in`signing\_region'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.177/lib/aws-sdk-core/signers/v4.rb:46:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:111:in`perform\_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon\_es-6.4.0-java/lib/logstash/outputs/amazon\_es/http\_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:245:in`block in healthcheck!'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:241:in`healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon\_es-6.4.0-java/lib/logstash/outputs/amazon\_es/http\_client/pool.rb:341:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:71:in`start'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http\_client.rb:302:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client.rb:64:in`initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http\_client\_builder.rb:103:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:99:in`build'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch.rb:234:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/common.rb:25:in`register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:102:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:46:in`register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:242:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:253:in`block in register\_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:253:in`register\_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:594:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:263:in`start\_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:200:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:160:in`block in start'"], :thread=\>"#\"} [2019-01-03T10:20:41,693][ERROR][logstash.agent] Failed to execute action {:id=\>:"askadoc-accesslogs", :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: PipelineAction::Create, action\_result: false", :backtrace=\>nil} [2019-01-03T10:20:42,101][ERROR][logstash.p''' ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 3, 2019, 10:41am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/8 "2019-01-03T10:41:18Z")
I have never used AWS ES nor the amazon\_es plugin, so will not be able to help much there. I generally use the [our Elasticsearch Service](https://www.elastic.co/cloud/elasticsearch-service) instead, which works well with the standard elasticsearch output plugin. It also supports hot/warm architectures and provides access to our commercial features. ---
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 3, 2019, 12:40pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/9 "2019-01-03T12:40:36Z")
HI Christian can you help with this please . I am trying to start the logstash and getting the below logs Jan 03 11:49:52 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: Unit logstash.service entered failed state. Jan 03 11:49:52 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: logstash.service failed. Jan 03 11:49:52 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: Started logstash. Jan 03 11:49:52 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: Starting logstash... Jan 03 12:32:16 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: Stopping logstash... Jan 03 12:33:38 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: logstash.service: main process exited, code=exited, status=143/n/a Jan 03 12:33:38 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: Unit logstash.service entered failed state. Jan 03 12:33:38 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: logstash.service failed. Jan 03 12:33:38 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: Started logstash. Jan 03 12:33:38 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: Starting logstash... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 3, 2019, 12:52pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/10 "2019-01-03T12:52:12Z")
What does the Logstash logs say? ---
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 3, 2019, 12:53pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/11 "2019-01-03T12:53:33Z")
I am trying to start the logstash with sudo service logstash start and getting logs ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2019-01-03 12:48:05 UTC; 10s ago Main PID: 14852 (logstash) CGroup: /system.slice/logstash.service ├─14852 /bin/bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash └─14853 /bin/bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash Jan 03 12:48:05 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: logstash.service: main process exited, code=exited, status=143/n/a Jan 03 12:48:05 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: Unit logstash.service entered failed state. Jan 03 12:48:05 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: logstash.service failed. Jan 03 12:48:05 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: Started logstash. Jan 03 12:48:05 ip-172-31-40-86.us-west-2.compute.internal systemd[1]: Starting logstash.. ---
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 3, 2019, 12:54pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/12 "2019-01-03T12:54:35Z")
its active but logstash.service:main process exited, code=exited, status=143/n/a ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 3, 2019, 12:55pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/13 "2019-01-03T12:55:43Z")
Did you resolve the issue with the plugin you mentioned earlier? Does it start when not started as a service? ---
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 3, 2019, 12:57pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/14 "2019-01-03T12:57:03Z")
yes I tired installing the amazon\_es plugin but it failed to install ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 3, 2019, 12:58pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/15 "2019-01-03T12:58:32Z")
Then that is probably what is causing the problems then. ---
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 3, 2019, 12:59pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/16 "2019-01-03T12:59:38Z")
yeah that is why I am uninstalling the logstash and reinstalling it ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 3, 2019, 1:01pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/17 "2019-01-03T13:01:27Z")
How did you install the plugin? Did you look at [the documentation](https://github.com/awslabs/logstash-output-amazon_es) about how to use it? As far as I know it is not a drop-in replacement for the elasticsearch plugin, and is configured differently. ---
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 3, 2019, 1:02pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/18 "2019-01-03T13:02:26Z")
yes i referred the same documentation ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 3, 2019, 1:03pm UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/19 "2019-01-03T13:03:11Z")
What does your config look like? Does it match the example in the documentation? ---
**Author:** ![arun5635](https://avatars.discourse-cdn.com/v4/letter/a/c89c15/32.png) [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Post date:** [January 4, 2019, 5:57am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761/20 "2019-01-04T05:57:55Z")
sorry for late reply. This is my conf file . Now i am not using the elastic search output plugin still i am getting same error. input { udp{ port =\> 5978 } } filter { grok{ match =\> {"message" =\> '(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP\_ISO8601:timestamp8601}) %{WORD:appName}/%{WORD:containerId}[%{INT:randomId:int}]: %{GREEDYDATA:logMessage}'} } } output { ``` file{ path => "/var/log/logstash/application/docker-%{appName}-%{containerId}.log" } file{ path => "/var/log/logstash/services/docker-%{appName}.log" } ``` } [Next page](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761.md?page=2)