# Logstash outputs elasticsearch: Marking url as dead **URL:** \ **Category:** Logstash\ **Created:** [July 2, 2018, 12:18pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-marking-url-as-dead/138204 "2018-07-02T12:18:52Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![anton-xqz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anton-xqz/32/77279_2.png) [@anton-xqz](https://discuss.elastic.co/u/anton-xqz)\ **Post date:** [July 2, 2018, 12:18pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-marking-url-as-dead/138204/1 "2018-07-02T12:18:52Z")
Hello, We are running ELK cluster on ec2 instance. Elasticsearch cluster in docker on ec2 instances in 3 regions. 33 nodes at total (i3.2xlarge for hot nodes and m5.4xlarge for warm nodes, 25TB data at total, ~2TB data each day) 2 hours ago we started to get connection error on logstash side. Logstash can't send data to elasticsearch via port 9200. No ssl used. Logstash instances sends data to elasticsearch nodes only in same region. Logstash instance lose connectivity, but we are able to perform telnet check successfully (telnet s1infra-esnode-us-1.s1.guru 9200) There is no any error on side of elasticsearch application + no errors in syslog. All ELK stack is running in docker. Logstash version 6.2.4 Ubuntu: 16.04 output { if [anchor] == "operations\_log" { elasticsearch { id =\> "operations\_output01" hosts =\> {{ elasticsearch\_output\_hosts }} user =\> {{ elasticsearch\_output\_user }} password =\> {{ elasticsearch\_output\_password }} index =\> "write\_s1-operations-%{+YYYY.MM}" document\_id =\> "%{log\_id}" manage\_template =\> false } } else if [anchor] == "aws\_billing" { elasticsearch { id =\> "operations\_output02" hosts =\> {{ elasticsearch\_output\_hosts }} user =\> {{ elasticsearch\_output\_user }} password =\> {{ elasticsearch\_output\_password }} index =\> "aws\_billing-%{+YYYY.MM}" document\_id =\> "%{fingerprint}" manage\_template =\> false } } else { elasticsearch { id =\> "logstash\_output01" index =\> "write\_s1-logstash-%{+YYYY.MM.dd}" hosts =\> {{ elasticsearch\_output\_hosts }} user =\> {{ elasticsearch\_output\_user }} password =\> {{ elasticsearch\_output\_password }} manage\_template =\> false } } } Errors: [2018-07-02T11:44:28,020][WARN][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [[http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/)][Manticore::SocketTimeout] Read timed out {:url=\>[http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/), :error\_message=\>"Elasticsearch Unreachable: [[http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/)][Manticore::SocketTimeout] Read timed out", :error\_class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} [2018-07-02T11:44:28,020][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error\_message=\>"Elasticsearch Unreachable: [[http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/)][Manticore::SocketTimeout] Read timed out", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will\_retry\_in\_seconds=\>2} [2018-07-02T11:44:30,409][WARN][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [[http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/)][Manticore::SocketTimeout] Read timed out {:url=\>[http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/), :error\_message=\>"Elasticsearch Unreachable: [[http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/)][Manticore::SocketTimeout] Read timed out", :error\_class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} [2018-07-02T11:44:30,409][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error\_message=\>"Elasticsearch Unreachable: [[http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/)][Manticore::SocketTimeout] Read timed out", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will\_retry\_in\_seconds=\>2} [2018-07-02T11:44:32,355][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/), :path=\>"/"} [2018-07-02T11:44:32,357][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/)"} [2018-07-02T11:44:32,358][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/), :path=\>"/"} [2018-07-02T11:44:32,360][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-6.s1.guru:9200/)"} [2018-07-02T11:44:33,499][WARN][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [[http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/)][Manticore::SocketTimeout] Read timed out {:url=\>[http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/), :error\_message=\>"Elasticsearch Unreachable: [[http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/)][Manticore::SocketTimeout] Read timed out", :error\_class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} [2018-07-02T11:44:33,499][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error\_message=\>"Elasticsearch Unreachable: [[http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/](http://logstash:xxxxxx@s1infra-esnode-eu-1.s1.guru:9200/)][Manticore::SocketTimeout] Read timed out", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will\_retry\_in\_seconds=\>2} All "limits" checked. No IO disk issues found. Do some one has idea where ti debug? ---
**Author:** ![anton-xqz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anton-xqz/32/77279_2.png) [@anton-xqz](https://discuss.elastic.co/u/anton-xqz)\ **Post date:** [July 4, 2018, 6:36pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-marking-url-as-dead/138204/2 "2018-07-04T18:36:55Z")
Logstash.yml: xpack.monitoring.enabled: true xpack.monitoring.elasticsearch: url: {{ xpack\_monitoring\_es\_url }} username: {{ xpack\_monitoring\_es\_username }} password: {{ xpack\_monitoring\_es\_password }} path.data: /usr/share/logstash/data path.config: /usr/share/logstash/pipeline http.host: "0.0.0.0" http.port: 9600 path.logs: /usr/share/logstash/logs pipeline.workers: 128 pipeline.batch.size: 1250 startup.options: JAVACMD=/usr/bin/java LS\_HOME=/usr/share/logstash LS\_SETTINGS\_DIR=/etc/logstash LS\_OPTS="--path.settings ${LS\_SETTINGS\_DIR}" LS\_JAVA\_OPTS="" LS\_PIDFILE=/var/run/logstash.pid LS\_USER=logstash LS\_GROUP=logstash LS\_GC\_LOG\_FILE=/var/log/logstash/gc.log LS\_OPEN\_FILES=16384 LS\_NICE=19 SERVICE\_NAME="logstash" SERVICE\_DESCRIPTION="logstash" jvm.options: -Xms55g -Xmx55g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 1, 2018, 6:36pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-marking-url-as-dead/138204/3 "2018-08-01T18:36:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.