# Errors with ElasticSearch Setup

**URL:** <https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049>\
**Category:** Elasticsearch\
**Created:** [September 26, 2018, 4:13pm UTC](https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049 "2018-09-26T16:13:14Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![psd401](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@psd401](https://discuss.elastic.co/u/psd401)\
**Post date:** [September 26, 2018, 4:13pm UTC](https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049/1 "2018-09-26T16:13:14Z")

</div>

This is my first time dabbling with ElasticSearch, and unfortunately, we are experiencing a multitude of errors and I am not entirely sure where to exactly go to pinpoint this problem. Basically, we are running a Linux server that is running six different ElasticSearch containers (via Docker) along with Kibana, LogStash, and Curator. All of these services are not functioning (as in, Kibana for instance, keeps displaying "Elasticsearch plugin is red").

I have attached logs for each of the services:

- **ElasticSearch container** (Host, Main):

> **Summary**
>
> ```
> [2018-09-26T15:53:35,467][WARN][o.e.g.DanglingIndicesState] [elasticsearch] [[logstash-printerlogs-2016.03.26/NhTPw446TgOrtQwccd4tKQ]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
> [2018-09-26T15:53:35,467][WARN][o.e.g.DanglingIndicesState] [elasticsearch] [[logstash-printerlogs-2017.06.01/Kfi1l-hWR0mh5ZWHJyHIfg]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
> [2018-09-26T15:53:36,218][WARN][o.e.c.a.s.ShardStateAction] [elasticsearch] [logstash-mitelcall-2018.09.26][3] received shard failed for shard id [[logstash-mitelcall-2018.09.26][3]], allocation id [cNnVZpWoQ_GfTmPWQvYuyA], primary term [2], message [mark copy as stale]
> [2018-09-26T15:53:37,730][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [elasticsearch] failed to put mappings on indices [[[winlogbeat-2018.09.18/zD0J53zoQx2hdtbFymLWGg]]], type [doc]
> org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
> at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$null$0(ClusterService.java:255) ~[elasticsearch-5.5.0.jar:5.5.0]
> at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_131]
> at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$onTimeout$1(ClusterService.java:254) ~[elasticsearch-5.5.0.jar:5.5.0]
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
> [2018-09-26T15:53:38,482][WARN][o.e.c.a.s.ShardStateAction] [elasticsearch] [logstash-mitelcall-2018.09.26][4] received shard failed for shard id [[logstash-mitelcall-2018.09.26][4]], allocation id [rmrZ38sVRu68T0etBwFbfQ], primary term [2], message [mark copy as stale]
> [2018-09-26T15:53:38,663][WARN][o.e.c.a.s.ShardStateAction] [elasticsearch] [logstash-nagios][4] received shard failed for shard id [[logstash-nagios][4]], allocation id [rQQjk0yYQuSz1N4reGF2DA], primary term [0], message [master {elasticsearch}{CgpEifTvQzCQWNYJtLhwqw}{e9aaCpnbRlSerje_tO-ocg}{172.18.0.10}{172.18.0.10:9300}{ml.enabled=true} has not removed previously failed shard. resending shard failure]
> 
> ```

- **ElasticSearch Container** (One of the nodes):

> **Summary**
>
> ```
> [2018-09-26T16:05:29,671][WARN][o.e.g.DanglingIndicesState] [elasticsearch1] [[logstash-test123-2016.05.25/TdsA38fHSr2jrFL9u_goSA]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
> [2018-09-26T16:05:29,671][WARN][o.e.g.DanglingIndicesState] [elasticsearch1] [[logstash-test123-2016.09.14/dxmirl33SiCUHsdIEqlhGQ]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
> [2018-09-26T16:05:29,671][WARN][o.e.g.DanglingIndicesState] [elasticsearch1] [[logstash-test123456-2017.07.23/AREbqw3WSsKEM-kASSWDlg]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
> [2018-09-26T16:05:29,671][WARN][o.e.g.DanglingIndicesState] [elasticsearch1] [[logstash-test1234-2017.07.31/Onj_5dnyRBKE0m3x89Oj-g]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
> [2018-09-26T16:05:29,671][WARN][o.e.g.DanglingIndicesState] [elasticsearch1] [[logstash-test123-2016.01.30/ZZhz7aJzSlyKIOt5AWAwmg]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
> 
> ```

Any help and pointers on where to go to solve this problem would be appreciated!

---

<div class="post-metadata">

**Author:** ![psd401](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@psd401](https://discuss.elastic.co/u/psd401)\
**Post date:** [September 26, 2018, 4:13pm UTC](https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049/2 "2018-09-26T16:13:47Z")

</div>

More error logs (unable to submit in original post due to character size limit):

- **LogStash Container**

> **Summary**
>
> ```
> 16:08:11.786 [[main]>worker8] WARN logstash.outputs.elasticsearch - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
> 16:08:11.786 [[main]>worker8] ERROR logstash.outputs.elasticsearch - Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
> 16:08:13.598 [Ruby-0-Thread-18: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:224] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>#<URI::HTTP:0x217248a7 URL:http://elasticsearch:9200/>}
> 16:08:16.504 [Ruby-0-Thread-13: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:224] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
> 16:08:16.508 [Ruby-0-Thread-13: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:224] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>#<URI::HTTP:0x31672d0a URL:http://elasticsearch:9200/>}
> 
> ```

- **Kibana Container**

> **Summary**
>
> ```
> {"type":"log","@timestamp":"2018-09-26T15:32:34Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
> {"type":"log","@timestamp":"2018-09-26T15:32:34Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
> {"type":"log","@timestamp":"2018-09-26T15:32:37Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
> {"type":"log","@timestamp":"2018-09-26T15:32:37Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
> {"type":"log","@timestamp":"2018-09-26T15:32:39Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
> {"type":"log","@timestamp":"2018-09-26T15:32:39Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
> {"type":"log","@timestamp":"2018-09-26T15:32:42Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
> {"type":"log","@timestamp":"2018-09-26T15:32:42Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
> 
> ```

- **Curator Container**

> **Summary**
>
> ```
> 2018-09-26 15:31:10,738 INFO Preparing Action ID: 1, "delete_indices"
> Traceback (most recent call last):
> File "/usr/bin/curator", line 9, in <module>
> load_entry_point('elasticsearch-curator==5.2.0', 'console_scripts', 'curator')()
> File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
> return self.main(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
> rv = self.invoke(ctx)
> File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
> return ctx.invoke(self.callback, **ctx.params)
> File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
> return callback(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/curator/cli.py", line 211, in cli
> run(config, action_file, dry_run)
> File "/usr/lib/python2.7/site-packages/curator/cli.py", line 158, in run
> client = get_client(**client_args)
> File "/usr/lib/python2.7/site-packages/curator/utils.py", line 800, in get_client
> 'Error: {0}'.format(e)
> elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch. Error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f1ce160fa90>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f1ce160fa90>: Failed to establish a new connection: [Errno 111] Connection refused)
> 2018-09-26 16:10:44,308 INFO Preparing Action ID: 1, "delete_indices"
> 2018-09-26 16:10:49,041 INFO Trying Action ID: 1, "delete_indices": Delete indices older than 30 days (based on index name), for logstash-eddata
> 
> ```

---

<div class="post-metadata">

**Author:** ![psd401](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@psd401](https://discuss.elastic.co/u/psd401)\
**Post date:** [September 27, 2018, 2:36pm UTC](https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049/3 "2018-09-27T14:36:37Z")

</div>

Just wanted to provide an update, after running the `curl -XGET 'http://elasticsearch:9200/_cluster/health?pretty'` command, my output is the following:

```
{
  "cluster_name" : "docker-cluster",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 7,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 11258,
  "active_shards" : 11258,
  "relocating_shards" : 0,
  "initializing_shards" : 16,
  "unassigned_shards" : 18227,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 11658,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 6410121,
  "active_shards_percent_as_number" : 38.161418257008236
}

```

I am going to assume that perhaps the crazy high amount of unassigned shards along with the pending\_tasks may be the culprit to all of these problems?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 27, 2018, 2:38pm UTC](https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049/4 "2018-09-27T14:38:22Z")

</div>

That is far too many shards for a cluster that size. Please read [this blog post on shards and sharding practices](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) for some practical guidelines.

---

<div class="post-metadata">

**Author:** ![psd401](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@psd401](https://discuss.elastic.co/u/psd401)\
**Post date:** [September 27, 2018, 2:45pm UTC](https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049/5 "2018-09-27T14:45:44Z")

</div>

Thank you for the response! I will make sure to go through that guide as I am taking over for the ElasticSearch side of things but this is my first time dabbling at this. Would setting up several new nodes (we are running this all through Docker) likely solve the issues?

Another question is whether having something such as ElasticSearch running on a Docker environment be a continued practice due to the high number of shards we may have. Docker may not be the issue here of course, but I wasn't sure if it is a common practice as I couldn't find much documentation around it.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 27, 2018, 2:48pm UTC](https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049/6 "2018-09-27T14:48:13Z")

</div>

Docker is not the issue here. That is far too many shard for an Elasticsearch cluster of that size no matter on what it is running on.

---

<div class="post-metadata">

**Author:** ![psd401](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@psd401](https://discuss.elastic.co/u/psd401)\
**Post date:** [October 4, 2018, 5:13pm UTC](https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049/7 "2018-10-04T17:13:52Z")

</div>

Hello quick question - It is obvious that we need to add more nodes to our cluster due to the amount of shards, I was thinking of adding perhaps 3-4 more shards. However, being that this is all in a Docker environment, can simply editing our Docker-compose with the new nodes configuration pointing to the other nodes simply get the job done? Or would there essentially be more configuration per each node?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 4, 2018, 5:16pm UTC](https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049/8 "2018-10-04T17:16:59Z")

</div>

I think you need to rethink how you approach sharding and reduce the shard count dramatically. Adding a few nodes may not make a massive difference.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 1, 2018, 5:19pm UTC](https://discuss.elastic.co/t/errors-with-elasticsearch-setup/150049/9 "2018-11-01T17:19:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.