# Timeout Errors when creating lots of new Indices? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 30, 2017, 8:00pm UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804 "2017-10-30T20:00:58Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![alexkeyes](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@alexkeyes](https://discuss.elastic.co/u/alexkeyes)\ **Post date:** [October 30, 2017, 8:00pm UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/1 "2017-10-30T20:00:58Z")
Hello, when I have an Elasticsearch cluster with nothing but .kibana and .marvel indicies in it, my consumer reading in the files will periodically encounter exceptions and close, due to the fact that ES is closing the connection ``` 17-10-20 17:15:33 epgidvledw1044 ERROR [signafire.packrat:23] - Uncaught exception on async-thread-macro-2 java.lang.Thread.run Thread.java: 745 java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 622 java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1152 ... clojure.core.async/thread-call/fn async.clj: 439 signafire.packrat.components.rodent.Rodent/fn rodent.clj: 97 signafire.packrat.components.rodent.Rodent/fn/fn rodent.clj: 131 clojure.core/dorun core.clj: 3009 clojure.core/seq core.clj: 137 ... clojure.core/map/fn core.clj: 2629 signafire.packrat.components.rodent.Rodent/fn/fn/fn rodent.clj: 136 signafire.packrat.components.rabbitmq/publish-failure! rabbitmq.clj: 78 langohr.queue/declare queue.clj: 75 com.rabbitmq.client.impl.recovery.AutorecoveringChannel.queueDeclare AutorecoveringChannel.java: 266 com.rabbitmq.client.impl.ChannelN.queueDeclare ChannelN.java: 844 com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc AMQChannel.java: 118 com.rabbitmq.client.impl.AMQChannel.privateRpc AMQChannel.java: 219 com.rabbitmq.client.impl.AMQChannel.rpc AMQChannel.java: 242 com.rabbitmq.client.impl.AMQChannel.quiescingRpc AMQChannel.java: 251 com.rabbitmq.client.impl.AMQChannel.quiescingTransmit AMQChannel.java: 316 com.rabbitmq.client.impl.AMQChannel.quiescingTransmit AMQChannel.java: 334 com.rabbitmq.client.impl.AMQCommand.transmit AMQCommand.java: 125 com.rabbitmq.client.impl.AMQConnection.flush AMQConnection.java: 518 com.rabbitmq.client.impl.SocketFrameHandler.flush SocketFrameHandler.java: 150 java.io.DataOutputStream.flush DataOutputStream.java: 123 java.io.BufferedOutputStream.flush BufferedOutputStream.java: 140 java.io.BufferedOutputStream.flushBuffer BufferedOutputStream.java: 82 java.net.SocketOutputStream.write SocketOutputStream.java: 161 java.net.SocketOutputStream.socketWrite SocketOutputStream.java: 115 java.net.SocketOutputStream.socketWrite0 SocketOutputStream.java java.net.SocketException: Broken pipe (Write failed) java.lang.Error: java.net.SocketException: Broken pipe (Write failed) ``` Packrat is the consumer I have that reads documents off RabbitMQ and into Elasticsearch search. When running this program under supervision, it will restart and grind through this process until the maximum number of indices needed are created and then everything runs well. I've found some other issues that seem to be related, but I'd like to note that I'm only creating 1 index for each month in a year and I'm also only allocating 5 shards for each date before 2000 and 25 for each date after 2000. I've attached the seemingly related issues below: > [@Problem when creating index : Request Timeout after 30000ms](https://discuss.elastic.co/t/problem-when-creating-index-request-timeout-after-30000ms/95593): > > Hi I get this error when I try to create a new index from kibana. (Management --\> Index pattern--\> Add new) When I try to search for that index in Dev tools I could see it. Request Timeout after 30000ms Version: 5.2.1 Build: 14707 Error: Request Timeout after 30000ms at [http://172.19.172.220:30067/bundles/kibana.bundle.js?v=14707:14:8629](http://172.19.172.220:30067/bundles/kibana.bundle.js?v=14707:14:8629) at [http://172.19.172.220:30067/bundles/kibana.bundle.js?v=14707:14:9050](http://172.19.172.220:30067/bundles/kibana.bundle.js?v=14707:14:9050) How do find the real cause of the issue & fix it ? > [@Kibana showing "Red" status, need help diagnosing/fixing](https://discuss.elastic.co/t/kibana-showing-red-status-need-help-diagnosing-fixing/93432): > > Hi all, ELK n00b here... I had a previously-working Kibana, but now it is in "Red" status, and I don't know how to go about diagnosing/fixing this... I will provide some details, and could someone kindly point me in the right direction slight_smile First off, running ELK stack version 5.4.1 on Ubuntu 16.04.2, all on a single server -- root@logstash01:/var/log# dpkg -l | grep -e elastic -e logstash -e kibana ii elasticsearch 5.4.1 … The ES cluster is 16 nodes, with 1 dedicated master, 2 dedicated clients and the rest are data nodes. The servers they are running on are all 8 core intel xeon e5 CPUs and 64GM of ram. Thanks! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 31, 2017, 3:27am UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/2 "2017-10-31T03:27:58Z")
What do the Elasticsearch logs show at this time? How do you know it's related to creating a lot of new indices? How many indices are you creating, and why? > [@alexatsignafire](#): > > I'm only creating 1 index for each month in a year and I'm also only allocating 5 shards for each date before 2000 and 25 for each date after 2000 Why so many for the newer data? > [@alexatsignafire](#): > > 1 dedicated master That's bad, see [Important Configuration Changes | Elasticsearch: The Definitive Guide [2.x] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/important-configuration-changes.html#_minimum_master_nodes) ---
**Author:** ![alexkeyes](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@alexkeyes](https://discuss.elastic.co/u/alexkeyes)\ **Post date:** [October 31, 2017, 8:46pm UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/3 "2017-10-31T20:46:03Z")
Hi Mark, To answer your questions: [2017-10-24 17:15:57,020][ERROR][marvel.agent] [[this.is.the.server.name](http://this.is.the.server.name)] background thread had an uncaught exception ElasticsearchException[failed to flush exporter bulks] at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:104) at org.elasticsearch.marvel.agent.exporter.ExportBulk.close(ExportBulk.java:53) at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:201) at java.lang.Thread.run(Thread.java:748) Suppressed: ElasticsearchException[failed to flush [default\_local] exporter bulk]; nested: ElasticsearchException[failure in bulk execution: [0]: index [.marvel-es-1-2017.10.24], type [node\_stats], id [AV9QPQR2OX3ZaVlWAb\_2], message [UnavailableShardsException[[.marvel-es-1-2017.10.24][0] primary shar d is not active Timeout: [1m], request: [BulkShardRequest to [.marvel-es-1-2017.10.24] containing [1] requests]]]]; at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:106) ... 3 more Caused by: ElasticsearchException[failure in bulk execution: [0]: index [.marvel-es-1-2017.10.24], type [node\_stats], id [AV9QPQR2OX3ZaVlWAb\_2], message [UnavailableShardsException[[.marvel-es-1-2017.10.24][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest to [.marvel-es-1-2017.10.24] containing [1] requests]]]] at org.elasticsearch.marvel.agent.exporter.local.LocalBulk.flush(LocalBulk.java:118) at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:101) ... 3 more Also there are so many more shards for the dates after 2000 because that's the lion's share of the data. The indices get created pertaining to the date the document was generated. an example of an index would be `sf_ab_doc __1998_03__ v2.1` . Right now in a testing ES cluster, we have about 606 indices with a few million documents and about 6500 shards. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 1, 2017, 2:39am UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/4 "2017-11-01T02:39:37Z")
> [@alexatsignafire](#): > > 606 indices with a few million documents and about 6500 shards. That's way too many for such a small dataset. ---
**Author:** ![alexkeyes](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@alexkeyes](https://discuss.elastic.co/u/alexkeyes)\ **Post date:** [November 1, 2017, 5:58pm UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/5 "2017-11-01T17:58:55Z")
oh okay, I didn't realize that. What number of indices/shards do you think I should have? Also do you think changing the way we index would help fix this problem? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 2, 2017, 6:47am UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/6 "2017-11-02T06:47:47Z")
Reducing the shard count will likely solve the problem. ---
**Author:** ![alexkeyes](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@alexkeyes](https://discuss.elastic.co/u/alexkeyes)\ **Post date:** [November 2, 2017, 10:27pm UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/7 "2017-11-02T22:27:36Z")
So I knocked the shards down to 5 per index, and that solved the problem! Thanks! I do have a few follow up questions though: 1. Why is creating shards, not indices what caused ES to time out? 2. Adding documents to ES is still relatively slow, at around 30 documents a second. What else can I do to speed this process up? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 3, 2017, 12:25am UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/8 "2017-11-03T00:25:52Z")
1. The indices are stored in cluster state. The shards are stored in a routing table, which is also in the cluster state. Every time a shard changes state (when created, started, recovered, initialised, moved) it needs to update the routing table and thus cluster state. And in 2.X the entire cluster state was updated and then sent to all other nodes in the cluster, whereas in 5.X we send just the changes that are made which is way more efficient. 2. Check `hot_threads`? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 3, 2017, 7:04am UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/9 "2017-11-03T07:04:04Z")
Cluster state updates are generally done single-threaded in order to ensure consistency. The changes are then propagated to all the nodes in the cluster. In addition to adding or altering indices and changing the location and state of shards, changes to mappings also require the cluster state to be updated. If you therefore are using dynamic mappings and have frequent changes/additions, this will result in an even larger number of cluster state changes. ---
**Author:** ![alexkeyes](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@alexkeyes](https://discuss.elastic.co/u/alexkeyes)\ **Post date:** [November 6, 2017, 4:01pm UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/10 "2017-11-06T16:01:26Z")
Thanks for the explanation guys! Really helpful insights. I see an update of ES in my future..... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 4, 2017, 4:05pm UTC](https://discuss.elastic.co/t/timeout-errors-when-creating-lots-of-new-indices/105804/11 "2017-12-04T16:05:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.